Subject: Re: [Idr] Fwd: I-D Action: draft-sas-idr-maxprefix-outbound-00.txt

From: Warren Kumari <[email-masked]>
Date: Tue, 6 Oct 2020 16:18:38 -0400

<no hats>
On Wed, Sep 30, 2020 at 10:53 AM Gyan Mishra <[email-masked]> wrote:
>
> Hi Authors
>
>
> Would it be possible to modify the action so that we have the option to not disconnect the peer and allow the peer to remain UP state but clip the routes above the upper limit and provide this option for both inbound and outbound directions.

Errrr... how sure are you that this is the behavior that you *want*?
If you have tripped the max-prefixes limit it's almost always:
1: things have been slowly growing over time, you've tripped your
warning limit. The right thing to do here is carefully look at the
prefixes, make sure they are what you expect, and bump it up by a bit.
2: you've just borked your filters and are now trying to leak full
tables to your peers. The right thing to do here is tear the session
down and go do penance...

I strongly suggest taking this question to GROW / NOGs before adding a
"send as many as you can and then start filtering" option; tripping
the hard limit should be the same as a circuit breaker, not a
resistor.

W

> This was the PE resources are not impacted as well as the customer peer still remains in an Up state.
>
> Thanks
>
> Gyan
>
> On Wed, Sep 30, 2020 at 9:34 AM Melchior Aelmans <[email-masked]> wrote:
>>
>> Hi IDR,
>>
>> As suggested in earlier WG meetings (both in GROW and IDR) we have split the Maximum Prefix Limits draft into Maximum Prefix Limits Outbound and Maximum Prefix Limits Inbound.
>> The authors are looking for your feedback and input on both.
>>
>> Thanks,
>> Melchior
>>
>> ---------- Forwarded message ---------
>> From: <[email-masked]>
>> Date: Wed, Sep 30, 2020 at 3:27 PM
>> Subject: I-D Action: draft-sas-idr-maxprefix-outbound-00.txt
>> To: <[email-masked]>
>>
>>
>>
>>
>>
>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>>
>>
>>
>>
>>
>>
>>
>>
>>         Title           : Revised BGP Maximum Prefix Limits Outbound
>>
>>
>>         Authors         : Melchior Aelmans
>>
>>
>>                           Massimiliano Stucchi
>>
>>
>>                           Job Snijders
>>
>>
>>         Filename        : draft-sas-idr-maxprefix-outbound-00.txt
>>
>>
>>         Pages           : 9
>>
>>
>>         Date            : 2020-09-30
>>
>>
>>
>>
>>
>> Abstract:
>>
>>
>>    This document updates RFC4271 by adding a control mechanism which
>>
>>
>>    limits the negative impact of outbound route leaks (RFC7908) in order
>>
>>
>>    to prevent resource exhaustion in Border Gateway Protocol (BGP)
>>
>>
>>    implementations.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The IETF datatracker status page for this draft is:
>>
>>
>> https://datatracker.ietf.org/doc/draft-sas-idr-maxprefix-outbound/
>>
>>
>>
>>
>>
>> There are also htmlized versions available at:
>>
>>
>> https://tools.ietf.org/html/draft-sas-idr-maxprefix-outbound-00
>>
>>
>> https://datatracker.ietf.org/doc/html/draft-sas-idr-maxprefix-outbound-00
>>
>>
>>
>>
>>
>>
>>
>>
>> Please note that it may take a couple of minutes from the time of submission
>>
>>
>> until the htmlized version and diff are available at tools.ietf.org.
>>
>>
>>
>>
>>
>> Internet-Drafts are also available by anonymous FTP at:
>>
>>
>> ftp://ftp.ietf.org/internet-drafts/
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>>
>> I-D-Announce mailing list
>>
>>
>> [email-masked]
>>
>>
>> https://www.ietf.org/mailman/listinfo/i-d-announce
>>
>>
>> Internet-Draft directories: http://www.ietf.org/shadow.html
>>
>>
>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>
>>
>>
>>
>> _______________________________________________
>>
>> Idr mailing list
>>
>> [email-masked]
>>
>> https://www.ietf.org/mailman/listinfo/idr
>>
> --
>
>
> Gyan Mishra
>
> Network Solutions Architect
>
> M 301 502-1347
> 13101 Columbia Pike
> Silver Spring, MD
>
>
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 7 Oct 2020 00:36:16 +0000

Or at least add some warning text to the option to the effect of:

If the max-prefix limit causes excess prefixes not to be announced
rather than a session termination, then the prefixes that are not announced
are unknown. Differing unknown sets of prefixes not being advertised
by multiple routers can cause forwarding loops.

Or specify the option for session non-termination for external BGP sessions only.

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Warren Kumari
Sent: Tuesday, October 6, 2020 1:19 PM
To: Gyan Mishra <[email-masked]>
Cc: idr <[email-masked]>
Subject: Re: [Idr] Fwd: I-D Action: draft-sas-idr-maxprefix-outbound-00.txt

<no hats>

W

---

From: Ben Maddison <[email-masked]>
Date: Wed, 7 Oct 2020 09:51:48 +0200

Hi all,

> the hard limit should be the same as a circuit breaker, not a

I'm with Warren on this.
If the log that the warning level generates goes unnoticed, then it's highly
likely that whatever byzantine forwarding issue the "clipping" creates
will do so too, and possibly persist for a long time.

I wouldn't want to be the on-call engineer in the middle of the night
that has to make the logical leap from a ticket that says "we're seeing
a weird-looking traceroute ..." to "we have half-tripped someone else's
max-prefix limit".

Additionally there is also no guarantee that in the event of a leak the
"leaked" prefixes are the ones that are "clipped". It's just as likely
that you end up with the worst case scenario of causing an outage for
the legit prefixes *and* allowing the leak to persist!

Cheers,

Ben

---

From: Job Snijders <[email-masked]>
Date: Wed, 7 Oct 2020 10:00:59 +0000

Dear Jakob, Warren, Ben, Gyan, others,

I agree that for Internet routing purposes it probably is very
undesirable to keep a connection up and 'clip' when the Maximum Prefix
Limit is reached. However, RFC 4271 explicitly permits this type of
behavior, RFC 4271 Section 6.7:

    "A BGP speaker MAY support the ability to impose a
    locally-configured, upper bound on the number of address prefixes
    the speaker is willing to accept from a neighbor.  When the upper
    bound is reached, the speaker, under control of local configuration,
    either (a) discards new address prefixes from the neighbor (while
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    maintaining the BGP connection with the neighbor), or (b) terminates
    the BGP connection with the neighbor."

I think Jakob's suggestion to add warning text is the best path forward,
I don't think this *clarification* RFC is the place to undefine this
type of (potentially problematic) functionality.

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Wed, 07 Oct 2020 07:00:44 -0700

> the legit prefixes *and* allowing the leak to persist!

consider me horrified.  this is a disaster that will not wait long
to happen.

[ i would have thought this obvious, but decided that saying nothing
  might be unwise.  sorry for the mailbox clutter. ]

randy

---

From: Brian Dickson <[email-masked]>
Date: Wed, 7 Oct 2020 13:10:05 -0700

On Wed, Sep 30, 2020 at 7:53 AM Gyan Mishra <[email-masked]> wrote:

> Hi Authors
>
>
> Would it be possible to modify the action so that we have the option to
> not disconnect the peer and allow the peer to remain UP state but clip the
> routes above the upper limit and provide this option for both inbound and
> outbound directions.  This was the PE resources are not impacted as well as
> the customer peer still remains in an Up state.
>
> Thanks
>
> Gyan
>

As most other folks responding have already said, this is a terrible idea.

At the very most, this sort of thing should only be an option for very
specific AFI/SAFI combos.

Please do NOT permit this behavior on the AFI/SAFI for unicast IPv4 or
unicast IPv6 (i.e., "the internet").

Brian Dickson
(speaking for myself, but representing I hope the entirety of the GROW
community and internet operators everywhere, who have suddenly had a
strange feeling that something bad is happening somewhere.)

---

From: Melchior Aelmans <[email-masked]>
Date: Fri, 16 Oct 2020 10:30:47 +0200

Hi IDR,

We have updated the maxprefix-inbound and -outbound drafts based on your
feedback.

Please find the new versions here:
https://tools.ietf.org/html/draft-sas-idr-maxprefix-inbound-01
https://tools.ietf.org/html/draft-sas-idr-maxprefix-outbound-01

As always we are very much looking forward to your feedback.

Cheers,
Melchior, Max, Job

On Wed, Sep 30, 2020 at 3:32 PM Melchior Aelmans <[email-masked]>
wrote:

> Hi IDR,
>
> As suggested in earlier WG meetings (both in GROW and IDR) we have split
> the Maximum Prefix Limits draft
> <https://datatracker.ietf.org/doc/draft-sa-grow-maxprefix/> into Maximum
> Prefix Limits Outbound
> <https://datatracker.ietf.org/doc/draft-sas-idr-maxprefix-outbound/> and Maximum
> Prefix Limits Inbound
> <https://datatracker.ietf.org/doc/draft-sas-idr-maxprefix-inbound/>.

---

From: Melchior Aelmans <[email-masked]>
Date: Tue, 20 Oct 2020 19:35:09 +0200

Hi Gyan,

We've uploaded new versions of both the in- and outbound draft. Please have
a look when you have time and comment.
Thanks again for your suggestions. The source of the drafts is to be found here
on GitHub <https://github.com/bgp/draft-ietf-maxprefix>. Feel free to sent
and suggest via GitHub if you prefer.

Cheers,
Melchior

On Thu, Oct 1, 2020 at 12:29 AM Gyan Mishra <[email-masked]> wrote:

>
> Hi Melchior
>
> Thanks for the quick response!
>
> For both inbound and outbound direction we have today the option to set
> prefix limit and threshold for the peer disconnect scenario, so now we
> would add a 3rd parameter is your suggestion.
>
> I agree and that makes sense.  With this option when the limit hits and we
> are clipping over the limit it would be possible to also do the warning
> flag with it so we are logging when clipping and maybe even number of
> routes clipped in the log message.
>
> If you have a GitHub for the draft I would be happy to co-author the draft
> and provide any additional feedback.
>
> Kind Regards
>
>
> Gyan
> On Wed, Sep 30, 2020 at 12:27 PM Melchior Aelmans <[email-masked]>
> wrote:
>
>> Hi Gyan,
>>
>> Thanks for the feedback!
>> Good point and agreed this could be useful. Would you be okay with having
>> this as an additional action next to shutdown?
>>
>> Cheers,
>> Melchior
>>
>> On Wed, Sep 30, 2020 at 4:53 PM Gyan Mishra <[email-masked]>
>> wrote:
>>
>>> Hi Authors
>>>
>>>
>>>> As suggested in earlier WG meetings (both in GROW and IDR) we have
>>>> split the Maximum Prefix Limits draft
>>>> <https://datatracker.ietf.org/doc/draft-sas-idr-maxprefix-outbound/>
>>>>  and Maximum Prefix Limits Inbound
>>> <http://www.verizon.com/>
>>>
>>> *Gyan Mishra*
>>>
>>> *Network Solutions A**rchitect *
>>>
>>>
>>>
>>> *M 301 502-134713101 Columbia Pike
>>> <https://www.google.com/maps/search/13101+Columbia+Pike%C2%A0+Silver+Spring,+MD?entry=gmail&source=g>*Silver
>>> Spring, MD
>>> <https://www.google.com/maps/search/13101+Columbia+Pike%C2%A0+Silver+Spring,+MD?entry=gmail&source=g>
>>>
>>>
>>>
>>>
>>
>> --
>
> <http://www.verizon.com/>
>
> *Gyan Mishra*
>
> *Network Solutions A**rchitect *
>
>
>
> *M 301 502-134713101 Columbia Pike *Silver Spring, MD
>
>

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 20 Oct 2020 13:59:14 -0400

Operators have in their toolbox a wide variety of filters to use from
prefix list filters to as-path to custom community based filters and of
course very coarse and granular filters.

>From an operators POV, the maximum prefix feature  is used only for special
use cases, and used for specific types of peering points namely external
connections outside of their administrative domain.

I think context is really important as it relates to the usage of maximum
prefix inbound or outbound.

>From an operators perspective POV, the most widely used use case customer
side of  PE-CE connections for managed MPLS / SR  service public or private
or MPLS service that is provided by same administrative domain.  The other
most widely used use case is PE operator side for both public internet and
private PE-CE “external” connections leaving or entering an administrative
domain.

The reason the maximum prefix command is important on customer side of
PE-CE for MPLS service is case where the same ASN is used for all customer
edges  connected to the managed service.  In such  case the provider does
an as-override towards the CE so the prefixes are not dropped from as-path
check loop detection. In such a case as-path filters cannot be used on the
customer side to permit interesting traffic so maximum prefix is required.
The main issue to be subverted is AS the was inadvertently used by another
customer and then with as-override on PE was hidden and a flood of prefixes
we’re advertised by the PE from this rogue AS.  The secondary issue to be
prevented is a flood of prefixes from the provider in the as-override case
where accidentally an RT was provisioned to be imported on the customer VPN
resulting in a flood.  In this particular case reason for the clipping
feature is on the customer side PE-CE private MPLS connection where the
maximum prefixes is a “known value” and not random, however disconnect of
the peer would result in an outage for the customer.  In this scenario the
 percent threshold with warning flag is not an option as that would not
protect the customer of a “flood” of prefixes based on the use case
described above.  In this particular case which I think is a common case
where you are connected to managed MPLS service or I house MPLS within same
administrative domain both private MPLS use case where you know the exact
maximum number of routes so you can set the maximum but if the maximum is
exceeded from a flood you don’t want that to result in loss of service
outage scenario - so it is this use case where clipping over the maximum is
much better then being in a hard down outage situation.  As Jacob mentioned
if a flag or log message was generated stating clipping over the maximum
that would alert the NOC that an event had occurred but at least an outage
had been subverted and during a scheduled maintenance window the problem
can be resolved.

The maximum prefix command is as well used widely on the PE-CE public
internet PE side see below use cases:

Internet use case:

Customer side PE-CE use case:

The use cases for customers connected to the internet in general not used
as most customers only accept a single default prefix and not the full
internet table.

Operator PE side PE-CE use case:

Internet providers PE side gateway PE-CE connections would would generally
set maximum prefix  to disconnect if maximum prefix is reached. This is a
secondary protection mechanism as the primary is prefix length and AS path
checks.

The routing loop would not occur as a result of this as the prefixes above
the maximum would be dropped and those would be more then likely result of
the flood of routes scenario so routes that should be filtered during the
flood event.  Even if some of interesting prefixes in the BGP update were
not accepted at least the end result would not be an outage as with the
peer disconnecting requiring manual intervention.

This clip feature would be an option elected by the operator PE side or CE
side to chose to use or not use.

In general this would never be used by operator PE side and really would be
an option on CE side for flexibility to prevent an outage.

Their maybe special corner case on PE side if there are resource issues on
the PE and maximum prefix is set per VPN is not enough and the operator on
PE side decides it does not want to disconnect critical customers per their
Ask to avoid an outage.

I understand this is more of a GROW topic and I can take it up with that
WG.

Also as an aside, as this is a command feature implemented by most vendors
and not anything intrinsic to BGP operation,  I was surprised that their is
a specification for maximum prefix.  As this is a locally significant
feature and does not require any interoperability could be something really
addressed directly by vendors as well.

Thanks

Gyan

-- 

*Gyan Mishra*

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 20 Oct 2020 14:03:57 -0400

Thank you!!

Sorry - I saw your reply after I drafted the email below.

Gyan

On Tue, Oct 20, 2020 at 1:35 PM Melchior Aelmans <[email-masked]>
wrote:

> Hi Gyan,
>
> We've uploaded new versions of both the in- and outbound draft. Please
> have a look when you have time and comment.
> on GitHub <https://github.com/bgp/draft-ietf-maxprefix>. Feel free to
> sent and suggest via GitHub if you prefer.
>
> Cheers,
> Melchior
>
>> I agree and that makes sense.  With this option when the limit hits and
>> we are clipping over the limit it would be possible to also do the warning
>> If you have a GitHub for the draft I would be happy to co-author the
>> draft and provide any additional feedback.
>>
>> Kind Regards
>>
>>
>> Gyan
>>> Good point and agreed this could be useful. Would you be okay with
>>> having this as an additional action next to shutdown?
>>>
>>> Cheers,
>>> Melchior
>>>

*Gyan Mishra*

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 25 Oct 2020 02:21:23 -0400

Hi Melchior

I read the two drafts rev 1 split into two separate drafts.  Looks good.

Let me know what you think of adding the use cases to the draft that I
mentioned in my rebuttal as to the context of the clipping of routes over
the maximum use case with logging.   Also this gives the operator
flexibility to disconnect, or log without disconnect of session, or clip at
the maximum without disconnect.

This 3rd option is important to operators to avoid an outage from
disconnect.

IDR WG

Please comment on the feedback context I have given on the clipping with
logging at the maximum prefix.

Kind Regards

Gyan

*Gyan Mishra*