Subject: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

From: Ben Cox <[email-masked]>
Date: Tue, 20 Apr 2021 14:47:35 +0100

Hi IDR folk,

In response to previous discussion over BGP protocol interactions with
TCP Zero Windows
(https://mailarchive.ietf.org/arch/msg/idr/L9nWFBpW0Tci0c9DGfMoqC1j_sA/)
me (who is new to the IETF, so please forgive me for my inevitable
errors) and Job Snijders have decided to attempt to address this and
focus discussion on handling this case with a internet draft, the gist
of it in the abstract of the draft so I won't re-word it here :)

--
   This document defines the SendHoldTimer session attribute for the
   Border Gateway Protocol (BGP) Finite State Machine (FSM).  A session
   should be terminated if the TCP receive window is zero for the
   duration of the Send Hold Timer, in this situation the peer is
   expected to terminate the connection.  For robustness, this document
   specifies that the local system should also close the connection.
   This document updates RFC4271.
--

We submitted our first draft today (
https://datatracker.ietf.org/doc/draft-spaghetti-idr-bgp-sendholdtimer/
) and we are looking for feedback knowing that it is not complete but
is likely in a state for some discussion.

Since this draft tweaks the BGP FSM, we would like to make sure that
it's done correctly and so we are soliciting help from this working
group.

We already have a test BGP speaker that triggers some of these zero
window edge cases. If that helps, please get in touch with us for the
peering information.

Looking forward to feedback!
Ben

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 23 Apr 2021 00:56:40 +0000

This phenomena is
   theorised to have contributed to IP traffic backholing events in
   global Internet routing system [l3outage]

Please remove the above statement.
Baseless accusations are not appropriate in a standards track RFC.

I propose to add this text:

It is possible that a BGP speaker presents a TCP zero receive window
after receiving a large batch of updates. Resetting the session to
such a speaker may just repeat the large batch, whereas waiting a little
longer may have given it a chance to recover. The possibility of this
occurrence makes it difficult to determine a good value for the
timer. The default value of the timer is therefore recommended
to be 15 minutes.

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Ben Cox
Sent: Tuesday, April 20, 2021 6:48 AM
To: [email-masked]
Subject: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Hi IDR folk,

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: Job Snijders <[email-masked]>
Date: Fri, 23 Apr 2021 14:43:02 +0200

Hi Jakob, group,

Thanks for the feedback, a -02 has been submitted.

> timer. The default value of the timer is therefore recommended

In this email John Scudder appears to suggest using a separate new
timer, and populate the timer with the default HoldTimer value (and the
-02 draft attempts to describe exactly that).
https://mailarchive.ietf.org/arch/msg/idr/E5pqxIObPS7e9A2IqIApTHVSfig/

you propose 15 minutes; so it seems there is some convergent thinking in
the group the proposed timers at hand is in the order of 'multiple
minutes'.

How long should a BGP speaker be willing to buffer on behalf of another
BGP speaker's inability to progress message processing?

If the oldest message in the queue is a KEEPALIVE message, and its been
stuck there for more than HoldTimer, wouldn't we have expected the
remote side to have killed the BGP session anyway? 

Would you mind sharing your thinking on why a default of 15 minutes is
better than for example 180 seconds? What do others think?

Kind regards,

Job

---

From: William McCall <[email-masked]>
Date: Fri, 23 Apr 2021 15:38:03 +0000

> better than for example 180 seconds? What do others think?

15 min seems a safe default, if somewhat arbitrary.

The default has to be safe enough to survive slowness, especially
during initial convergence, across a wide variety of deployments.

What is "slow" is going to depend on the profile of the deployment. So
tunability in implementation is probably going to be more critical (at
least in my thoughts) than the default in practice.

I think this is only a piece of what BGP really ought to measure and
act against in these cases. BGP should have some understanding of
far-side message processing latency. Because if my queue is
progressing at 1 update/10 min, it is going to be really behind even
when there is progress.  I'm a big fan of a ping + nonce capability to
estimate this latency in addition to watching update outQ time. They
handle for complimentary conditions.

-- 
William McCall

---

From: Enke Chen <[email-masked]>
Date: Fri, 23 Apr 2021 10:30:44 -0700

Hi, Folks:

The holdtimer already conveys whether the session should be timed out
sooner or later. It makes sense to tie the new timer with it, perhaps with
a configurable multiplier. The default could be 1, or 2.

Thanks.   -- Enke

On Fri, Apr 23, 2021 at 5:43 AM Job Snijders <job=
[email-masked]> wrote:

> https://urldefense.proofpoint.com/v2/url?u=https-3A__mailarchive.ietf.org_arch_msg_idr_E5pqxIObPS7e9A2IqIApTHVSfig_&d=DwICAg&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=bQFpOg20Jvvi5Fdn0FbsPn3umJ_yRN4nNcXfCsfl5ek&s=ic9GqvTL_AVNp--whyEU4xnDPEpswdHPBNt_XoBG99g&e=
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_idr&d=DwICAg&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=bQFpOg20Jvvi5Fdn0FbsPn3umJ_yRN4nNcXfCsfl5ek&s=PHKZKmQoCjqpWGGBhDIpOnAaSlDjeRsLdEPlOwChzyw&e=
>

---

From: Brian Dickson <[email-masked]>
Date: Fri, 23 Apr 2021 11:51:16 -0700

> a configurable multiplier. The default could be 1, or 2.
This is a bad idea (having the only "knob" be a multiplier).

It might be okay for the default to be derived from some multiplier over
the default for holdtimer, but it is not okay for the knob to be ONLY a
multiplier.

Here's why:

A router with a large number of BGP sessions, where those sessions have
different characteristics, such as RTT, loss, bandwidth, etc., are very
likely to behave quite differently.
(There are any number of operational reasons a router might have
connections of this sort, it's really not worth detailing all the reasons,
but some examples would include a router with both local LAN peers
(customers) and satellite peers; peering extension providers over
multi-city links; underprovisioned peer routers; software router peers,
etc.)

The operator should (MUST) have exposed controls to adjust timers on a
per-neighbor basis, and/or per-router basis (when managing large numbers of
routers).

The purpose of this proposal is not met if that level of granularity is
unavailable.

The operator has the ability to understand why individual peering sessions
may be behaving this way, and will need to adjust the parameters on a
per-peer basis.
Fixing a problem for one peer, should not introduce problems for another
peer.

This isn't 1995.

Respectfully,
Brian

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 23 Apr 2021 20:08:56 +0000

Phenomena that cause keepalives (or any packets) to be delayed are
different to phenomena that cause zero window.
Network delays caused by buffering in intermediate switches as well
as at the TCP endpoints will delay keepalives and will also delay the
clearing of a zero window condition.
There are a myriad of other phenomena that can cause zero window
to persist that are not network related.
Ideally, even a slow receiver should be punctuating periods of zero
window with short periods of non-zero window as it moves its
in-queue. However, that's not a requirement. An implementation
that takes a big gulp and then lies down to chew its cud for 10
minutes is not breaking any rules. I don't know of any such
implementations, but I've not seen them all.

I prefer the medicine to be safe before effective, so I will advocate
for a long default timer.

Regards,
Jakob.

From: Enke Chen <[email-masked]>
Sent: Friday, April 23, 2021 11:54 AM
To: Brian Dickson <[email-masked]>
Cc: Job Snijders <[email-masked]>; Jakob Heitz (jheitz) <[email-masked]>; Ben Cox <[email-masked]>; [email-masked]
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Hi, Brian:

Isn't the holdtimer already per-neighbor based?

Regards,  -- Enke

On Fri, Apr 23, 2021 at 11:51 AM Brian Dickson <[email-masked]<mailto:[email-masked]>> wrote:

On Fri, Apr 23, 2021 at 10:31 AM Enke Chen <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Folks:

The holdtimer already conveys whether the session should be timed out sooner or later. It makes sense to tie the new timer with it, perhaps with a configurable multiplier. The default could be 1, or 2.

It might be okay for the default to be derived from some multiplier over the default for holdtimer, but it is not okay for the knob to be ONLY a multiplier.

Here's why:

A router with a large number of BGP sessions, where those sessions have different characteristics, such as RTT, loss, bandwidth, etc., are very likely to behave quite differently.
(There are any number of operational reasons a router might have connections of this sort, it's really not worth detailing all the reasons, but some examples would include a router with both local LAN peers (customers) and satellite peers; peering extension providers over multi-city links; underprovisioned peer routers; software router peers, etc.)

The operator should (MUST) have exposed controls to adjust timers on a per-neighbor basis, and/or per-router basis (when managing large numbers of routers).

The purpose of this proposal is not met if that level of granularity is unavailable.

The operator has the ability to understand why individual peering sessions may be behaving this way, and will need to adjust the parameters on a per-peer basis.
Fixing a problem for one peer, should not introduce problems for another peer.

This isn't 1995.

Respectfully,
Brian

Thanks.   -- Enke

On Fri, Apr 23, 2021 at 5:43 AM Job Snijders <job=[email-masked]<mailto:[email-masked]>> wrote:
Hi Jakob, group,

Kind regards,

Job

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_idr&d=DwICAg&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=bQFpOg20Jvvi5Fdn0FbsPn3umJ_yRN4nNcXfCsfl5ek&s=PHKZKmQoCjqpWGGBhDIpOnAaSlDjeRsLdEPlOwChzyw&e=
_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_idr&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=ecQzdLStIaoAFzguVP-QtyJpUCwi8K4lXMCt2ey-mZE&s=GquAlygyileCKhzvo6AEjoKhUu6R2Z3sSmft31rxH9s&e=>

---

From: Enke Chen <[email-masked]>
Date: Fri, 23 Apr 2021 11:54:21 -0700

Hi, Brian:

Regards,  -- Enke

On Fri, Apr 23, 2021 at 11:51 AM Brian Dickson <
[email-masked]> wrote:

>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_idr&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=ecQzdLStIaoAFzguVP-QtyJpUCwi8K4lXMCt2ey-mZE&s=GquAlygyileCKhzvo6AEjoKhUu6R2Z3sSmft31rxH9s&e=>
>>
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 23 Apr 2021 17:23:48 -0400

> it's done correctly and so we are soliciting help from this working

As someone who spent a lot of time working with the editors of what became
RFC 4271, you're going to find there's a lot of corner cases in the FSM to
deal with.  :-)  We can work on the text once it's clearer what we should
do.

A number of stream of thought comments about the topic covered by the draft:

I don't object quite as much as Jakob does about the issues of claiming
implementations that are exhibiting a stuck session as contributing to stale
routes and blackholes in the Internet.  A simple truth of BGP as a
hop-by-hop protocol is that if you can't distribute your updates, the
downstreams are operating on an incorrect version of reality.  

That said, the problem statement isn't really crisp enough I'd want to make
such a large claim.

TCP applications will exhibit zero-windowing naturally over the course of a
session's lifetime.  This typically will occur when there's some level of
backpressure on the application due to I/O issues or even CPU.  While these
situations aren't desirable since they signal congestion in the application,
congestion SHALL happen.[1]

Most of the headaches I'd discusss for the problem this draft attempts to
solve revolve around the consequences of this statement in section 2 of the
draft:

:   Generally BGP implementation have no visibility into lower-layer
:   subsystems such as TCP or the peer's current Receive Window.
:   Therefor this document banks on BGP implementations being able to
:   detect an inability to push more data to the remote peer, at which
:   point the SendHoldTimer starts.

It is quite true that it is difficult to get a sense of what's going on when
you have simple socket (for example) backpressure.  What I believe an
application really wants to know for some sort of send-hold-timer is the
following:

- A session's TCP window has reached zero availability at a given sequence
  number.
- It stays at that sequence number with zero availability for "too long".

Typical POSIX socket APIs won't give you this kind of certitude.  The
condition, as written in the draft, effectively manifests as a write() to
the socket that fails.  But how does it fail:
- Is it because there is no space in the socket to accept data?
- Is it because there are no local resources (e.g. mbufs) to push the next
  bit of data?
- Is it because a kernel timer hasn't triggered a bit of TCP behavior to try
  to move some data?
- Is it because we had blocked and we're waiting for select/kevent to fire
  and let us know that there's space - and it simply hasn't bothered even
  though there's space in the window?
- Is it because a low-watermark feature is preventing the socket from being
  ready even though the window has advanced?
- Is it because the session is draining VERY.... SLOWLY... and the kernel
  won't wake us up even though the window is no longer zero - just not big
  enough? (A version of the prior issue.)

Some things that can cause zero windowing may be worth keeping in mind to
decide how bad they can be and what an implementation may want to do about
it:

- Perhaps it's doing a live-upgrade and needs to wedge the incoming session
  to do so? (ISSU, e.g.)
- It might be configured with incoming session prioritization, and your
  socket isn't important enough by policy.  (Corrollary: Not all routes are
  created equal.)
- It may be behind in servicing the socket. For example, too much work,
  perhaps due to a reconfiguration event that will resolve at some point.
- A firewall may be interfering with TCP and the receiver is actually fine.

Finally, I want to highlight something rather critical:  
Just because you reset your session, in some of these pathologies the router
you're resetting your session with may not withdraw your routes downstream
fast enough.  

This is the stale situation highlighted in the draft.  The distinction is
that sometimes thrashing makes it worse.

If you're going to disconnect from such a peer, you had better have enough
routing diversity to survive disconnecting from it.  You also better be
prepared to keep the session down for some time.  

Keeping the session down is not addressed in the draft.

-- Jeff

[1] When I give presentations discussing BGP, I tend to note that BGP is for
the most part a simple protocol with a lot of optional features... and the
hard part is making it scale.

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 24 Apr 2021 00:23:15 +0200

One thing which also I am worried about with this proposal is that data
plane may be working just fine (imagine stub ASN where it advertised a
prefix and received default) yet zero window was signalled by the peer for
any of those reasons Jeff nicely enumerated.

So what we are discussing is breaking data plane just because control plane
has experienced 15 min (or worse recommended 4 min) inability to send
keepalives.

Even for dual connected stub sites usually it means different exit
interface switchover and different NAT breaking all end user's connections.

Of course none of this applies when you are a transit router with many
peers.

And I am not even touching the control plane separation from data plane
architectures here.

So two questions ..

* Should we perhaps test data plane before declaring peer's failure and
before we reset the session ? (I understand that the paramount motivation
is BGP consistency here though - but this is one of those cases where one
size may not fit all).

* Should we first withdraw received routes from our peers before resetting
the session ? At least data plane will have a chance to converge to a
different set of links with no sudden packet drops.

Thx,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 23 Apr 2021 20:48:39 -0400

Robert,

> has experienced 15 min (or worse recommended 4 min) inability to send

A good analogy is the negative impacts of stale routes when you use Graceful
Restart for BGP.  Can you live with the routes in that flavor of stale for
that long?

Arguably, knowing what may be queued up - or pushed to the socket but not
acknowledged - is possibly part of the decision process to decide if you
really need to drop the session.  If you have a stable topology and it's
just keepalives queued up, you may not care as much.

> is BGP consistency here though - but this is one of those cases where one

In many of these scenarios, BFD or ping would show the interface up.  It's
the TCP session that is stalled out.

> different set of links with no sudden packet drops.

Would you describe the drain scenario with the involved parties and what the
congestion state is as part of that?  I don't think I'm understanding the
above point.

-- Jeff

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Sat, 24 Apr 2021 01:55:48 +0000

Perhaps the draft should be clearer about the distinction
between TCP zero window and BGP's failure to write to the socket.
The condition to monitor is TCP zero window.
How that's done is an engineering problem, not one of standardization.

To be clear, I'm not against the draft or the technique.
It addresses a valid problem.
I want the draft to make the risk clear and to make sure that
operators factor it into the setting of the timer.

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Jeffrey Haas
Sent: Friday, April 23, 2021 5:49 PM
To: Robert Raszuk <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Robert,

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 24 Apr 2021 11:27:50 +0200

Hi Jeff,

> Restart for BGP.  Can you live with the routes in that flavor of stale for

Unfortunately the answer is "it depends". If my stale route is single
default to the world with working data plane I think the answer is clearly
YES.

So that IMHO sort of raises the question if this should be the default
behaviour.

Then honestly I am not quite clear how it should be handled on sessions
which are setup with draft-ietf-idr-long-lived-gr-00

> the TCP session that is stalled out.

True.

I was rather thinking about checking data plane not to the subject peer but
beyond it (through it). For iBGP - testing next hops. For eBGP some known
test servers in yr network or in the Internet.

> congestion state is as part of that?  I don't think I'm understanding the

If I know I will bring BGP sessions down to you (in any case not only with
this draft) I should withdraw paths from all of my peers received over such
session and previously advertised (as best or as add-paths) before
resetting it. So the control plane clears before we touch the data plane.
Just to make the end to end reachability with no or minimal data plane
impact.

 Thx,
R.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Sun, 25 Apr 2021 06:01:06 +0000

A long time of TCP zero window does not indicate a data plane
problem, nor a problem with routes received from the stuck peer.
The blockage is in one direction only. The local speaker is unable
to end routes to the stuck peer, but is able to receive routes
from the stuck peer just fine.

Therefore, I would propose that the response of the local speaker
should be to retain the routes of the stuck peer when it resets the
session, GR style.

Indications of data plane problems or inability to receive routes from
the stuck peer are separately handled.

Regards,
Jakob.

From: Idr <[email-masked]> On Behalf Of Robert Raszuk
Sent: Saturday, April 24, 2021 2:28 AM
To: Jeffrey Haas <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Hi Jeff,

Unfortunately the answer is "it depends". If my stale route is single default to the world with working data plane I think the answer is clearly YES.

So that IMHO sort of raises the question if this should be the default behaviour.

Then honestly I am not quite clear how it should be handled on sessions which are setup with draft-ietf-idr-long-lived-gr-00

True.

I was rather thinking about checking data plane not to the subject peer but beyond it (through it). For iBGP - testing next hops. For eBGP some known test servers in yr network or in the Internet.

If I know I will bring BGP sessions down to you (in any case not only with this draft) I should withdraw paths from all of my peers received over such session and previously advertised (as best or as add-paths) before resetting it. So the control plane clears before we touch the data plane. Just to make the end to end reachability with no or minimal data plane impact.

 Thx,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 25 Apr 2021 13:50:56 +0200

Hey Jakob,

Yes that was exactly my point.

However the fundamental idea around this proposal was to actually reset the
session and trigger withdraw suspecting that the peer is not "feeling
well".

That is unfortunately the problem with completely different deployment
scenarios for BGP which BGP spec so far is trying to treat all in the same
way.

And IMHO the only good way to solve it is to follow what other protocols
did in similar situations and formally define notion of BGP (deployment)
profiles.

What's good for one profile may be pretty bad for the other one (Internet
Tier 1 peer vs stub CE as example). Today such profiles are already in
place except each network creates their own flavor of it. Maybe its time to
simplify BGP deployments and either in IDR or in GROW start work on few key
deployment scenarios where vendors could adjust bgp behaviour with single
per peer profile configuration ?

Thx,
R.

On Sun, Apr 25, 2021 at 8:01 AM Jakob Heitz (jheitz) <[email-masked]>
wrote:

> *From:* Idr <[email-masked]> *On Behalf Of * Robert Raszuk
> *Sent:* Saturday, April 24, 2021 2:28 AM
> *To:* Jeffrey Haas <[email-masked]>
> *Cc:* idr@ietf. org <[email-masked]>; Ben Cox <ben=
> [email-masked]>
> *Subject:* Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback
> requested
>
>
>
> Hi Jeff,
>
>
>
> which are setup with *draft-ietf-idr-long-lived-gr-00*
>
>
>
> I was rather thinking about checking data plane not to the subject peer
> but beyond it (through it). For iBGP - testing next hops. For eBGP some
> known test servers in yr network or in the Internet.
>
>
>
>
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sun, 25 Apr 2021 12:01:57 +0000

+1

Jim Uttaro

From: Idr <[email-masked]> On Behalf Of Jakob Heitz (jheitz)
Sent: Sunday, April 25, 2021 2:01 AM
To: Robert Raszuk <[email-masked]>; Jeffrey Haas <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Regards,
Jakob.

From: Idr <[email-masked]<mailto:[email-masked]>> On Behalf Of Robert Raszuk
Sent: Saturday, April 24, 2021 2:28 AM
To: Jeffrey Haas <[email-masked]<mailto:[email-masked]>>
Cc: idr@ietf. org <[email-masked]<mailto:[email-masked]>>; Ben Cox <ben=[email-masked]<mailto:ben=[email-masked]>>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Hi Jeff,

True.

 Thx,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Sun, 25 Apr 2021 15:27:05 -0400

Jakob,

> from the stuck peer just fine.

This tickles a related point:

If a peer has zero-windowed you, you are unable to send any ACK
frames to that peer.  This means that we're potentially in a very slow race
for the remote peer to eventually send enough data that it similarly zero-
windows.

At that point, the remote peer should eventually give up trying to transmit
as well. Locally, the hold timer would expire.

> should be to retain the routes of the stuck peer when it resets the

Having GR is a reasonable mitigation when you're not concerned about the
routes the local router having already been stuck for a very long time.

In some circumstances it may make better sense to get the router with the
misbehaving TCP session out of the forwarding path as possible.

-- Jeff

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Sun, 25 Apr 2021 23:28:43 +0000

My understanding of TCP zero window is different.
It is the peer sending 0 in the "window" field of the TCP header
in the 16 bit word, at offset 14 bytes into the header.
That means it will not accept any TCP data from you,
but it is perfectly capable of sending you data and
accepting your acks for it.
If it were not so, then it would not be able to send you keepalives
and your hold timer would expire.

If a peer sends you zero window, it means one thing only:
That it does not have your latest routes.

It does not mean that all its other routes are also stale.
If that were the case, then it would send zero window to its
other peers and they would take care of it themselves.

It does not mean that the data plane is broken.
If that were the case, you would get either no keepalives
from the peer or no more acks to your own keepalives.

If forwarding is broken one way from you to your peer, then
you will no longer get ACKs back from keepalives or other
BGP packets you send to your peer. This is not a zero window
condition. This may be another condition we wish to monitor.
Note, there are other ways to monitor this condition, e.g. BFD.

Regards,
Jakob.

-----Original Message-----
From: Jeffrey Haas <[email-masked]> 
Sent: Sunday, April 25, 2021 12:27 PM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Robert Raszuk <[email-masked]>; idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Jakob,

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Sun, 25 Apr 2021 20:43:02 -0400

Jakob,

> accepting your acks for it.

You're quite correct and I was mistaken here.

> and your hold timer would expire.

Eventually, which was my incorrect thinking.

> from the peer or no more acks to your own keepalives.

For my part, I don't believe that "broken data plane" is the main issue
here.  It's simply the fact that since the peer is blocked from sending
updates, we have a flavor of the dreaded "stuck route" problem.

> BGP packets you send to your peer.

Forwarding for directly attached peers (same subnet) often will behave well
when forwarding for remotely attached peers may fail.  A quirk of various
silicon.  And certainly not a guaranteed behavior.

>  This is not a zero window

I might know a little about that.

-- Jeff

---

From: tom petch <[email-masked]>
Date: Mon, 26 Apr 2021 16:07:59 +0000

From: Idr <[email-masked]> on behalf of Jeffrey Haas <[email-masked]>
Sent: 23 April 2021 22:23

<tp>
After which we will likely want to change the specification:-)  This I-D will cause extensive updates to s.8.2.2 about when the sendholdtimer is started, when it is stopped, when it is cleared.  Since this is to do with TCP, then the starting point for the specification would seem to be as soon as the TCP session is established ie once the FSM moves out of Idle state; but even that will not preclude a timer expiry in the Idle state so all states are affected.

You have six states and 28 events each of which have to be considered for each state and the effect of some events is affected by what else is happening, e.g. what other timers are running and by how the state was entered, which is often the same thing.

So,  yes, there are a lot of corner cases and then there are all the points that Jeff so accurately describes below..  This is a substantial change to the core routing protocol.

Tom Petch

. 

-- Jeff

---

From: tom petch <[email-masked]>
Date: Tue, 27 Apr 2021 08:22:14 +0000

inline <tp>

From: Idr <[email-masked]> on behalf of Jakob Heitz (jheitz) <jheitz=[email-masked]>
Sent: 26 April 2021 00:28

It does not mean that the data plane is broken.
If that were the case, you would get either no keepalives
from the peer or no more acks to your own keepalives.
<tp>
I agree with you up to the last sentence  If the TCP window is zero, then you cannot send anything, apart from ACK and similar control packets (since ACK controls the window and so has to get through).  Your own keepalives cannot be sent, just as updates cannot be sent.  I think that this is spelt out in the I-D.

I would say that the TCP dataplane is working just fine, it is the BGP dataplane that has become simplex and so not much use for routing!

Tom Petch
Tom Petch

Regards,
Jakob.

-----Original Message-----
From: Jeffrey Haas <[email-masked]>
Sent: Sunday, April 25, 2021 12:27 PM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Robert Raszuk <[email-masked]>; idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Jakob,

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 27 Apr 2021 08:47:24 -0400

> from the stuck peer just fine.

A further bit of thinking for the problem:

While it's true that the blockage is one way, it's not necessarily
guaranteed that you're getting routes from the peer.  This is true even if
you might be receiving keepalives.

The example case would be a BGP that had a decoupled keepalive mechanism for
its FSM.  The implementation may not be draining its incoming rib-in from a
socket, and similarly may be stuck in publishing its rib-out.

Of course, there's no way to tell if this is happening.

This isn't a strong argument that Graceful Restart proecures may not still
be appropriate.  After all, we have some portion of a rib that may be
correct.  However, there's perhaps an argument that that it might be less in
sync in some circumstances than others.

-- Jeff

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 27 Apr 2021 15:15:07 +0000

Comments In-Line..

Thanks,
	Jim Uttaro

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Jeffrey Haas
Sent: Tuesday, April 27, 2021 8:47 AM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>; Robert Raszuk <[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

[Jim U>] Certain BGP FOUs i.e Kompella, BGP LU are less dynamic implying that the specification of state is relatively static. For certain FOUs I accept less than perfection in the interest of maintaining a service or network. This discussion would benefit from recognizing the differences. 

-- Jeff

_______________________________________________
Idr mailing list
[email-masked]
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!BhdT!ySJMnbkQy8PZfA8mjrwVNx0cDwNiXNMgceFAghXreoy-Lu7ZeZ4ZRNka8T0$

---

From: heasley <[email-masked]>
Date: Tue, 27 Apr 2021 17:41:20 +0000

> I agree with you up to the last sentence  If the TCP window is zero, then you cannot send anything, apart from ACK and similar control packets (since ACK controls the window and so has to get throu…

Doesn't this depend on the TCP implementation, which should send a probe
with the seq of the last sent seq (snd.nxt - 1).  unless it is an old
impl., the probe data size should be zero - ie: no seq advancement.

if it could not send KAs, these tcp sessions would have expired after ~2h.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Tue, 27 Apr 2021 23:42:19 +0000

You're right Tom. inline.

Regards,
Jakob.

-----Original Message-----
From: tom petch <[email-masked]> 
Sent: Tuesday, April 27, 2021 1:22 AM
To: Jakob Heitz (jheitz) <[email-masked]>; Jeffrey Haas <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Ben Cox <[email-masked]>; Robert Raszuk <[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

inline <tp>

It does not mean that the data plane is broken.
If that were the case, you would get either no keepalives
from the peer or no more acks to your own keepalives.
<tp>
I agree with you up to the last sentence  If the TCP window is zero, then you cannot send anything, apart from ACK and similar control packets (since ACK controls the window and so has to get through).  Your own keepalives cannot be sent, just as updates cannot be sent.

[Jakob]
Whoops, you got me. Quick save: I meant no acks back from the zero-window probes.

  I think that this is spelt out in the I-D.

Tom Petch
Tom Petch

Regards,
Jakob.

-----Original Message-----
From: Jeffrey Haas <[email-masked]>
Sent: Sunday, April 25, 2021 12:27 PM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Robert Raszuk <[email-masked]>; idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

Jakob,

-- Jeff

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 28 Apr 2021 00:15:52 +0000

Jeff,

Your point, if I get it, is that if the peer is stuck receiving my
routes, it's likely stuck on receiving other routes too.

My response: It's a risk, but let the other routers take care of that
by resetting their sessions too.

I think the draft should have a discussion section to list all the
points we made in this thread to give operators a good base on which
to decide their own configurations.

I think the default should be a long timer and to retain the routes
when resetting the session. However, this choice should be configurable.

Regards,
Jakob.

-----Original Message-----
From: Jeffrey Haas <[email-masked]> 
Sent: Tuesday, April 27, 2021 5:47 AM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Robert Raszuk <[email-masked]>; idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 28 Apr 2021 12:25:08 +0200

> when resetting the session. However, this choice should be configurable.

There could be potentially another option ...

Once we can not write to a peer over existing TCP session we could try to
OPEN a new TCP session to the same address:179 with the same
security strings. We could use same or different BGP_ID.

/* When we discussed eBGP peering between VRFs of the same PE it was
confirmed that most if not all implementations (at that time) do not check
for different BGP_ID when accepting an OPEN). */

If a new session succeeds and we sync routes we are good. Old session can
be terminated. No data plane disruption at all.

If not it would be indeed a sign that peer's BGP is not doing that well and
perhaps confirmation that in some deployment situations it would be good to
remove him from the picture after draining his routes from our peers.

Best,
R.

---

From: Adam Chappell <[email-masked]>
Date: Wed, 28 Apr 2021 13:01:34 +0200

Hi Ben, I think you have a good draft here, but personally I think
this probably should be a simple advisory to implementors, eg. an
implementation MAY want to consider terminating sessions to peers with
whom they've been unable to write() to for a configurable amount of
time, maybe similar to Job's original change of wording[1] to sec6.5.
Leaves the door open for "profiles" as some have said eg.
"timeliness", "longevity" etc...

It is significant to me that at least two production implementations
I've seen offer visible CLI instrumentation when keepalives haven't
been written on schedule, so it seems that implementors are definitely
aware of the weakness, even if not acting on the situation today and
could do. Same viz a viz so-called update-groups and dynamic
definition of same.

I read your compelling blog post[2] and investigations but I have
missed the conclusive link between the phenomenon of zombies
(undisputed) and TCP sessions regularly stalled in this way. I
understand that you can synthesise the problem by creating a BGP
speaker that starves his peer of the oxygen to talk thus polluting
your own RIB; but I missed the pointer to evidence that it is indeed
this that is generally occurring and causing zombies. Not disputing
that it may be. I realise it may be bad form to associate names of
operators and vendors with bad events, but I think it is important
that we investigate exceptional events so that policy and protocol
changes like this have a sound basis.

[1] https://mailarchive.ietf.org/arch/msg/idr/L9nWFBpW0Tci0c9DGfMoqC1j_sA/
[2] https://blog.benjojo.co.uk/post/bgp-stuck-routes-tcp-zero-window

-- 

-- Adam.

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 28 Apr 2021 09:33:46 -0400

Adam,

On Wed, Apr 28, 2021 at 01:01:34PM +0200, Adam Chappell wrote:
> could do.

For scaled systems, scheduler slips for timers can be a serious challenge.
This is even without the headache of whether your network stack will
actually manage to get the scheduled packet to the remote device on time.

> that it may be.

"Stuck route" or "missing route/withdraw" bugs of various sorts are an
indisputable fact of prior and sometimes current BGP implementations.

Some of the circumstances described in this thread are, as you had noted,
perhaps not the issue itself but a visible manifestation of the issue.  An
implementation can't tell when it sees zero-windowing whether there's a TCP
issue, a starving reader on the remote side, or related problems.  It simply
knows that it can't push routing state that it wants to.

Prior and ongoing discussions about BGP troubleshooting (see for example the
thread in grow about the BGP Looking Glass Capability) can be partially
informed by these discussions.  This form of congestion means that an
in-band solution to such troubleshooting is problematic for some scenarios.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 28 Apr 2021 09:42:22 -0400

Jim,

> [Jim U>] Certain BGP FOUs i.e Kompella, BGP LU are less dynamic implying that the specification of state is relatively static. For certain FOUs I accept less than perfection in the interest of main…

A valid point.  And perhaps a related warning about the "all eggs in one
basket" problem of everything on the same session.  Structural separation of
address families is one way to avoid that problem.

This is the usual point where the conversation starts swinging towards the
BGP multisession document. 

And it's worth noting that an implementation may decide that during Graceful
Restart that it doesn't want to preserve the incoming RIB and can do so on a
per-family basis.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 28 Apr 2021 09:46:45 -0400

Jakob,

> when resetting the session. However, this choice should be configurable.

One of the things I think is valuable about the discussion is it likely
belongs in a document for users of TCP for routing - and perhaps other
applications.  BGP tends to just be the most visible "victim" of these sort
of problems:
- The sessions are long-lived.
- The amount of data is modest, but the timeliness requirements are
  important.
- The routing system is negatively impacted when this sort of congestion
  occurs.

I don't think we have strong consensus for what BGP should do when these
problems occur, but we're starting to build quite the attack tree for the
scenario.

-- Jeff

---

From: tom petch <[email-masked]>
Date: Wed, 28 Apr 2021 16:01:56 +0000

From: Idr <[email-masked]> on behalf of Robert Raszuk <[email-masked]>
Sent: 28 April 2021 11:25

Once we can not write to a peer over existing TCP session we could try to OPEN a new TCP session to the same address:179 with the same security strings. We could use same or different BGP_ID.

<tp>
As per RFC4271, that sets up a second FSM and leads to connection collision and the shutdown of one system, the one being dependent on who initiated the stuck session, so it could fall to the stuck system to act, except it is stuck so it cannot.

I have not seen an analysis of exactly what has gone wrong but cannot help thinking that it could well be a bug, in TCP or BGP, in the peer in which case expecting rational behaviour of it might be expecting too much.

A different approach, leading on from this idea, would be to reserve another TCP port number for BGP use which BGP could use to check out whether or not it can establish a TCP connection with the peer.  If not, then the peer is beyond help.  (TCP ports are a scarce resource but then functioning BGP is rather important to the health of the Internet).  If it can establish a TCP connection, then there are plenty of options available.

Tom Petch

/* When we discussed eBGP peering between VRFs of the same PE it was confirmed that most if not all implementations (at that time) do not check for different BGP_ID when accepting an OPEN). */

If a new session succeeds and we sync routes we are good. Old session can be terminated. No data plane disruption at all.

If not it would be indeed a sign that peer's BGP is not doing that well and perhaps confirmation that in some deployment situations it would be good to remove him from the picture after draining his routes from our peers.

Best,
R.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 28 Apr 2021 16:46:42 +0000

This is just "the other way" to cause graceful restart on the peer.
One way is to send it RST. The other way is to send it a new session.
BGP_ID same or different doesn't matter. It's a collision either way.

As you point out Tom, if the peer does not have GR enabled, then the
new session doesn't work.

We must send it a new session anyway. The preceding RST makes sure that
the new session works.

Regards,
Jakob.

-----Original Message-----
From: tom petch <[email-masked]> 
Sent: Wednesday, April 28, 2021 9:02 AM
To: Robert Raszuk <[email-masked]>; Jakob Heitz (jheitz) <[email-masked]>
Cc: idr@ietf. org <[email-masked]>; Ben Cox <ben=[email-masked]>
Subject: Re: [Idr] draft-spaghetti-idr-bgp-sendholdtimer - Feedback requested

<tp>

Tom Petch

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 28 Apr 2021 18:52:17 +0200

> OPEN a new TCP session to the same address:179 with the same security
> strings. We could use same or different BGP_ID.
>
> <tp>
> As per RFC4271, that sets up a second FSM and leads to connection
> collision and the shutdown of one system, the one being dependent on who
> initiated the stuck session, so it could fall to the stuck system to act,
> except it is stuck so it cannot.
>

As I mentioned in comment brackets last time we investigated this topic the
conclusion was reached that most BGP implementations do not follow RFC in
that respect. We were even considering adding a knob "rfc-compliant" to bgp
:)

But I 100% agree that we should do a bit more "investigation" what's wrong
before resetting the session. Otherwise next session may experience the
same phenomenon.

Ideally it would be best to use bgp diagnostic message to exchange some
parameters before resetting some session (but we are not there yet with
this communication channel). Ideally it should run on different port.

Many thx,
R.

>
> I have not seen an analysis of exactly what has gone wrong but cannot help
> thinking that it could well be a bug, in TCP or BGP, in the peer in which
> case expecting rational behaviour of it might be expecting too much.
>
> A different approach, leading on from this idea, would be to reserve
> another TCP port number for BGP use which BGP could use to check out
> whether or not it can establish a TCP connection with the peer.  If not,
> then the peer is beyond help.  (TCP ports are a scarce resource but then
> functioning BGP is rather important to the health of the Internet).  If it
> can establish a TCP connection, then there are plenty of options available.
>
> Tom Petch
>

> If not it would be indeed a sign that peer's BGP is not doing that well
> and perhaps confirmation that in some deployment situations it would be
> good to remove him from the picture after draining his routes from our
> peers.
>
> Best,
> R.
>

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 30 Apr 2021 02:12:40 +0000

Another consideration.
A BGP speaker may be doing certain maintenance operations
while still in service. Not all maintenance operations
require a speaker to be taken out of service, but they
may require certain operations to be quiesced for several
minutes.

Regards,
Jakob.