Subject: [Idr] Interaction of TCP Window Size and BGP Keepalive Behaviour

From: "Shakir, Rob" <[email-masked]>
Date: Tue, 6 Jul 2010 08:20:28 +0100

Hi IDR,

We currently have a query around the correct parsing of RFC 4271, when
combining an implementation with mechanisms to avoid control-plane
congestion via means of resetting the TCP session's window size to zero.

The scenario that we are examining is one in which a BGP speaker
experiences control-plane congestion, in order to begin to reduce the
load, the device chooses to indicate to all peers that it does not wish
to receive further packets to process, until the period of congestion is
alleviated. To do this, it sets the TCP receive window (i.e. the peer's
send window) to zero. The relevant sentence in RFC 793 here is the
following:

  "... when the receive window is zero no segments should be
  acceptable except ACK segments."

In this period, there is therefore a requirement for the remote BGP
speaker to send no data, including throttling the KEEPALIVEs that are
being sent on a session. In this case, it would be necessary for the
device experiencing congestion to cease to track the hold timer, and
leave the session established, despite not receiving any BGP packet
within the hold time, since this is what the underlying TCP connection
has signalled for the remote peer to do.

In an example topology:

  rtr-A                   rtr-B
  (congested c-p)         (uncongested c-p)
  send window: >0         send window: 0
  recv window: 0          recv window: >0	

In this case we expect:
 a) rtr-B does not send any BGP packet (KEEPALIVE/UPDATE/NOTIFICATION)
to rtr-A in normal operating circumstances.
 b) rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B. The
session remains established even if no packet is received in the
holdtime.
 c) rtr-A continues to send KEEPALIVE packets to rtr-B.

This behaviour both allows the control plane congestion to be
alleviated, whilst still ensuring some form of liveness detection to
occur. Should an error in the underlying connection between the two
peers occur, we would expect that rtr-B is able to tear down the session
due to the fact that it does not receive a KEEPALIVE from rtr-A within
the hold time.

I believe that the above reasoning is merely a repetition of what
Section 6.5 of RFC 4271 states:

  "If a system does not receive successive KEEPALIVE, UPDATE, and/or
  NOTIFICATION messages within the period specified in the Hold Time
  field of the OPEN message, then the NOTIFICATION message with the
  Hold Timer Expired Error Code is sent and the BGP connection is
  closed."

With this in mind, we appear to be observing some different behaviour,
that does not appear to comply with this standard:

1) rtr-A signals control plane congestion through means of setting the
TCP recv window to 0, but continues to send KEEPALIVEs to rtr-B
2) rtr-B does not send any KEEPALIVE/UPDATE to rtr-A and indicates this
fact through CLI output.
3) After t=HOLDTIME, rtr-B sends NOTIFICATION to rtr-A indicating that
the hold time has expired.

It would appear that the reason that rtr-B sends this NOTIFICATION is
the fact that it has not been able to send a KEEPALIVE within the hold
time. I do not believe that this is valid behaviour, since there is no
mention of such a requirement within RFC 4271.

I'd very much appreciate comments on whether this behaviour should be
expected, and how those implementing a ground-up BGP-4 implementation
should treat this scenario?

In addition, if there are comments around the behaviour of setting the
recv window size to 0 during congestion, these would be greatly
appreciated.

Many thanks in advance for any comments.

Kind regards,
Rob

-- 
Rob Shakir                        <[email-masked]>	   
IP&D Network Designer             Cable&Wireless Worldwide

This e-mail has been scanned for viruses by the Cable & Wireless Worldwide e-mail security system - powered by MessageLabs. For more information on a proactive managed e-mail security service, visit http://www.cw.com/managed-exchange

The information contained in this e-mail is confidential and may also be subject to legal privilege. It is intended only for the recipient(s) named above. If you are not named above as a recipient, you must not read, copy, disclose, forward or otherwise use the information contained in this email. If you have received this e-mail in error, please notify the sender (whose contact details are above) immediately by reply e-mail and delete the message and any attachments without retaining any copies.

Cable and Wireless Worldwide plc 
Registered in England and Wales. Company Number 07029206
Registered office: Liberty House, 76 Hammersmith Road, London W14 8UD, England

---

From: Mitchell Erblich <[email-masked]>
Date: Tue, 6 Jul 2010 01:21:40 -0700

>  acceptable except ACK segments."

Within TCP. Why not just just incrementally add latency to TCP ACKs per flow
thus reducing the flow?

Mitchell Erblich

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 06 Jul 2010 10:37:41 +0200

Hi Rob,

IMHO I think you may have just found a bug in RFC 4271 section 6.5. It 
seems what the text should be saying is this:

"6.5.  Hold Timer Expired Error Handling

    Unless BGP speaker has indicated window size of zero towards it's
    TCP peer if a system does not receive successive KEEPALIVE, UPDATE,
    and/or NOTIFICATION messages within the period specified in the Hold
    Time field of the OPEN message, then the NOTIFICATION message with
    the Hold Timer Expired Error Code is sent and the BGP connection is
    closed."

Now the natural question about some other approaches to the problem 
would be to analyze what are any other cases where TCP session will 
correctly stay up and BGP can not communicate. If window is the only 
such trigger then we are OK with the above amendment (modulo point 
below). If there can be other such triggers I think we should analyze it 
one by one.

In general it is normally taken for granted that BGP has it's own 
keepalive machinery, but this covers both the path liveness (TCP could 
do that too), but also BGP processes liveness itself ?

So while sending window of 0 towards the peer sender blocks himself from 
knowing the BGP process state of the peer which may result in potential 
traffic disruption (TCP/kernel being up and BGP process dead).

To address this case the alternative approach would be to either like 
Mitchell proposed slow down TCP while still keeping bidirectional 
communication or if we really can not accept anything from the peer drop 
the session with NOTIFICATION message indicating the cause and 
indicating the willingness to restart in specified period of time.

Cheers,
R.

>    send window:>0         send window: 0
>    recv window: 0          recv window:>0	
>
> In this case we expect:

---

From: Paul Jakma <[email-masked]>
Date: Tue, 6 Jul 2010 09:38:48 +0100 (BST)

> to rtr-A in normal operating circumstances.

Why do you think this? Or at least, from what level are you 
considering this?

The BGP implementation on B should continue to generate BGP messages. 
The TCP implementation on B should not send them yet though, should 
buffer.

> session remains established even if no packet is received in the

As above, this would be surprising to me. See below.

>  Hold Timer Expired Error Code is sent and the BGP connection is

This justifies B tearing down a session if it does not receive a BGP 
KEEPALIVE. I'm not sure how it justifies that the BGP protocol must 
be able to have special insight and control into what TCP does (i.e. 
BGP being aware that TCP is throttling) - which is what would be 
required for BGP on B to stop generating messages to A, and/or for 
BGP on A to not expect BGP messages from B.

Given that TCP has its own keepalive mechanism (and there are 
standard APIs for enabling it), and given that the BGP designers 
chose to also have further keepalives at the BGP layer, it seems 
clear to me that the BGP designers intended for BGP KEEPALIVE to 
measure liveness from one BGP stack layer to other - /not/ just the 
TCP layer. I.e. it seems clear there is an intentional layering, and 
that BGP does not intend to be defer its liveness tests to TCP.

> It would appear that the reason that rtr-B sends this NOTIFICATION 
> is the fact that it has not been able to send a KEEPALIVE within 
> the hold time. I do not believe that this is valid behaviour, since 
> there is no mention of such a requirement within RFC 4271.

No such requirement needs be explicitly stated in the BGP RFC. It 
would be somewhat unconventional to have a protocol above TCP specify 
itself to depend very intimately on the current internal state of TCP 
- state which I'm not sure TCP implementations even make available to 
their users (how would you do it, using typical sockets APIs, out of 
curiosity?). I.e it would be a layering violation.

> I'd very much appreciate comments on whether this behaviour should 
> be expected, and how those implementing a ground-up BGP-4 
> implementation should treat this scenario?

It seems expected and normal to me. :)

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
Integrity has no need for rules.

---

From: "Shakir, Rob" <[email-masked]>
Date: Tue, 6 Jul 2010 12:05:51 +0100

Hi All,

Thanks for the responses thus far.

> -----Original Message-----
> From: Paul Jakma [mailto:[email-masked]]
> Sent: 06 July 2010 09:39
> To: Shakir, Rob
> Cc: [email-masked]
> Subject: Re: [Idr] Interaction of TCP Window Size and BGP
> Keepalive Behaviour
> 
>> In this case we expect: a) rtr-B does not send any BGP packet
>> (KEEPALIVE/UPDATE/NOTIFICATION) to rtr-A in normal operating 
>> circumstances.
> 

This would appear to me to be an internal process of rtr-B. In this
circumstance I find it useful to consider what rtr-A sees. If rtr-B is
queueing packets, rtr-A should not see them, providing the TCP stack
acts as it should be doing. If the window remains zero for a period
greater than the hold time, then rtr-B will not have any opportunity to
send the packet(s) before the session should be torn down unless rtr-A
ignores the hold timer. If rtr-B sends packets then, as long as rtr-A
complies with RFC 793, they should not ever be received by the BGP
process, since only ACK segments should be valid in this circumstance. 

The interesting point around the above observation is that, in my
opinion, this appears quite harmful behaviour. In the situation where
there is some transport disruption between rtr-A and rtr-B, if the
packets are generated by the BGP process, but queued by the underlying
TCP implementation, then what is the behaviour when it comes to sending
a NOTIFICATION to indicate that the hold time has expired as rtr-A is
not sending rtr-B KEEPALIVEs? I would expect in this scenario, that I do
not see a session teardown after t=HOLDTIME, but rather only when rtr-A
signals a non-zero window, in which case, we observe:

1) A flood of KEEPALIVEs (the required number for the length of time for
which we saw congestion)
2) A NOTIFICATION after this, since there was transport disruption.

If this were the case, then there is a relatively unknown period of time
(assuming that TCP does not deal with this - which is perhaps
unrealistic), for which the prefixes that we are holding in the RIB due
to this session are of unknown validity.

> As above, this would be surprising to me. See below.

I believe this is required behaviour, if one is to utilise a TCP window
size of zero to indicate congestion on a BGP session. Another approach
(where some process has a requirement to change the window size to zero)
would result in the transport requiring one type of behaviour from the
peer, yet BGP requiring an incompatible behaviour. This would of course
assume some form of IPC between the TCP stack and the BGP daemon to
avoid this circumstance.

> KEEPALIVE. I'm not sure how it justifies that the BGP protocol must be

> able to have special insight and control into what TCP does (i.e. BGP 
> being aware that TCP is throttling) - which is what would be required 
> for BGP on B to stop generating messages to A, and/or for BGP on A to 
> not expect BGP messages from B.
> 
> Given that TCP has its own keepalive mechanism (and there are standard

> APIs for enabling it), and given that the BGP designers chose to also 
> have further keepalives at the BGP layer, it seems clear to me that 
> the BGP designers intended for BGP KEEPALIVE to measure liveness from 
> one BGP stack layer to other - /not/ just the TCP layer. I.e. it seems

> clear there is an intentional layering, and that BGP does not intend 
> to be defer its liveness tests to TCP.

This would suggest that then BGP trying to utilise TCP to control
congestion would be some behaviour that goes against this idea. If the
object of KEEPALIVE is to check that the BGP daemon on the other side is
alive, then should it not be the case that we do not use TCP to control
a flow, after all, in a period that we are doing this, then we have
almost required the remote peer not to send data that would indicate
that it's BGP daemon is alive?

I also feel that perhaps I did not articulate myself properly here - one
of the behaviours that I am interested in the validity of is the
inability to send a KEEPALIVE within the hold time resulting in the
session being torn down. I think that the analysis you've presented here
implies that the BGP daemon should not do this, as it has no requirement
to know the state of the TCP session via which it is being transported,
and hence to the knowledge of the BGP daemon, it has been able to
generate KEEPALIVEs successfully. The fact that they have not been
received by the peer is not known to the BGP daemon.

> It seems expected and normal to me. :)

Hmm, I am not sure that I understand you completely here. If rtr-A tore
down the session after t=HOLDTIME, I'd consider this "expected and
normal" (no keepalives received in hold time, therefore NOTIFICATION is
required), but, in this case, I appear to see that rtr-B sends the
NOTIFICATION because it could not send a KEEPALIVE.

Whilst I see your implementation points - my issue here is that there
seems to be an inherent problem in throttling packets via means of TCP,
when there is a timer tracking packets that may be throttled by this. In
order to avoid this, I would suggest that there is some language in RFC
4271 that explicitly prohibits the use of underlying layers for
controlling the flow of BGP packets, and requires an implementions that
desire such message-pacing behaviour to implement internal queueing of
relevant packets.

Apologies if I've missed something obvious in the above. I'm just
unclear as to the justification for what we believe we observe, rather
than either the session remaining established, or rtr-A causing the
session to be torn down.

Many thanks in advance.

Kind regards,
Rob

---

From: Paul Jakma <[email-masked]>
Date: Tue, 6 Jul 2010 14:34:58 +0100 (BST)

On Tue, 6 Jul 2010, Shakir, Rob wrote:

> circumstance I find it useful to consider what rtr-A sees. If rtr-B 
> is queueing packets, rtr-A should not see them, providing the TCP 
> stack acts as it should be doing. If the window remains zero for a 
> period greater than the hold time, then rtr-B will not have any 
> opportunity to send the packet(s) before the session should be torn 
> down unless rtr-A ignores the hold timer.

Right.

> If rtr-B sends packets then, as long as rtr-A complies with RFC 
> 793, they should not ever be received by the BGP process, since 
> only ACK segments should be valid in this circumstance.

> opinion, this appears quite harmful behaviour. In the situation 
> where there is some transport disruption between rtr-A and rtr-B, 
> if the packets are generated by the BGP process, but queued by the 
> underlying TCP implementation, then what is the behaviour when it 
> comes to sending a NOTIFICATION to indicate that the hold time has 
> expired as rtr-A is not sending rtr-B KEEPALIVEs?

You mean that A is not receiving KEEPALIVE from B? A's BGP would 
tear-down, and close down at least its side of the session - sending 
a NOTIFICATION. Prior to that, A could still send messages to B just 
fine, thus B need not reach its hold timeout.

Further, if A closes its TCP connection then B potentially still can 
send its buffered packets to A, as part of shutting down its side of 
the TCP connection.

This is what I would expect anyway.

> I would expect in this scenario, that I do not see a session 
> teardown after t=HOLDTIME, but rather only when rtr-A signals a 
> non-zero window, in which case, we observe:

>From who to what? You're saying that you:

- observe A shutting down the session
- but /expect/ A to stop counting down the hold-timer when its
   receive window is 0

Have I understood it right?

(Note that a BGP peer can tear-down a session and close() it, but TCP 
can still send those messages much later (and the TCP FIN to end)).

TCP and BGP are decoupled from each other though. So I don't see how 
BGP in A could ever reasonably be expected to know what TCPs receive 
window was.

> If this were the case, then there is a relatively unknown period of 
> time (assuming that TCP does not deal with this - which is perhaps 
> unrealistic), for which the prefixes that we are holding in the RIB 
> due to this session are of unknown validity.

How so, can you give a more detailed example?

> I believe this is required behaviour, if one is to utilise a TCP 
> window size of zero to indicate congestion on a BGP session.

Exactly how does a BGP implementation "utilise a TCP window size of 
0", to indicate congestion?

It almost sounds like you are reasoning about BGP as if it and TCP 
are monolithic. However, that's generally not the case, BGP and TCP 
generally would be quite decoupled protocols, operating quite 
independently - other than a few well defined sequence points.

> I also feel that perhaps I did not articulate myself properly here 
> - one of the behaviours that I am interested in the validity of is 
> the inability to send a KEEPALIVE within the hold time resulting in 
> the session being torn down. I think that the analysis you've 
> presented here implies that the BGP daemon should not do this, as 
> it has no requirement to know the state of the TCP session via 
> which it is being transported, and hence to the knowledge of the 
> BGP daemon, it has been able to generate KEEPALIVEs successfully. 
> The fact that they have not been received by the peer is not known 
> to the BGP daemon.

Right. BGP sends and reads stuff, and TCP does or does not deliver it 
within some unknown amount of time. BGP has its own keepalive and 
its own timer by which to judge whether or not, at a *BGP* level, 
communication is working sufficiently.

> Hmm, I am not sure that I understand you completely here. If rtr-A 
> tore down the session after t=HOLDTIME, I'd consider this "expected 
> and normal" (no keepalives received in hold time, therefore 
> NOTIFICATION is required), but, in this case, I appear to see that 
> rtr-B sends the NOTIFICATION because it could not send a KEEPALIVE.

So

> Whilst I see your implementation points - my issue here is that 
> there seems to be an inherent problem in throttling packets via 
> means of TCP, when there is a timer tracking packets that may be 
> throttled by this. In order to avoid this, I would suggest that 
> there is some language in RFC 4271 that explicitly prohibits the 
> use of underlying layers for controlling the flow of BGP packets, 
> and requires an implementions that desire such message-pacing 
> behaviour to implement internal queueing of relevant packets.

I have to say, I don't see the problem. BGP and TCP are not 
synchronised to each other, and trying to do so likely is not 
reasonably achievable.

Further, even if you did achieve this (having A not count down its 
BGP hold-time so long as its receive window was 0 - i.e. exclude that 
time from counting for hold-time), what's the point of it exactly? It 
doesn't seem to achieve anything except complexity and requiring 
fairly non-standard TCP<->userspace hacks.

> unclear as to the justification for what we believe we observe, 
> rather than either the session remaining established, or rtr-A 
> causing the session to be torn down.

Hmm, then it's not 100% clear to me what exactly you see, versus what 
you expect to happen.

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
Every successful person has had failures but repeated failure is no
guarantee of eventual success.

---

From: "Shakir, Rob" <[email-masked]>
Date: Tue, 6 Jul 2010 15:19:26 +0100

Paul,

Thanks for you reply.

I think that the problem that I'm having is that my interpretation of this scenario is that I'm basing some of this argument on what we are observing in our lab, rather than an overview of what _should_ happen. Apologies for this.

>> packets are generated by the BGP process, but queued by the 
> tear-down, and close down at least its side of the session - sending a 
> NOTIFICATION. Prior to that, A could still send messages to B just 

A does not receive KEEPALIVE from B. A tolerates this and does not track hold time.

A sends KEEPALIVE to B at the required interval.

> send its buffered packets to A, as part of shutting down its side of 

Where A stops tracking the hold timer. A does not close the connection to B, based on hold timer. In 'normal operating circumstances', I'd then expect that there is no reason that this session should be terminated.

>> I would expect in this scenario, that I do not see a session teardown 
>> after t=HOLDTIME, but rather only when rtr-A signals a non-zero 
>> window, in which case, we observe:
>> 
>> 1) A flood of KEEPALIVEs (the required number for the length of time 
>> for which we saw congestion) 2) A NOTIFICATION after this, since 
>> there was transport disruption.
> 

In our scenario, we observe that B logs that A has requested a 0 window size.

We also see that A stops tracking the hold timer.

Both of these observations would seem to imply that there nis some communication between the status of the TCP connection, and the status of the BGP session.

The point above is in the case that:

- A's KEEPALIVEs to B are not delivered before the hold timer expires.
- B therefore wishes to send NOTIFICATION.
- B's TCP implementation queues these packets and does not deliver them.

If TCP were to deliver packets after some time period, and when the connection parameters allow (i.e. the window is >0), then we would expect to see these packets ingress to rtr-A only when the window is > 0.
 
> How so, can you give a more detailed example?

With the above in mind:

- B's TCP has not delivered a NOTIFICATION to A.
- A is not tracking a hold time, therefore A does not have a reason to send a NOTIFICATION to B.
- B has sent a NOTIFICATIOIN, and hence will remove all NLRI related to this session from the RIB - session has been closed from B's point of view.
- A does not have any indication that the session has failed, and hence the prefixes remain within the RIB.

Until A either has a TCP failure/closes the TCP session, or is able to receive queued packets from B, these prefixes will remain within the RIB, and are of unknown validity.
 
> 0", to indicate congestion?

My assumption here is probably naïve - and assumes some IPC between the TCP stack and the BGP implementation. I assume that BGP is able to tell TCP to signal on the connection that the window should be zero, in order to stop receiving UPDATEs. I am unaware of any mechanism within BGP that would result in a pause in UPDATEs being delivered. The rationale for this observation is that this is what we appear to observe in the lab.

> It almost sounds like you are reasoning about BGP as if it and TCP are 
> monolithic. However, that's generally not the case, BGP and TCP 

This is almost certainly the error that I am making. The problem I have with this statement, is that we appear to have implementation evidence that there is at least some signalling between the two processes in terms of reacting to the window size during session activity. 

The actual observations that we have are that:

- A does stop tracking the hold time, and this does not appear to be a problem - and this corresponds with periods when the TCP window is set to 0.
- B stops sending KEEPALIVEs during this period (as to whether they are delayed by TCP, I am unsure).
- A does not set a window > 0 for a complete hold time.
- B terminates the session -  due to not being able to send a KEEPALIVE within a hold time.

My primary concern here is that B's behaviour is invalid. There is no RFC 4271 requirement for B to terminate the session due to not being able to transmit a KEEPALIVE.

I hope this is somewhat clearer!

Thanks very much for your input on this matter. I think that if the scenario existed where BGP did not react at all to what the state of the TCP transport is, I agree with your reasoning, and would have observed a result that I expect.

Kind regards,
Rob

---

From: Anton Kapela <[email-masked]>
Date: Tue, 6 Jul 2010 10:30:57 -0400

> Both of these observations would seem to imply that there nis some communication between the status of the TCP connection, and the status of the BGP session.

I too can confirm this behavior indeed does occur on at least some IOS-based BGP implementations. 

I (and countless other operators) have found ample evidence that IOS BGP, in particular, makes use of a 'dedicated' TCP implementation. Assembly and functional analysis of IOS binaries confirms that code is present within memory space used by the "BGP I/O" process for handling many types of TCP-related events.

Interestingly, IOS does provide more than one might normally assume a *nix-style socket API might:

[..]

iss: 1874633331  snduna: 1880189269  sndnxt: 1880189269     sndwnd:  16289
irs: 4161282174  rcvnxt: 4167258695  rcvwnd:      16365  delrcvwnd:     19

SRTT: 300 ms, RTTO: 303 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 652 ms, ACK hold: 200 ms

[..]

Rob, can you share details about the vendor hardware/software that you're working with?

-Tk

---

From: Paul Jakma <[email-masked]>
Date: Tue, 6 Jul 2010 15:47:08 +0100 (BST)

On Tue, 6 Jul 2010, Shakir, Rob wrote:

> I think that the problem that I'm having is that my interpretation 
> of this scenario is that I'm basing some of this argument on what 
> we are observing in our lab, rather than an overview of what 
> _should_ happen. Apologies for this.

No worries. Was slightly hard to tell in previous which behaviour was 
the observed. ;)

> A does not receive KEEPALIVE from B. A tolerates this and does not 

Ah. I guess A can tolerate what it want.

> Where A stops tracking the hold timer. A does not close the 
> connection to B, based on hold timer. In 'normal operating 
> circumstances', I'd then expect that there is no reason that this 
> session should be terminated.

If A deliberately stops its hold-timer while it knows its TCP isn't 
accepting packets, then I'd agree A's BGP shouldn't shut down.

Presuming A is sending BGP KEEPALIVES, and B's BGP is receiving 
those, then I'd not expect B to shut down the session either.

If B's BGP is not receiving KEEPALIVES, then it should shut down 
obviously. A's BGP will eventually be notified that the TCP 
connection went.

> In our scenario, we observe that B logs that A has requested a 0 
> window size. We also see that A stops tracking the hold timer.

I don't think A's BGP behaviour is required by the standards. If it 
is B's BGP which logs that fact, then that's a bit interesting - if 
it is its TCP layer, then that's fine.

> Both of these observations would seem to imply that there nis some 
> communication between the status of the TCP connection, and the 
> status of the BGP session.

> If TCP were to deliver packets after some time period, and when the 
> connection parameters allow (i.e. the window is >0), then we would 
> expect to see these packets ingress to rtr-A only when the window 
> is > 0.

Yes.

> - B has sent a NOTIFICATIOIN, and hence will remove all NLRI
>   related to this session from the RIB - session has been closed  from
>   B's point of view.

> - A does not have any indication that the session has failed, and 
> hence the prefixes remain within the RIB.

> Until A either has a TCP failure/closes the TCP session, or is able 
> to receive queued packets from B, these prefixes will remain within 
> the RIB, and are of unknown validity.

Well, the problem here is that A is not letting B send. Either 
because:

1. A is not reading from its BGP socket for B

2. A has some other method internally to keep its receive window at
    0, even when its BGP process has read all the data.

In either case, it sounds like a problem with A.

> My assumption here is probably naïve - and assumes some IPC between 
> the TCP stack and the BGP implementation. I assume that BGP is able 
> to tell TCP to signal on the connection that the window should be 
> zero, in order to stop receiving UPDATEs. I am unaware of any 
> mechanism within BGP that would result in a pause in UPDATEs being 
> delivered. The rationale for this observation is that this is what 
> we appear to observe in the lab.

Sure. Implementations can do that. They can do what they like.

The question, I thought, is what the standards say. My understanding 
is that the standards do not require anything but normally layered 
TCP for BGP.

> This is almost certainly the error that I am making. The problem I 
> have with this statement, is that we appear to have implementation 
> evidence that there is at least some signalling between the two 
> processes in terms of reacting to the window size during session 
> activity.

The implementation you have in your lab may do whatever it want. 
Other than a few, people here generally can not make factual 
statements about what your unnamed implementations is likely to do.

We can only generally give our opinions about what we think the 
standards say is and is not required. ;)

> - A does stop tracking the hold time, and this does not appear to 
> be a problem - and this corresponds with periods when the TCP 
> window is set to 0.

It's not a problem - however note that this would be vendor specific 
behaviour/extension.

> My primary concern here is that B's behaviour is invalid. There is 
> no RFC 4271 requirement for B to terminate the session due to not 
> being able to transmit a KEEPALIVE.

I would suggest that B is shutting the session down because it is has 
failed to /receive/ a KEEPALIVE from A, as you say this is the case 
above. Which would be perfectly normal behaviour.

Unless of course B itself has further non-standard extensions..

> Thanks very much for your input on this matter. I think that if the 
> scenario existed where BGP did not react at all to what the state 
> of the TCP transport is, I agree with your reasoning, and would 
> have observed a result that I expect.

Sure. Note that I do not think the BGP RFC implies that any of the 
coupling of TCP and BGP is required, further I don't think such 
coupling should be required in any way.

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
Truly simple systems... require infinite testing.
 		-- Norman Augustine

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 10:56:42 -0400

Rob,

I normally don't top post but in this case I think the response is to
the message as a whole rather than one part of it.

When a TCP peer is experiences control-plane congestion is simply is
not able to read off the TCP socket (replace with OS specific
mechanism of your choice if not BSD-like) for a given peer.  It should
still be able to write an occasional keep-alive to that socket.  As
soon as the application does read, it will get whatever is in the
socket buffer and that would satisfy the keepalive requirement.

When the BGP application does not read from the socket and the read
buffer fills, no more TCP ACKs are sent.  That is simply how TCP
works.  Nothing special needs to be done by the application.  On the
other side the write to a BGP socket which is full at the receiver,
will either block (this is bad - shoot programmer and get another one)
or it returns an EWOULDBLOCK or EAGAIN error.  There are numerous OS
machanisms to wake up on any one of multiple read or write inputs, but
the common one in BSD is the select system call.  If there is a
programmer who doesn't get this, they should not be working on BGP
until they get a better grasp on TCP basics.

In the past (ie: nearly 20 years ago) we did have BGP implementations
that were broken enough to declare the other end down on an
EWOULDBLOCK or EAGAIN error but that is just bad coding.

If you have a "query" around this topic the best advice is either
"shoot programmer and get another one" or enlighten the existing
programmer.  I sugest the W. Richard Stevens books (which I've
recommended before, see archives).

TCP basics have been discussed on this list ad nauseum in the past
(circa 2000 when RFC4271 was being advanced) and really should not
have to be discussed here because it is just too basic.  This is
really an implementation question rooted in an limited understanding
of how TCP works and how OS facilities supporting TCP are used.

btw- If a BGP peer is too busy to read its own socket, it MUST NOT
declare the other side down because it hasn't read anything.  A BGP
peer should never be too busy to try to send something in the three
keepalive interval, and if it is too broken (in terms of coding) to be
able to do that, the other end MUST declare that peer to be down.

If you would like to take this off list, it would be very appropriate
to do so and I would be willing to provide some help to whoever has
initiated this "query" (which doesn't sound like is you since CW is
not likely to be writing BGP code).

Curtis

ps - there is no bug here in RFC4271 or RFC793 so I'd prefer we take
this off list.

In message <[email-masked]>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 06 Jul 2010 17:49:53 +0200

Hi Curtis,

Let's consider below hypothetical deployment case:

Imagine router willing to have a unidirectional iBGP session to 
monitoring station such that monitoring station will not send anything 
even at the TCP level towards the router. So operator sets tcp rcvwnd to 
zero towards the monitoring station.

It seems obvious to me that in those cases (no congestion at all 
anywhere) BGP should not expect to receive any keepalives or update 
messages from such peer what essentially means that today's text from 
RFC4271 will not be honored.

TCP continues to work fine (acks are exchanged) and till anything 
happens with TCP session itself or till any of the sides closes the 
connection such state continues.

Isn't this the simplest solution to unidirectional BGP session 
requirement without going into specific monitoring protocols (bmp/mrt) 
direction ? And without disclosing too much of other deployment 
scenarios I think it is quite likely that there could be more then just 
monitoring station application for this.

Cheers,
R.

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 6 Jul 2010 18:16:15 +0200

On Tue, Jul 06, 2010 at 05:49:53PM +0200, Robert Raszuk wrote:
> monitoring station such that monitoring station will not send
> anything even at the TCP level towards the router. So operator sets
> tcp rcvwnd to zero towards the monitoring station.
> 

Then this is not a BGP session. All BGP session are bidirectonal.

> messages from such peer what essentially means that today's text
> from RFC4271 will not be honored.
> 

A receive window of 0 is a congestion. It is a congestion between the TCP
stack and the userland of the box sending out a 0 window. This is how TCP
works and redefining it will not change the fact that a 0 size window
means that the process behind that TCP session is unable to read (in most
of the cases because the process or the OS is too busy with other stuff).

> connection such state continues.

But no data is exchanged. It is for example impossible for the other
system to send out necessary messages. Additionaly it is almost impossible
for the remote userland process to probe if the TCP session is still valid.

> requirement without going into specific monitoring protocols
> (bmp/mrt) direction ? And without disclosing too much of other
> deployment scenarios I think it is quite likely that there could be
> more then just monitoring station application for this.
> 

For monitoring sessions you can raise the hold timer to a very large
value and seriously if that timer still fires on the monitoring system
then I would suggest that it is time to look for better HW or SW.

-- 
:wq Claudio

---

From: Tony Li <[email-masked]>
Date: Tue, 6 Jul 2010 09:22:38 -0700

> I (and countless other operators) have found ample evidence that IOS BGP, in particular, makes use of a 'dedicated' TCP implementation. Assembly and functional analysis of IOS binaries confirms tha…

In fact, this is incorrect.  All of IOS is structured in a single address space, so the fact that TCP is in the same address space is not indicative of anything.  IOS BGP uses the same TCP functionality used elsewhere in the OS, and thus is not 'dedicated'.

Tony

---

From: Tony Li <[email-masked]>
Date: Tue, 6 Jul 2010 09:29:55 -0700

On Jul 6, 2010, at 1:38 AM, Paul Jakma wrote:

> The BGP implementation on B should continue to generate BGP messages. The TCP implementation on B should not send them yet though, should buffer.

From an implementation perspective, this is not practical.  B's BGP must not continue to generate messages significantly past the point where its own socket buffers have filled, or it will simply deplete its own memory with queued updates.

The correct implementation approach is to stop generating messages (both keepalive and updates) when the socket is full.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 13:37:09 -0400

In message <[email-masked]>
> zero towards the monitoring station.

That sort of breaks TCP.  Set it big enough to handle an occasional
TCP segment (ie: maybe 1KB).

You still have to get past the BGP open exchange and then there is no
way to send a CEASE or any session related informative message.

> RFC4271 will not be honored.

I think what you have is a bad idea.  There is no point setting recv
window to zero.

> monitoring station application for this.

Unidirectional BGP would have to be another RFC.

At that point one could argue that "hinting through TCP" is a bad
solution relative to an explicit "send nothing" which could even be
done with ORF.

It should also be fine to pitch everything sent by the other side with
a policy filter to "deny all" as is current practice.  The receiving
end could special case the blanket "deny all" to pitch everything at
the adj-in processing stage and/or have a special config that saves no
adj-rib-in but instead resets the connection if the policy every
changed (which it would not for a monitoring station).

Curtis

---

From: Mitchell Erblich <[email-masked]>
Date: Tue, 6 Jul 2010 11:17:16 -0700

> The correct implementation approach is to stop generating messages (both keepalive and updates) when the socket is full.

Looking at this purely from a memory resource perspective, with respect to a single well known memory allocation
subsystem, if ATOMITICY is needed, then GFP_NOWAIT should be used. It allows a NOWAIT allocations,
but will not consume / deplete the last of the memory.

Before a socket becomes FULL, anticipating it at the RED point and doing some form of equivalent ECN would
almost make sense.

If a socket becomes FULL, then no KEEPALIVES & UPDATES are sent, wouldn't this BGP peer would
still be considered ALIVE for some time and which routing will now be impared until the system has died. It
would almost make ECN type sense at a before the socket full condition, to slow down BGP ?KEEPALIVE? msgs, 
so when it stops generating msgs, the BGP peer would be considered DEAD ASAP.

Mitchell Erblich

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 6 Jul 2010 14:18:44 -0400

If I set holdtime to X, it means if I can not get a message
to my peer in time X, REGARDLESS OF THE REASON, then the
transport layer is broken and a NOTIFICATION is in order.

Upgrade your Avian Carrier. Add QOS. Whatever...

Perhaps there is a use case: Maybe you want to put the
connection "on hold" while doing maintenance without having
to bounce the session.

Wasn't there a draft to renegotiate OPEN messages during
established state? What was it?

--
Jakob Heitz.

---

From: Tony Li <[email-masked]>
Date: Tue, 6 Jul 2010 11:27:46 -0700

> Before a socket becomes FULL, anticipating it at the RED point and doing some form of equivalent ECN would

Seems like it's more trouble than it's worth.  The socket buffers are allocated.  Why not use them?

> still be considered ALIVE for some time and which routing will now be impared until the system has died.

Or, until the congestion is resolved.

> It

Again, when the socket is full, there is no point in generating keepalives at all.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 14:41:13 -0400

In message <[email-masked]>
Tony Li writes:
> > I (and countless other operators) have found ample evidence that IOS
> > BGP, in particular, makes use of a 'dedicated' TCP
> > implementation. Assembly and functional analysis of IOS binaries
> > confirms that code is present within memory space used by the "BGP
> > I/O" process for handling many types of TCP-related events.
>  
>  
> In fact, this is incorrect.  All of IOS is structured in a single
> address space, so the fact that TCP is in the same address space is
> not indicative of anything.  IOS BGP uses the same TCP functionality
> used elsewhere in the OS, and thus is not 'dedicated'.
>  
> Tony

Maybe "dedicated" was meant to mean "not of BSD or Linux vintage".
[and I'll add "but rather an inferior strain".  :) ]

Curtis

ps - BSD vs Linux vs Cisco TCP followup to /dev/null please, as
tempting as it might be.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 15:04:18 -0400

In message <[email-masked]>
> so when it stops generating msgs, the BGP peer would be considered DEAD ASAP.

Mitchell,

When a socket is full, there is a full buffer of data at the receive
side.  The sender doesn't have to send anything.  As soon as the
receiver gets less busy it will see packets already in its memory.

No further TCP data flows until the receiver reads from its socket
buffer and frees up some space in doing so.  At that point the sender
can send something (an update if it has one or a keepalive).

It is quite normal for socket buffers to fill when a big routing
transient happens.

No need for drop or for ECN.  This is just a case where receive side
CPU is temporarily swamped.  There is also no need to change the TCP
receive buffer size.  The sender sends that much outstanding data
without ACK and then waits for the ACKed seq num to begin to catch
up.  The recv buf size remains constant.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 15:12:52 -0400

> transport layer is broken and a NOTIFICATION is in order.

If you are receiving keepalives you should not drop the session.

> established state? What was it?

The way BGP is designed, receiving keepalives indicates that the other
side is alive.  If you can't send updates or keepalives because the
socket buffer is full but you are receiving updates or keepalives then
the other side is alive but very busy.  This is particularly true if
someone has configured fast timers for quick detection and is
operation in a heavy load environment (ie: a provider).

If you want to build product that fails in the lab or in the filed
under heavy load (and then fails in the market), then go ahead and
drop the session.

Curtis

---

From: "FUBEL, BOB (BOB)" <[email-masked]>
Date: Tue, 6 Jul 2010 15:28:26 -0400

If the remote side is too busy to process keepalives then how do you know it can forward traffic ? I think the link should be declared down and traffic rerouted.

Bob Fubel

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Curtis Villamizar
Sent: Tuesday, July 06, 2010 3:13 PM
To: Jakob Heitz
Cc: [email-masked]
Subject: Re: [Idr] Interaction of TCP Window Size and BGP Keepalive Behaviour

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 15:29:19 -0400

> allocated.  Why not use them?

A socket buffer is maybe 64KB.  The path between a pair of routers is
maybe 10 Gb/s.  A link along that path isn't going to notice a 1/2 Mb
burst and do RED or ECN.

How about some simple math.  The delay between two routers is 10 msec.
US coast to coast is 35 msec one way by fiber so these are close
together.  A BGP update can be 4KB.  16 BGP updates fills a 64KB
socket buffer.

A router with 200 peers needs 256 x 64KB or 16MB of TCP receive buffer
memory with the same needed on the send buffer side (still not much
today) so that is why the socket buffers are not huge.  With a 10 msec
RTT, TCP can reach 6.4 MB/sec which is plenty for BGP.  Even 100 msec
RTT, which is across the Atlantic get 640 KB/sec.  For very long delay
paths, operators can increase socket buffer sizes if they need to.

If there was some previous activity that left the TCP window open,
then the 64KB burst can go out all at once and the send side socket is
blocked before the recieve side has even seen a single packet.

If the receive side is not busy it quickly ACKs.  If it is very busy
(maybe a lot of other BGP speakers had sent it updates just prior),
then the 64KB may sit in the recv buffer until the receive side gets
time to read it.

This is BGP and TCP 101.

Tony has it right.  The socket buffers are all that comes into play.
On a provider link the BGP traffic is way down in the noise and would
have no effect on RED or ECN which act on a whole link worth of
traffic or at most a queue.

Curtis

---

From: Tony Li <[email-masked]>
Date: Tue, 6 Jul 2010 12:34:01 -0700

> If the remote side is too busy to process keepalives then how do you know it can forward traffic ? I think the link should be declared down and traffic rerouted.

Well, you could have other evidence of that, such as BFD (thanks to Jakob for a reminder).  And reverse BGP traffic can be used to infer BGP liveness.  TCP window probes can also ascertain that the peer's TCP is still there.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 15:59:49 -0400

In message <[email-masked]>
"FUBEL, BOB (BOB)" writes:
>  
>  
> If the remote side is too busy to process keepalives then how do you
> know it can forward traffic ? I think the link should be declared down
> and traffic rerouted.
>  
> Bob Fubel

The last 15 years of routing experience run contrary to your instinct
on this.

If a router is too busy and looks down, declaring it down causes more
route change which makes more routers busy which may cause them to
look down.  If timers are set too tightly, then a threshhold is
reached where the network just melts down when a big route change
happens and then comes back up.  This is not a desireable behavior.

With good implementations timers can be set low (even down to 1 second
keepalive send, 3 second wait) without melting the network, only if
receiving a keepalive is counted as alive and a blocked write socket
is tolerated.

It is much easier for a busy router to send a keepalive than it is to
empty and process the BGP updates in a socket buffer.  Going through
the whole set of peers and sending one keepalive each second is not
hard even if very busy.

AFAIK avaya doesn't make anything that would have hundreds of BGP
peers, but other vendors do.

Curtis

---

From: "Raut, Devendra Y (Devendra)" <[email-masked]>
Date: Tue, 6 Jul 2010 16:26:54 -0500

In other words, the inability of the Sender to handle back pressure from TCP, because the receiver is busy or the network is congested, is an implementation issue.

It is quite easy to suppress KeepAlives or Updates when send TCP buffer is full AND still have no buffer build-up as one could implement data structures where KeepAlives as well as Same Route Updates can be dampened. I think, as long as the BGP Peer receives KeepAlives, the remote BGP Peer is should be considered up & running.

-Deven 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Curtis Villamizar
Sent: Tuesday, July 06, 2010 1:00 PM
To: FUBEL, BOB (BOB)
Cc: [email-masked]
Subject: Re: [Idr] Interaction of TCP Window Size and BGP Keepalive Behaviour

> know it can forward traffic ? I think the link should be declared down 

The last 15 years of routing experience run contrary to your instinct on this.

If a router is too busy and looks down, declaring it down causes more route change which makes more routers busy which may cause them to look down.  If timers are set too tightly, then a threshhold is reached where the network just melts down when a big route change happens and then comes back up.  This is not a desireable behavior.

With good implementations timers can be set low (even down to 1 second keepalive send, 3 second wait) without melting the network, only if receiving a keepalive is counted as alive and a blocked write socket is tolerated.

It is much easier for a busy router to send a keepalive than it is to empty and process the BGP updates in a socket buffer.  Going through the whole set of peers and sending one keepalive each second is not hard even if very busy.

AFAIK avaya doesn't make anything that would have hundreds of BGP peers, but other vendors do.

Curtis

> In message 
> <[email-masked]
> n.se>
> Jakob Heitz writes:
> >  
> > If I set holdtime to X, it means if I can not get a message to my 
> > peer in time X, REGARDLESS OF THE REASON, then the transport layer 
> > is broken and a NOTIFICATION is in order.
>  
> > Perhaps there is a use case: Maybe you want to put the connection 
> > "on hold" while doing maintenance without having to bounce the 
> > session.
> >  
> > Wasn't there a draft to renegotiate OPEN messages during established 
> > state? What was it?
> >  
> > --
> > Jakob Heitz.
>

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jul 2010 17:34:56 -0400

In message <[email-masked]>
"Raut, Devendra Y (Devendra)" writes:
>  
> In other words, the inability of the Sender to handle back pressure
> from TCP, because the receiver is busy or the network is congested, is
> an implementation issue.

It is an implementation decision.  Certain bad decisions are allowed
by the protocol.  If a compliant implementation which can become
unstable is acceptable to your customers, then either choice is fine.

> It is quite easy to suppress KeepAlives or Updates when send TCP
> buffer is full AND still have no buffer build-up as one could
> implement data structures where KeepAlives as well as Same Route
> Updates can be dampened. I think, as long as the BGP Peer receives
> KeepAlives, the remote BGP Peer is should be considered up & running.

Exactly - almost.  You don't need to buffer the keepalives and there
are more efficient ways to keep track of the backlog such that memory
footprint is reduced and changes back and forth end up sending only
the most recent state for a given NRLI.  Gated for example needed one
bit per peer per route by using a bit mask of peers for which a route
update had been sent and it needed a per AS-Path and other update
attributes circular queue (work list) for routes that had not yet gone
to all peers.  But otherwise exactly.

Curtis

---

From: john heasley <[email-masked]>
Date: Tue, 6 Jul 2010 16:42:42 -0700

> If the remote side is too busy to process keepalives then how do you know it can forward traffic ? I think the link should be declared down and traffic rerouted.

given the separation of control and forwarding planes, just because it
can send or receive BGP updates/keepalives does not mean it can forward
traffic sent to it.

---

From: Mitchell Erblich <[email-masked]>
Date: Tue, 6 Jul 2010 16:56:08 -0700

> can send or receive BGP updates/keepalives does not mean it can forward

I think you are reverse thinking.

The forwarding plane on most of the higher performance 
routers can forward,
even when the control plane is undergoing congestion.

Mitchell Erblich

---

From: john heasley <[email-masked]>
Date: Tue, 6 Jul 2010 17:12:56 -0700

Tue, Jul 06, 2010 at 03:47:08PM +0100, Paul Jakma:
>> Where A stops tracking the hold timer. A does not close the connection 
>> to B, based on hold timer. In 'normal operating circumstances', I'd 
>> then expect that there is no reason that this session should be 
>> terminated.
>

or A just considers that because it can read from the socket, ie: there
is data waiting to be read from the socket buffer - poll(POLLIN), it
should reset/advance the hold timer?

---

From: john heasley <[email-masked]>
Date: Tue, 6 Jul 2010 17:51:17 -0700

> even when the control plane is undergoing congestion.

just because it forwards to the control plane makes no guarantee
about transit.

---

From: Paul Jakma <[email-masked]>
Date: Wed, 7 Jul 2010 09:21:15 +0100 (BST)

On Tue, 6 Jul 2010, Mitchell Erblich wrote:

> The forwarding plane on most of the higher performance routers can 
> forward, even when the control plane is undergoing congestion.

The forwarding plane might forward, but to out-of-date places if the 
control plane is unresponsive.

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
Well, the handwriting is on the floor.
 		-- Joe E. Lewis

---

From: Keyur Patel <[email-masked]>
Date: Wed, 07 Jul 2010 10:18:02 -0700

Rob,

I agree with Paul's comments. IMHO, we dont need to change BGP spec at 
all. Given that BGP has its own Keepalive mechanism, implementations 
adhering to RFC 4271 are expected to generate BGP Keepalives.

Question for you: What was the keepalive interval configured between 
these BGP Peers in question? IMHO, the question we should asking is why 
did a BGP speaker (in your example rtr-B) kept its window closed for an 
entire holdtime interval.

Best Regards,
Keyur

Paul Jakma wrote:
> Why do you think this? Or at least, from what level are you considering 
> this?
> 
> The TCP implementation on B should not send them yet though, should buffer.
> 
> have further keepalives at the BGP layer, it seems clear to me that the 
> BGP designers intended for BGP KEEPALIVE to measure liveness from one 
> BGP stack layer to other - /not/ just the TCP layer. I.e. it seems clear 
> there is an intentional layering, and that BGP does not intend to be 
> defer its liveness tests to TCP.
> 
> No such requirement needs be explicitly stated in the BGP RFC. It would 
> be somewhat unconventional to have a protocol above TCP specify itself 
> to depend very intimately on the current internal state of TCP - state 
> which I'm not sure TCP implementations even make available to their 
> users (how would you do it, using typical sockets APIs, out of

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 08 Jul 2010 08:43:32 -0400

> control plane is unresponsive.

Rewind to mid-1990s:

  Router performance requirements

  [...]

  1.  stability under adverse conditions 
  2.  low loss to stable destinations 
  3.  reasonable response under congestion 

  http://www.faster-light.net/ans-archive/scaling/scaling.html
  [ archived white paper from about 1994 ]

Under conditions of extreme high route churn routes which are stable
should see no drop

At that time drop used to happen in moderate route churn with Cisco
routers of the time that used Cisco's route cache bad idea.

At that time networks would go unstable in conditions of very high
route churn.  This was reproducible in lab conditions but not easily.

The "right behavior" is to keep forwarding with the best information
available.  For routes that are stable there is zero loss.  For routes
that are changing, convergence time will increase as load increases.

In the real world these conditions of extremely high route churn are
transient (characteristic of a very major outage, and much more rare
today).  It is better to endure a brief period of slower convergence.
If routers declare a neighbor down in this situation they contribute
more to the churn and can turn high churn into network meltdown.

An assumption here is that high performance routers have the
forwarding plane and control plane completely decoupled.  The BGP
keepalive mechanism only checks for liveness of the control plane.
The control plane can be live and forwarding broken (for example,
firmware bug) or forwarding can be working and the control plane dead
(graceful restart).

Today only in the most extremely low end products does the CPU
handling the routing protocols have anything to do with forwarding.

We are deep into implementation details and this is not a dicussion
that is relevant to protocols.  It is however one that comes up every
few years.  Perhaps an informational draft would be helpful but there
is none to point to and this topic has always been regarded as
"implementation detail" by the WG.

Curtis

---

From: Paul Jakma <[email-masked]>
Date: Thu, 8 Jul 2010 15:02:00 +0100 (BST)

On Thu, 8 Jul 2010, Curtis Villamizar wrote:

> The "right behavior" is to keep forwarding with the best 
> information available.  For routes that are stable there is zero 
> loss.  For routes that are changing, convergence time will increase 
> as load increases.

There are other strategies developed since, e.g. there is the ordered 
FIB updates stuff from the people in Uni Louvain.

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
CCI Power 6/40: one board, a megabyte of cache, and an attitude...

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 08 Jul 2010 11:55:54 -0400

> FIB updates stuff from the people in Uni Louvain.

Ordered FIB is orthogonal to BGP and some form of ordered FIB (IGP
routes before BGP routes) has been in used for more than 15 years.

Curtis

---

From: Paul Jakma <[email-masked]>
Date: Thu, 8 Jul 2010 17:20:49 +0100 (BST)

Ordered updates also affect order of how BGP might issue updates of 
routes to FIB and peers. A peer might want to optimise for working 
routes, over minimising churn.

I take your point that churn is bad, but BGP already has damping 
mechanisms (and IDR BGP today is perhaps over-damped?). Wrt to 
Keepalive, default hold-times tend to be generous, so if a peer 
really can't respond in that time, do you really want to keep routing 
via it?

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
Every time I lose weight, it finds me again!

---

From: Tony Li <[email-masked]>
Date: Thu, 8 Jul 2010 12:16:09 -0700

On Jul 8, 2010, at 9:20 AM, Paul Jakma wrote:

> I take your point that churn is bad, but BGP already has damping mechanisms (and IDR BGP today is perhaps over-damped?). Wrt to Keepalive, default hold-times tend to be generous, so if a peer really can't respond in that time, do you really want to keep routing via it?

Perhaps.  

Churn is so bad that flapping an entire session (which can, in the right circumstances, ring throughout the DFZ) is not necessarily the right answer.

Let's put it more directly: one of the worst bugs that was introduced in the design of BGP was the notion that we needed to flap a peer anytime there was an error.  At the time, the thinking was that this was necessary to get sufficient attention to correct the problem.  We've been through MANY, MANY examples of why this was a Really Bad Idea and one by one, we've decided that for a large class of errors that we will not flap.  The correct default response to errors that we should always have had was to scream bloody murder via the control plane, but to keep the peer up if at all possible.

This is just another case in point.  

If the holdtime has expired and we're receiving keepalives and our forwarding plane shows liveness (i.e., BFD is happy), then it's pretty clear that the peer is still tenable...  for a while.  

On the other hand, if BFD is down, and we're not receiving keepalives, then our inability to send is probably a data point that suggests that this peer is toast and its in everyone's interest to close the connection ASAP.

So, basically, this boils down to an implementation tradeoff.

At best, we could contemplate an amendment to 4271 that clarified these tradeoffs.

Is this horse dead yet?  ;-)

Regards,
Tony

---

From: john heasley <[email-masked]>
Date: Thu, 8 Jul 2010 12:33:45 -0700

> firmware bug) or forwarding can be working and the control plane dead
	...

The point was that the TCP write buffer being full is not an indicator
that the remote peer is dead (for whatever degree of dead) and as Tony
mentioned, nothing should be inferred from it by BGP.  BGP has it's
keepalive mechanism, those can be augmented by notification from BFD,
layer 2 state, and TCP keepalives.  Those don't need to be reimplemented
in BGP.

---

From: Claudio Jeker <[email-masked]>
Date: Thu, 8 Jul 2010 21:46:00 +0200

On Thu, Jul 08, 2010 at 12:16:09PM -0700, Tony Li wrote:
> Churn is so bad that flapping an entire session (which can, in the right
> circumstances, ring throughout the DFZ) is not necessarily the right
> answer.
> 
> Let's put it more directly: one of the worst bugs that was introduced in
> the design of BGP was the notion that we needed to flap a peer anytime
> there was an error.  At the time, the thinking was that this was
> necessary to get sufficient attention to correct the problem.  We've
> been through MANY, MANY examples of why this was a Really Bad Idea and
> one by one, we've decided that for a large class of errors that we will
> not flap.  The correct default response to errors that we should always
> have had was to scream bloody murder via the control plane, but to keep
> the peer up if at all possible.
> 

I don't think so. Maybe the people building >20k$ routers should invest
more then $5 to end up with a totaly underpowered control plane CPU.

> 
> If the holdtime has expired and we're receiving keepalives and our
> forwarding plane shows liveness (i.e., BFD is happy), then it's pretty
> clear that the peer is still tenable...  for a while.  
> 

If keepalives are received the holdtime should not fire, since the timer is
reset on every received packet. This is why this discussion is more
theoretical then practical. BGP does not really care if it can send or not.
So the send window does not matter that much. Your only hope is that the
other side will read some data so that new updates can be sent soonish.

On the other hand if no packets are received the hold timer MUST fire no
matter if the send window size is 0 or not. The TCP send window size does
not matter for the hold timer at all, it has no valuable information about
the state of the remote side (it is actually a good indication that the
other end is way overloaded and that's why the hold timer fired in the
first place).

> Is this horse dead yet?  ;-)

I hope so.

-- 
:wq Claudio

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 8 Jul 2010 16:16:49 -0400

Maybe or maybe not, but it doesn't invalidate Tony's point.

---

From: Mitchell Erblich <[email-masked]>
Date: Thu, 8 Jul 2010 16:17:43 -0700

> more then $5 to end up with a totaly underpowered control plane CPU.

Can't all recent decent hi-end routers forward without a control plane?

And whose control plane needs throughput that is the summation of
the peak capacities of its interfaces?

Network buffers, queuing, and proper interrupt handling, etc  SHOULD suffice
for temporary peak datagram processing of the control plane.

Lastly, if alternate routes exist who cares if a BGP peer flaps and if their aren't
alternate routes, would't you rather be able to periodicly flap a peer or have
a larger percentage of rexmits and sometimes reach those other routes on a retry
versus just black-holing them?

Mitchell Erblich

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Jul 2010 06:21:27 +0200 (CEST)

Yes.

> the peak capacities of its interfaces?

Never, that wasn't the argument, it's that there are quite a few vendors 
who use low-power PPC processors for control plane that are quite slow 
compared to what's available on the market.

> for temporary peak datagram processing of the control plane.

Processing a full Internet BGP table takes some power, seems hard to 
multithread, so you want a fast CPU for all of this.

Classic is when you have 200 peers at LINX and people have slow CPUs in 
their routers, when things start to go haywire it never converges. People 
end up shutting down all their BGP peers, wait a few hours and bring them 
up again and then hope they're the only one during those 1-2 minutes it 
takes to come up. I guess this is an interaction between not being able to 
send/process keepalives at the same time as tcp queuing is going on, 
because you're completely busy doing distance-vector processing plus 
programming the forwarding plane plus sending updates to your iBGP full 
mesh. That's a lot of peers...

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Shakir, Rob" <[email-masked]>
Date: Fri, 9 Jul 2010 08:25:18 +0100

Hi,

Thanks for all the responses on and off-list, a number have been very
valuable for what we've come across, since we're able to clarify from an
implementation perspective what we think we should see, and compare this
to what we are seeing.

The points relating to why this behaviour would occur, and why the send
window of a peer might be closed for a whole hold time is also very
valid - and something that we're examining. I believe that the recent
mails on this subject provide some insight here - lots of real-life
routing equipment does not ship with a CPU that is akin to what is
available on the market. This tends to result in very high CPU loads in
a number of cases for relatively extended periods of time. 

> BGP does not really care if it can send or not. So the send window 
> does not matter that much.

This is another confirmation that has been really useful - we're working
on addressing an implementation that does not appear to conform to this.

Once again, thanks for the input on this matter.

Kind regards,
Rob

Cable & Wireless Worldwide plc 
Registered in England and Wales. Company Number 07029206
Registered office: Liberty House, 76 Hammersmith Road, London W14 8UD, England

---

From: Claudio Jeker <[email-masked]>
Date: Fri, 9 Jul 2010 14:47:45 +0200

> Can't all recent decent hi-end routers forward without a control plane?
 
Sure but that does not mean that these expensive high end routers should
have a control plane that is so weakly powered that the bgp process is
unable to send out a packet in 90s! Seriously if your control plane is
locked solid for 90 Second (the default hold timer in bgp) then this is
not a decent router.

If the control plane of one of my nexthop routers is locked for such a
long time I prefer to route my traffic around that system. The router is
either under attack or so pegged that I must assume that there will be
packetloss. This is the reason why I don't think this has anything todo
with the flaps caused by error notifications (especially those errors that
are introduced further up the stream). There is a valid point in keeping
sessions alive in those cases but there is no valid case that allows routers
to not send out BGP keepalives and but still considering the session as up.

> the peak capacities of its interfaces?
 
I never said that. I said the control plane should have enough CPU power
to let the bgp process not lock for seconds. The minimal hold timer of bgp
is 3 seconds and the default is 90 seconds. This is a hell of a lot time
for todays CPUs to send out one single packet.

> versus just black-holing them?

I would prefer the black-holing at least then everybody realizes that
something is fubar and either the system is replaced or an alternate link
is provided. Hunting packetloss without having a single indication is no
fun.

-- 
:wq Claudio

---

From: Scott Brim <[email-masked]>
Date: Fri, 09 Jul 2010 08:58:03 -0400

> Rewind to mid-1990s:

Even before that Mike Karels said that when EGP lost a peer the router
should try to forward based on the old info.

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 09 Jul 2010 09:36:56 -0400

Paul,

BGP keepalive defaults are extremely generous but are often configured
a lot lower.  At least AFAIK being in vendor space for a decade as
opposed to being in ops and seeing first hand.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 09 Jul 2010 10:55:07 -0400

> should try to forward based on the old info.

And you think that was a good idea?

Maybe a hold up if there was no other route and if the EGP theory
about routing being a strict hierarchy was valid (which it never was),
then it might have been a good idea.

Curtis

---

From: "Raut, Devendra Y (Devendra)" <[email-masked]>
Date: Tue, 6 Jul 2010 15:56:16 -0500

-Deven

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Curtis Villamizar
Sent: Tuesday, July 06, 2010 1:00 PM
To: FUBEL, BOB (BOB)
Cc: [email-masked]
Subject: Re: [Idr] Interaction of TCP Window Size and BGP Keepalive Behaviour

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 09 Jul 2010 11:20:33 -0400

In message <[email-masked]>
> compared to what's available on the market.

Actually the processors were state of the art for a processor that
could handle the heat at the time of the product introduction.  There
is a big difference between the fastest processor available and the
fastest processor that won't be toast in the high ambient inside a
high end router with all the forwarding cards plugged in and
forwarding.  Some of these routers double as 13KW room heaters.

> mesh. That's a lot of peers...

What router is this?  A response offline might be best.

This sound like bad code.  Any processor should converge over time
(minutes) and not drop sessions if the code is "well written" by circa
1995 standards.

In the NSFNET we had routers that were paging the routing code to
disk.  This was a sure sign we needed to add memory, but unlike other
routers that ran out of memory we didn't crash and we didn't drop BGP
sessions either.  We just converged kind of slow, and slower the
longer we fought with top management to shell out the cash ($17k for a
memory upgrade, but I forgot what the increments were).

Curtis

---

From: Paul Jakma <[email-masked]>
Date: Fri, 9 Jul 2010 17:08:41 +0100 (BST)

> Is this horse dead yet?  ;-)

I don't know, but it's been a very interesting horse.

Thanks for the insights,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
A strong conviction that something must be done is the parent of many
bad measures.
 		-- Daniel Webster

---

From: Tony Li <[email-masked]>
Date: Fri, 9 Jul 2010 11:05:07 -0700

Internet history below, hit 'd' if you're not interested...

> then it might have been a good idea.

If you back track a bit further to the late 80's and very early 90's, when EGP was in use on the ARPAnet, yes, it was a good idea.  In fact, it was the ONLY way to stay connected.

At the time, the ARPAnet core gateways were massively overloaded.  BBN implemented a windowing scheme so that a core that actually had a peer got priority and new peers were ignored until resources became available.  Frequently that could take _days_.  So, the alternative was to continue to use the routes that you had, or drop off the net.  Guess how long it took to make that call?  ;-)

So now you know why some of us were SO happy to get away from EGP.

Tony

p.s. Before Curtis has a cow, I'll point out that the situation with NSFnet was completely different.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 9 Jul 2010 18:46:23 +0000

On Thu, Jul 08, 2010 at 12:33:45PM -0700, john heasley wrote:
> mentioned, nothing should be inferred from it by BGP.

And really, the best you can derive from it is that your BGP information
that you have queued up is waiting to get through.  Most likely, that
information is UPDATEs and the guy who is blocking TCP is forwarding based
on stale information.

BGP's convergence is known to be bad even when the system is allowed to
converge with no transport induced latency. :-)

An important thing that I think that has been glossed over in these
discussions is that TCP is effectively a black box.  Even if BGP can tell
that the window is exactly 0 beyond getting EWOULDBLOCK or similar from the
socket, what does that mean?  The other side could be exercising flow
control (poorly).  The other side's routing software may be fine but low on
resources.  The other side's socket may be attached to a hung process.

We can't know.

At some point we have to decide that having that neighbor out of synch with
your forwarding state is a bad enough thing to do something about it.
Unfortunately, since BGP is stateful there are limited things you can do
about it.  You can't withdraw your reachability toward that peer - it's not
accepting UPDATEs.  There are limited things you could advertise to your
other peers in order to make it avoid the congested router. 

In such circumstances, dropping the session may be the only fast way to try
to get the congested router out of the forwarding path - and maybe not even
then.

-- Jeff

---

From: Renwei Li <[email-masked]>
Date: Fri, 09 Jul 2010 11:46:21 -0700

It could work that way, but only when GR is mutually agreed and supported.

/RL

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Curtis
Villamizar
Sent: Friday, July 09, 2010 7:55 AM
To: Scott Brim
Cc: john heasley; [email-masked]; Paul Jakma
Subject: Re: [Idr] Interaction of TCP Window Size and BGP Keepalive
Behaviour

Curtis

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 22 Jul 2010 19:14:42 -0400

Paul and Tony:

I'm happy to see the horse put to rest.  

If this horse comes back to life at any time regarding the BGP RFC, I will
hope to see discussions on:

1. Why or why not HoldTime of zero (added in RFC 4271);  
2. FSM failures for Establish state FSM (pages 71-74); and 
3. discussions on the impact of Graceful restart (GR).  

Please note that RFC 4271's focus was to capture the range of valid BGP
implementations in the Internet. There is some latitude built into the FSM
system for what the WG considered to be valid reasons (during the RFC 4271
debate). 

If you are interested in the RFC 4271 debate, Andrew Lange wrote up the
issues as of: 
http://datatracker.ietf.org/doc/draft-lange-bgp-issues

Since, I've seen this debate several times - perhaps we really  should make
these notes an Informational RFC.

-------------

I'm also happy to see the horse shot and buried.

Lastly, the draft on BGP open negotiation used dynamic Capabilities (a draft
the IESG determined was dead).  

http://tools.ietf.org/id/draft-hares-bose-dynamic_as.

If you are interested in dynamic capabilities that enables established state
negotiation of OPEN state capabilities, I am will happy to discuss offline
any of these issues.

The GR also is a potential way to change the HoldTimer, but may be much more
intrusive to the application.

Cheers, 

Sue Hares
mail ids: 
[email-masked]
[email-masked] 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul
Jakma
Sent: Friday, July 09, 2010 12:09 PM
To: Tony Li
Cc: [email-masked]
Subject: Re: [Idr] Interaction of TCP Window Size and BGP Keepalive
Behaviour

Thanks for the insights,