Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

From: Enke Chen <[email-masked]>
Date: Wed, 20 Jan 2021 10:20:47 -0800

Hi, Folks:

Here is an update on the TCP_USER_TIMEOUT option in Linux for the
zero-window case:

1) There is a bug in the code, and a fix has been committed to Linux's
networking git:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=9d9b1ee0b2d1c9e02b2338c4a4b0a062d2d3edac

2) A patch has been committed to Linux's man-page repo to clarify that the
option also covers the case that buffered data remain untransmitted:

https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=1942e41202aa5cc39dd8970ab62cd1b288277753

Thanks.   -- Enke

On Fri, Dec 18, 2020 at 2:32 PM John Scudder <[email-masked]> wrote:

> On Dec 18, 2020, at 1:09 PM, Enke Chen <[email-masked]>
> wrote:
> >
> > No, I am not assuming that packets are getting somewhere. The
> TCP_USER_TIMEOUT would work as long as there is "pending data" (either
> unacked, or locally queued). The data can be from the local BGP Keepalives
> or the TCP_KEEPALIVE.
>
> Apart from the other objections to relying on TCP_USER_TIMEOUT, which I
> think are sufficient, it’s not clear to me that implementations will
> provide the desired semantics. RFC 793 seems like it specifies the right
> semantics (“get this data to the peer within N seconds or close”):
>
>         The timeout, if present, permits the caller to set up a timeout
>         for all data submitted to TCP.  If data is not successfully
>         delivered to the destination within the timeout period, the TCP
>         will abort the connection.  The present global default is five
>         minutes.
>
> However the Linux man page documents different semantics:
>
>        TCP_USER_TIMEOUT (since Linux 2.6.37)
>               This option takes an unsigned int as an argument.  When the
>               value is greater than 0, it specifies the maximum amount of
>               time in milliseconds that transmitted data may remain
>               unacknowledged before TCP will forcibly close the
>               corresponding connection and return ETIMEDOUT to the
>               application.  If the option value is specified as 0, TCP will
>               use the system default.
>
> The important difference being that whereas 793 implies data written to
> the socket, the Linux man page says “transmitted” data, which seems like it
> must mean data TCP has written to the network. These are two very different
> things! If Linux (or another stack) implements what the man page seems to
> say, it’s not useful for our purposes.
>
> —John

---

From: Job Snijders <[email-masked]>
Date: Wed, 20 Jan 2021 20:09:21 +0100

Dear Enke, group, 

> https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=1942e41202aa5cc39dd8970ab62cd1b288277753

Wow, thank you, this is a welcome development!

Two follow-up questions for the group:

1) Should the *default* BGP HoldTimer value be the input to mechanisms
   such as the Linux TCP_USER_TIMEOUT option? (a 'Send Hold Timer') Earlier
   on in the thread Tony Li suggested "For robustness, it makes sense that
   the transmitter also close the connection."
   ref: https://mailarchive.ietf.org/arch/msg/idr/CS7VOx42V76RfGLmXhQqM8DyNjk/

2) Should we request IANA to assign a new "BGP Error Notification Code"
   for the purpose of locally correctly logging the reason the remote
   peer was shutdown, or also use Code 4 (Hold Timer Expired) for this
   case?

John Scudder brought up that in the current version of the FSM this
would be a 'ManualStop', but I'd like to explore the option of an
automated termination of the connection (until IdleTimer expires).

Kind regards,

Job

---

From: Enke Chen <[email-masked]>
Date: Wed, 20 Jan 2021 16:29:10 -0800

Hi, Job:

Thanks for bringing up the stuck BGP issue, which has led to the unearth of
the Linux kernel bug and also the fix.

Here are my comments regarding your follow-up questions:

On #1: It makes sense to me. The value for TCP_USER_TIMEOUT associated with
the BGP session should be made configurable, though.

On #2: My recommendation is to use Code 4 (Hold Timer Expired), and
introduce several subcodes something like:
            1) Hold-timer expiration - Received.
            2) Hold-timer expiration - Sent.
            3) Hold-timer expiration - transport timeout.

Regards,   -- Enke

On Wed, Jan 20, 2021 at 11:09 AM Job Snijders <[email-masked]> wrote:

> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pub_scm_linux_kernel_git_netdev_net.git_commit_-3Fid-3D9d9b1ee0b2d1c9e02b2338c4a4b0a062d2d3edac&d=DwIBAg&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=YwLHSMUV8lCZF0-sG2EqV_NqDMsN_NK2NwDb3tQ6mMc&s=wJGNu2HSsZGTN8hE4F3nm5mHYno1xfnPYEOvXUbWUXU&e=
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pub_scm_docs_man-2Dpages_man-2Dpages.git_commit_-3Fid-3D1942e41202aa5cc39dd8970ab62cd1b288277753&d=DwIBAg&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=YwLHSMUV8lCZF0-sG2EqV_NqDMsN_NK2NwDb3tQ6mMc&s=8x1m3aoMVKYAwmKwGXCPL4vlWRH45CKcHnqwCbLtHnE&e=
> >
> > Thanks.   -- Enke
>
>    ref:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__mailarchive.ietf.org_arch_msg_idr_CS7VOx42V76RfGLmXhQqM8DyNjk_&d=DwIBAg&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=YwLHSMUV8lCZF0-sG2EqV_NqDMsN_NK2NwDb3tQ6mMc&s=RkMmOhnl1QTDLAEAO4UFa8h5mGazaiU2sLSlj9oJIjw&e=
>