Subject: MTU mismatches and routing hangups

From: Tom Taylor <[email-masked]>
Date: Tue, 04 Sep 2012 13:24:30 -0400

PLEASE NOTE THE CROSS-POSTING when responding. Please direct responses 
to the authors and the PIM WG only, so the broader community doesn't get 
bombarded with responses. This message is also being sent to the OSPF 
and ISIS WGs.

draft-lts-pim-hello-mtu-00.txt was inspired by customer reports of 
multicast routing hangups caused by MTU mismatches at the two ends of a 
link. The end with the larger MTU sends a message within its MTU size 
that is discarded at the other end because it is larger than that end's 
MTU size. The proposal in draft-lts-pim-hello-mtu-00.txt was for 
adjacent routers to exchange information on their MTUs on each link, 
then restrict message sizes to the smaller of the MTUs.

Working Group discussion in Vancouver raised the obvious question: do 
MTU mismatches cause problems for other routing protocols? And do any of 
these protocols have a solution to the problem?

I raised the issue of the effect of MTU mismatches on the NANOG list.
There were quite a few responses. Amongst other comments:

- OSPF hangs at a particular point in the state machine and fails to 
come up. Debugging can take a while if you don't think of MTU mismatch 
as a possible cause.

- Cisco has published a note on the subject, to be found at
 
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080093f0d.shtml
(sorry about the wrapping).

- BGP can hang after a link flap when two specific products [identities 
omitted] are at the two ends of the link and the configured MTUs are 
identical, because one end includes the ethernet overhead and the other 
doesn't.

- a good MTU debugging tool can be found at
    http://www.elifulkerson.com/projects/mturoute.php

This message is written to ask the broader routing community what, if 
anything, routing protocols should do about identifying MTU mismatches, 
and beyond that, accommodating them. As requested above, please direct 
responses to the authors and the PIM WG only.

Tom Taylor

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 6 Sep 2012 15:47:38 +0200

On 4 Sep 2012, at 19:24 , Tom Taylor <[email-masked]> wrote:

> - BGP can hang after a link flap when two specific products [identities omitted] are at the two ends of the link and the configured MTUs are identical, because one end includes the ethernet overhead and the other doesn't.

Huh?

BGP runs over TCP and TCP has the maximum segment size option that helps the sender to keep its packets to a size that the receiver can handle. As such there shouldn't be any problems unless the MSS option is incorrectly implemented.

---

From: Tony Li <[email-masked]>
Date: Thu, 6 Sep 2012 08:07:40 -0700

> BGP runs over TCP and TCP has the maximum segment size option that helps the sender to keep its packets to a size that the receiver can handle. As such there shouldn't be any problems unless the MS…

This is generally a symptom of a path MTU implementation problem and is not a real protocol issue.

Regards,
Tony

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 6 Sep 2012 17:29:49 +0200

On 6 Sep 2012, at 16:12 , Tom Taylor <[email-masked]> wrote:

> The original message reporting this condition is at
> http://mailman.nanog.org/pipermail/nanog/2012-August/051374.html

Some people think an ethernet frame/packet is 1500 bytes, others 1514, and a third group 1518. There is no substitute for RTFM. But like I said, the MSS option is supposed to take care of that for TCP.

I have an old draft to allow nodes with different MTUs on the same subnet, I'm interested to hear if anyone thinks this would be useful here:

http://tools.ietf.org/html/draft-van-beijnum-multi-mtu-03

---

From: Jie Dong <[email-masked]>
Date: Mon, 10 Sep 2012 03:31:59 +0000

> This is generally a symptom of a path MTU implementation problem and is not a

Agreed. And my question here is that can the path MTU always be obtained through data plane probing? Or do we need control plane mtu discovery mechanism for some cases, e.g. mpls lsps? 

Some related documents: 

 rfc3988: MTU Signalling Extensions for LDP
 rfc3209 (RSVP-TE), sec 2.6: Path MTU
 draft-zeng-idr-bgp-mtu-extension: MTU ext-community for BGP-4

Regards,
Jie

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Tony Li
> Sent: Thursday, September 06, 2012 11:08 PM
> To: Iljitsch van Beijnum
> Cc: [email-masked]; Tom Taylor
> Subject: Re: MTU mismatches and routing hangups
> 
> 
> omitted] are at the two ends of the link and the configured MTUs are identical,
> because one end includes the ethernet overhead and the other doesn't.
> >
> > Huh?
> >
> > BGP runs over TCP and TCP has the maximum segment size option that helps
> the sender to keep its packets to a size that the receiver can handle. As such
> there shouldn't be any problems unless the MSS option is incorrectly
> implemented.
> 
> 
> 
> Regards,
> Tony
> 
> 
> _______________________________________________
> routing-discussion mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/routing-discussion

---

From: Tony Li <[email-masked]>
Date: Sun, 9 Sep 2012 23:18:45 -0700

???  PMTUD already specifies data plane probing.  Adding control plane infra on it would be a major step backwards.

Tony

---

From: Jie Dong <[email-masked]>
Date: Mon, 10 Sep 2012 06:37:56 +0000

Yes PMTUD defines the data plane probing procedure. What I want to ask is that is the data plane probing applicable in all the cases? 

As quoted in my previous email, control plane MTU discovery mechanisms already exist for lsps, such as rfc3988 for LDP and sec 2.6 of rfc3209 for RSVP-TE. This is probably because the intermediate nodes of the lsps may not know how to send the "Datagram Too Big" message back to the source of the probing packet.

Regards,
Jie

> From: Tony Li [mailto:[email-masked]]
> Sent: Monday, September 10, 2012 2:19 PM
> To: Jie Dong
> Cc: Iljitsch van Beijnum; [email-masked]; Tom Taylor
> >> This is generally a symptom of a path MTU implementation problem and is
> not a
> >> real protocol issue.
> >
> > Agreed. And my question here is that can the path MTU always be obtained
> through data plane probing? Or do we need control plane mtu discovery
> mechanism for some cases, e.g. mpls lsps?
> >
> > Some related documents:
> >
> identical,

---

From: Tony Li <[email-masked]>
Date: Mon, 10 Sep 2012 00:56:19 -0700

> Yes PMTUD defines the data plane probing procedure. What I want to ask is that is the data plane probing applicable in all the cases? 

By definition, data plane probing is the ONLY mechanism that we have that can follow the entire path of a TCP connection.  It is effective even in cases where protocols are not in use, such as when someone is doing static routes.  It is (somewhat) effective through firewalls.

> As quoted in my previous email, control plane MTU discovery mechanisms already exist for lsps, such as rfc3988 for LDP and sec 2.6 of rfc3209 for RSVP-TE. This is probably because the intermediate…

True enough, but BGP doesn't require MPLS.  Nor should it.

Tony

---

From: Jie Dong <[email-masked]>
Date: Mon, 10 Sep 2012 08:39:55 +0000

Hi Tony, 

> True enough, but BGP doesn't require MPLS.  Nor should it.

The BGP extension is defined for RFC3107, in which BGP is used to advertise MPLS labels. 

Regards,
Jie

> Sent: Monday, September 10, 2012 3:56 PM
> To: Jie Dong
> > Yes PMTUD defines the data plane probing procedure. What I want to ask is
> that is the data plane probing applicable in all the cases?
> 
> 
> By definition, data plane probing is the ONLY mechanism that we have that can
> follow the entire path of a TCP connection.  It is effective even in cases where
> protocols are not in use, such as when someone is doing static routes.  It is
> (somewhat) effective through firewalls.
> 
> 
> 
> > As quoted in my previous email, control plane MTU discovery mechanisms
> already exist for lsps, such as rfc3988 for LDP and sec 2.6 of rfc3209 for
> RSVP-TE. This is probably because the intermediate nodes of the lsps may not
> know how to send the "Datagram Too Big" message back to the source of the
> probing packet.
> 
>

---

From: Tony Li <[email-masked]>
Date: Mon, 10 Sep 2012 07:27:50 -0700

> The BGP extension is defined for RFC3107, in which BGP is used to advertise MPLS labels. 

???  Yes, but how does that help?  The point at issue is MTU of the full path, including such things as L2 switches, for BGP in the general case.  

Tony

---

From: Jie Dong <[email-masked]>
Date: Tue, 11 Sep 2012 01:24:34 +0000

Tony, 

Sorry for the confusions caused. I admit that the control plane MTU discovery is not directly related to the issue Tom mentioned. 

The control plane MTU discovery is for MTU discovery of LSPs which could be used to transport both data packets and control messages. As I saw your comments on PMTU implementation, I just want to know your opinions if for LSPs (signaled by LDP, RSVP-TE or BGP), control plane MTU discovery would be helpful.

Regards,
Jie

> Sent: Monday, September 10, 2012 10:28 PM
> To: Jie Dong
> ???  Yes, but how does that help?  The point at issue is MTU of the full path,
> including such things as L2 switches, for BGP in the general case.
> 
> Tony

---

From: Tony Li <[email-masked]>
Date: Mon, 10 Sep 2012 18:43:29 -0700

Hi Jie,

Again, the control plane path is fine for distributing the information that you know.  But it is insufficient because there are always restrictions that do not (and will not) participate in the control plane.  Ergo, data plane processes are always going to be necessary.

What is truly unfortunate is that we didn't get this right back in 4.2 BSD.  Then more stacks would have done the right thing.

Regards,
Tony

---

From: Jie Dong <[email-masked]>
Date: Wed, 12 Sep 2012 12:37:12 +0000

Hi Tony, 

Agree that for end-to-end PMTU discovery, data plane mechanism would be necessary. The control plane mechanisms are not to replace data plane probing, but trying to be complementary.

Control plane mechanisms defined in those RFCs/drafts are to help the MTU discovery of LSPs, so the end-to-end data plane discovery can treat the whole LSP as one-hop link, and the PMTU problem with intermediate LSRs is solved. 

And with control plane mechanism, the ingress can dynamically learn the PMTU without the need of frequently probing.

Best regards,
Jie

> Sent: Tuesday, September 11, 2012 9:43 AM
> To: Jie Dong
> Again, the control plane path is fine for distributing the information that you
> know.  But it is insufficient because there are always restrictions that do not
> (and will not) participate in the control plane.  Ergo, data plane processes are
> always going to be necessary.
> 
> What is truly unfortunate is that we didn't get this right back in 4.2 BSD.  Then
> more stacks would have done the right thing.
> 
> Regards,
> Tony
> 
> 
> > Sorry for the confusions caused. I admit that the control plane MTU discovery
> is not directly related to the issue Tom mentioned.
> >
> > The control plane MTU discovery is for MTU discovery of LSPs which could be
> used to transport both data packets and control messages. As I saw your
> comments on PMTU implementation, I just want to know your opinions if for
> LSPs (signaled by LDP, RSVP-TE or BGP), control plane MTU discovery would be
> helpful.
> >
> > Regards,
> > Jie
> >