Subject: Re: [Int-area] Multi-MTU subnet draft

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 3 Mar 2008 11:27:43 +0100

On 29 feb 2008, at 17:52, Templin, Fred L wrote:

> I'm not looking for the RA to tell an MTU size that will
> work across the tunnel once and for always. I want the RA
> coming from the tunnel far end to tell the tunnel near end
> the MRU the TFE is able to receive w/o respect to the tunnel
> path MTU. That way, the TNE can tell whether it even makes
> sense to admit a large packet into the tunnel since there
> is no point in doing so if the TFE's MRU is too small.

Do you mean MRU in the sense of largest packets that can be  
reassembled? In that case I don't see too much of a problem because  
this can be set sufficiently high and isn't dependent on any network  
properties but just on the capabilities of the endpoint.

>>> 1) Section 1, text beginning: "However, [RFC4821] must
>>> be implemented in every transport protocol...", I'm not
>>> sure I understand/agree with the point. My understanding
>>> of RFC4821 was that incremental deployment was supported.

>> Well, if you have RFC 4821 in your TCP implementation that doesn't  
>> buy
>> you anything for NFS or DNS with EDNS0 over UDP.

> Oh - I thought when you said "every transport protocol"
> you were talkng about every transport protocol on all
> nodes everywhere. It is clear now that you were talking
> about every transport protocol *of the same node*. I'm
> not sure on that point though, because I thought that
> at one point RFC4821 had something to say about MTU
> sharing between transports of the same node. Need to
> check to see if it is still there...

That might be helpful, but it's not a solution because you can't be  
sure that if you have RFC 4821 in TCP but not in UDP, you'll be  
running TCP with large packets before you run UDP with large packets.

[...]

> This isn't a failure case for running or not RFC4821, though;
> this is a failure case for allowing nodes with diverse MTUs
> on links that don't support diverse MTUs. Indeed, this failure
> case has nothing to do with RFC4821.

Well, someone suggested that probing was unnecessary if you have RFC  
4821. In the current version of the draft, it's allowed to substitute  
explicit probing as described in the draft with RFC 4821. But this  
isn't completely bulletproof...

I'll have to think about this some more, maybe it would help to  
specify that since routers can't do RFC 4821 they must stick to safe  
packet sizes in the absense of explicit probing.

>> The idea behind SLOWMTU is twofold: it could be that the slow hosts  
>> are behind switches that don't support the larger MTU, and even if  
>> the slow hosts COULD send large packets, that may be undesirable  
>> because it increases RTTs and jitter.

> I am considering the case of fast links at the edges and
> slow links in the middle. Something like GigE at the edges
> and 100Mbps Ethernet in the middle, i.e., the "dumbell
> configuration" yet again! If the nodes on fast links think
> they can send large packets based on their attached link
> bandwidths, it could be a problem.

Right, but how is the situation where two 1000 Mbps clouds are  
connected through a 100 Mbps link different from the situation where  
two 10 Mbps clouds are connected through a 1 Mbps link? This is what  
the MAXMTU is for.

>>> 3) Section 4.2, perhaps this text is part of what you were
>>> meaning about being careful about node, host, route, etc.
>>> Particularly interesting is the part beginning: "Devices
>>> not acting as IPv6 routers ... MAY send out a router
>>> advertisement ...". I think this is worth considering,
>>> and also worth considering is whether two routers connected
>>> by a virtual link that spans an interdomain routing region
>>> (default-free, perhaps) can mutually send RAs with multi-MTU
>>> options to one another (with Router Lifetime=0, of course).

>> Why would they send out router advertisements? The information about
>> the supported MTUs is available in neighbor solicitations/
>> advertisements, which routers DO send to each other.

> I thought you were saying the NS/NA functions were optional?

Yes. But ND is the right place for this functionality. It doesn't make  
sense to replicate it in another place, either people think it's not  
worth the effort, or they should implement it properly. It's not like  
implementing the ND MTU option is extremely complex. I can see how  
someone would want to forego probing, but just advertising the local  
maximum isn't hard.

> Also routers sending RAs to each other can be used as a means
> to propagate RFC4191 Route Information Options (RIOs).

Routers use routing protocols for this. (Although those don't carry  
MTU information.)

>>> The Ras might also contain RFC4191 RIOs, but no RFC4861 PIOs
>>> and such.

>> Hey, a new neighbor discovery RFC... Managed to miss that.

> No; not at all. Standard ND combined with RFC4191.

The RFC is new, though.  :-)

Iljitsch
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 3 Mar 2008 09:12:55 -0800

Iljitsch, 

>-----Original Message-----
>From: Iljitsch van Beijnum [mailto:[email-masked]] 
>Sent: Monday, March 03, 2008 2:28 AM
>To: Templin, Fred L
>Cc: Internet Area
>Subject: Re: [Int-area] Multi-MTU subnet draft
>

I mean that the MRU should be set to the maximum of
2KB and the connected linkMTU of the ETE. That way, if
the ETE's connected linkMTU is 4KB and the ITE is asked
to forward a 9KB packet, the ITE will drop the packet
and send back a PTB instead of admitting the packet into
the tunnel. The next SEAL draft version will say this.
 
>isn't completely bulletproof...

IMHO, endpoints that are sending large packets but not
doing RFC4821 are most likely sending the large packets
infrequently and setting DF=0 (for IPv4). Either that,
or doing host-based fragmentation to a fragment size
that is assured of making it through to the other end
(e.g., 1280 bytes for IPv6). 

>packet sizes in the absense of explicit probing.

Again, the philosophy adopted by SEAL is "take care of
the smalls, and let the bigs take care of themselves".

>two 10 Mbps clouds are connected through a 1 Mbps link? This is what  

How can the router know there is a slow inner link
in-between fast outer links? Is this strictly an
adminstrative function, where the administrator
must have full knowledge of all links in the network? 

>Routers use routing protocols for this. (Although those don't carry  

This gets down again to what is a router and what is a
host? For example, what would you call a device that
turns on IP forwarding and sends RAs with RFC4191 RIOs,
but does not run a routing protocol (as supported under
systems like Linux)? RFC4191 uses the term "type A/B/C
host", but some might think of it as a router of sorts.
Maybe we need to also talk about "type A/B/C" routers?

>The RFC is new, though.  :-)

Not so new that it hasn't been implemented under widely
deployed OSs, though...

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Mar 2008 10:57:06 +0100

> the tunnel. The next SEAL draft version will say this.

The interesting thing is that you don't need to reassemble the whole  
packet in order to return the too big: for IPv4, once you have the  
first fragment and a fragment that is outside of your reassembly  
buffer, you can return the too big. For IPv6, you'd need to reassemble  
the first 1280 - IP - ICMP bytes so you can return those bytes in the  
too big.

> (e.g., 1280 bytes for IPv6).

I don't see that. RFC 4821 is a per-transport solution, which means  
that the maximum packet size should be chosen per-transport, and you  
still have the issue that if the other side doesn't support it and  
there are no too bigs, you can have black holes. As such, RFC 4821  
isn't completely without risks. But as I write in the draft, if you  
are bitten by PMTUD black holes because you use a larger than normal  
MTU, you can always dial back your MTU.

> must have full knowledge of all links in the network?

Yes. All the values in the router advertisement MultiMTU option are  
set administratively. Only the values in the neighbor discovery MTU  
option are discovered automatically. I'll make this more clear in the  
next iteration of the draft.

> Maybe we need to also talk about "type A/B/C" routers?

Is it worth the effort addressing these cases explicitly?

Iljitsch

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 4 Mar 2008 07:29:11 -0800

Iljitsch, 

>Sent: Tuesday, March 04, 2008 1:57 AM
>To: Templin, Fred L
>Cc: Internet Area

>the first 1280 - IP - ICMP bytes so you can return those bytes in the  

Or, in the case of SEAL, you don't need to reassemble the
whole packet in order to return the fragmentation report,
as said in the document. 

>MTU, you can always dial back your MTU.

I don't think we are seeing this quite the same way. AFAICT,
one of the principle advantages of RFC4821 is that it can be
implemented on one side of the connection only and will work
even if no PTBs are received. Maybe you are thinking of
transports other than TCP, SCTP, etc?

>next iteration of the draft.

If the administrator knows that there are 10Mbps links
interspersed in the otherwise 1Gbps topology, why not
just get rid of the 10Mbps links? 

>Is it worth the effort addressing these cases explicitly?

Not sure what you mean, but the cases do exist.

Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 5 Mar 2008 10:16:26 +0100

On 4 mrt 2008, at 16:29, Templin, Fred L wrote:

>> RFC 4821 is a per-transport solution, which means

> even if no PTBs are received.

You are comparing the situation where there are black holes because  
the ICMP too bigs don't make it to the situation where you have RFC  
4821. The latter situation is obviously better than the former, even  
if one end supports RFC 4821 and the other doesn't, that way you at  
least solve the black holes in one direction.

What I'm comparing is the situation where MTU information is  
distributed through RA and ND options and probing, which SHOULD be  
100% foolproof on a given subnet (although you can still have the  
black holes on longer paths) with the situation where one end of the  
communication implements RFC 4821 with no RA or ND options or at least  
no probing, and the other end doesn't implement RFC 4821 but does  
implement RA/ND options and probing. When looking at it this way, the  
former is superior to the latter. Only when BOTH sides implement RFC  
4821 things get better, but even then only for a subset of all  
transports.

> just get rid of the 10Mbps links?

That would be the optimal solution, yes. But until you can do that,  
you can set MAXMTU low so you don't get trouble because of the large  
packets on the slow link.

> Not sure what you mean, but the cases do exist.

Yes, the situation exists, but does it make sense to implement extra  
logic to make it work better than it does today? The current model for  
handling MTU sizes is discovery, not explicit signaling. You could  
imagine a situation where routing protocols carry MTU information for  
all destinations, but we don't do that. I want to be careful with  
adding more complexity that only has benefits in a small number of  
installations because the feedback I've been getting so far has  
consistently been in the direction of "it has to be simpler".

Iljitsch

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 5 Mar 2008 08:35:11 -0800

Iljitsch, 

>Sent: Wednesday, March 05, 2008 1:16 AM
>To: Templin, Fred L
>Cc: Internet Area

>4821 things get better, but even then only for a subset of all  

I'm afraid we are still disconnecting; path MTU discovery
is a unidirectional function, and needs to be handled
seperately over the paths from A->B and B->A - even if
A and B are neighbors on the same link. RFC4821 makes things
better going from A->B even if only node A implements it.
(The same is true for going from B->A.) About the RA/ND
options, they may only apply for links that occur in the
middle, whereas path MTU concerns end-to-end.

>packets on the slow link.

So, if there *might* be some 10Mbps links in an otherwise
1Gbps topology, set MAXMTU=1500 for all and forget about it? 

>consistently been in the direction of "it has to be simpler".

This I think is starting to spin off in random directions
that have nothing to do with this discussion; to be sure,
no one is talking about adding any extra logic.

Fred