Subject: Re: (ngtrans) Re: I-D ACTION:draft-ietf-ngtrans-mech-v2-00.txt

From: [email-masked]
Date: Thu, 08 Aug 2002 17:43:05 +0900

sorry, i commented on ipngwg by mistake.  i'll paste the whole thread
	until now.  it seem to me there are some issues with link MTU of
	tunnel pseudo interface, which needs to be cleared up.

itojun

Subject: IPv6 MTU for tunnel pseudo interfaces
From: [email-masked]
To: [email-masked]
Date: Thu, 08 Aug 2002 14:39:22 +0900

	many of existing implementations abstract IPv6-over-IPv4 tunnels, like
	following, as pseudo interfaces:
	- configured tunnels (RFC2893 section 4) - gif(4) in KAME
	- 6to4 tunnels (RFC3056) - stf(4) in KAME
	- isatap

	none of the above document specify what IPv6 MTU to use for these
	interfaces.  (RFC3056 has a pointer to RFC2893, but RFC2893 has nothing
	about IPv6 MTU)

	the lack of standard IPv6 MTU could lead to interoperability problem.
	consider the following scenario:
	- node A is using MTU X for tunnel interface (X >= 1280)
	- node B is using MTU Y for tunnel interface (Y > X >= 1280)

	now, if B transmits packets larger than X, A may drop it as it is
	larger than MTU (assuming MRU == MTU), causing blackhole.
	also, MTU mismatch could have some impact to PMTUD if A and B are
	routers.

	with configured tunnels, the issue is not that serious as we only need
	two party to agree on what MTU to use (so if operators are
	clever/careful enough, we are okay).  for 6to4 and isatap, the issue
	is serious as there are multiple nodes sharing the same virtual link.

	we actually have experienced interoperability problem between KAME and
	JunOS for configured tunnels, as KAME is using 1280 as MTU and JunOS
	is using infinity (so PMTUD does not occur).

	therefore, I propose:
	- to specify default IPv6 MTU of RFC2893 tunnel interface be 1280,
	  in RFC2893bis (so, 6to4 MTU will become 1280).  of course operators
	  can override if they wish to, specifically for configured tunnel case.
	- add reference from isatap document to 2893bis to follow it.

	any comments?

itojun
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

Subject: Re: IPv6 MTU for tunnel pseudo interfaces
From: Pekka Savola <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Thu, 8 Aug 2002 09:24:22 +0300 (EEST)

On Thu, 8 Aug 2002 [email-masked] wrote:
> 	the lack of standard IPv6 MTU could lead to interoperability problem.
> 	consider the following scenario:
> 	- node A is using MTU X for tunnel interface (X >= 1280)
> 	- node B is using MTU Y for tunnel interface (Y > X >= 1280)
> 
> 	now, if B transmits packets larger than X, A may drop it as it is
> 	larger than MTU (assuming MRU == MTU), causing blackhole.
> 	also, MTU mismatch could have some impact to PMTUD if A and B are
> 	routers.

MRU == MTU seems an invalid assumption (in practise).

Nodes must be able to receive packets with at least 1500 bytes (I tried 
looking up where this was stated but couldn't remember where).

But what is the problem here?  If node A drops the packet, it sends packet 
too big message back, and the sources adjust?
 
> 	with configured tunnels, the issue is not that serious as we only need
> 	two party to agree on what MTU to use (so if operators are
> 	clever/careful enough, we are okay).  for 6to4 and isatap, the issue
> 	is serious as there are multiple nodes sharing the same virtual link.
> 
> 	we actually have experienced interoperability problem between KAME and
> 	JunOS for configured tunnels, as KAME is using 1280 as MTU and JunOS
> 	is using infinity (so PMTUD does not occur).

Every node must be capable of at least being a recipient to PTMUD (that 
is, adjusting the sending MTU based on packet too big messages).  Else we 
have way too many problems.
 
> 	therefore, I propose:
> 	- to specify default IPv6 MTU of RFC2893 tunnel interface be 1280,
> 	  in RFC2893bis (so, 6to4 MTU will become 1280).  of course operators
> 	  can override if they wish to, specifically for configured tunnel case.
> 	- add reference from isatap document to 2893bis to follow it.

Operationally, the 1280 default for configured tunnels seems very 
unoptimal.  Source nodes which use a larger MTU, like 1500, need to 
initiate PMTUD if they use links with tunnels; not good.

For automatic tunnels (incl. 6to4 etc.) 1280 might be a safe choice.

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

Subject: Re: IPv6 MTU for tunnel pseudo interfaces
From: [email-masked]
To: Pekka Savola <[email-masked]>
Cc: [email-masked]
Date: Thu, 08 Aug 2002 15:28:41 +0900

>MRU == MTU seems an invalid assumption (in practise).
>Nodes must be able to receive packets with at least 1500 bytes (I tried 
>looking up where this was stated but couldn't remember where).
>But what is the problem here?  If node A drops the packet, it sends packet 
>too big message back, and the sources adjust?

	i know of no nodes that transmit packet too big messgages, when
	incoming packets exceed MRU.  too big messages are generated
	when outgoing packets exceed MTU.

itojun
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

Subject: Re: IPv6 MTU for tunnel pseudo interfaces
From: [email-masked]
To: Pekka Savola <[email-masked]>, [email-masked]
Date: Thu, 08 Aug 2002 15:33:58 +0900

>>Nodes must be able to receive packets with at least 1500 bytes (I tried 
>>looking up where this was stated but couldn't remember where).

	the above statement is about the packet size after reassembly,
	not about the link MRU.

itojun
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

Subject: Re: IPv6 MTU for tunnel pseudo interfaces
From: [email-masked]
To: Pekka Savola <[email-masked]>
Cc: [email-masked]
Date: Thu, 08 Aug 2002 15:39:12 +0900

>> 	we actually have experienced interoperability problem between KAME and
>> 	JunOS for configured tunnels, as KAME is using 1280 as MTU and JunOS
>> 	is using infinity (so PMTUD does not occur).
>Every node must be capable of at least being a recipient to PTMUD (that 
>is, adjusting the sending MTU based on packet too big messages).  Else we 
>have way too many problems.

	let me annotate.  we experienced troubles under the following
	configuration.  MTU mismatch occured between routers, and caused
	troubles to the end nodes that are served by these routers.

end node --- KAME ============== Juniper --- end node
		  1280     infty

itoju
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

Subject: Re: IPv6 MTU for tunnel pseudo interfaces
From: Pekka Savola <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Thu, 8 Aug 2002 09:41:37 +0300 (EEST)

On Thu, 8 Aug 2002 [email-masked] wrote:
> >MRU == MTU seems an invalid assumption (in practise).
> >Nodes must be able to receive packets with at least 1500 bytes (I tried 
> >looking up where this was stated but couldn't remember where).
> >But what is the problem here?  If node A drops the packet, it sends packet 
> >too big message back, and the sources adjust?
> 
> 	i know of no nodes that transmit packet too big messgages, when
> 	incoming packets exceed MRU.  

No nodes do; but I know of no nodes that drop packets they're able to 
receive based on some configured maximum value either.

> too big messages are generated
> 	when outgoing packets exceed MTU.

But if a node X is able to send packet on a medium (in case of tunneling,
nothing physical though), and node Y is able to receive it, why should it
check MRU?

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

Subject: Re: IPv6 MTU for tunnel pseudo interfaces
From: [email-masked]
To: Pekka Savola <[email-masked]>
Cc: [email-masked]
Date: Thu, 08 Aug 2002 15:49:49 +0900

>> 	i know of no nodes that transmit packet too big messgages, when
>> 	incoming packets exceed MRU.  
>No nodes do; but I know of no nodes that drop packets they're able to 
>receive based on some configured maximum value either.
>> too big messages are generated
>> 	when outgoing packets exceed MTU.
>But if a node X is able to send packet on a medium (in case of tunneling,
>nothing physical though), and node Y is able to receive it, why should it
>check MRU?

	since there's no standard, some implementation can be picky and
	assume MRU == MTU.  maybe RFC2893 should talk more about MRU and MTU.

itojun
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

Subject: Re: IPv6 MTU for tunnel pseudo interfaces
From: Pekka Savola <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Thu, 8 Aug 2002 10:01:04 +0300 (EEST)

On Thu, 8 Aug 2002 [email-masked] wrote:
> >> 	i know of no nodes that transmit packet too big messgages, when
> >> 	incoming packets exceed MRU.  
> >No nodes do; but I know of no nodes that drop packets they're able to 
> >receive based on some configured maximum value either.
> >> too big messages are generated
> >> 	when outgoing packets exceed MTU.
> >But if a node X is able to send packet on a medium (in case of tunneling,
> >nothing physical though), and node Y is able to receive it, why should it
> >check MRU?
> 
> 	since there's no standard, some implementation can be picky and
> 	assume MRU == MTU.  maybe RFC2893 should talk more about MRU and MTU.

I've configured Ethernet MTU to be 1500 on the sender side and 1280 on the
recipient side (MRU is generally unconfigurable); it works without
problems as far as I've noticed.

These issues are not specific to tunnels as far as I can see.

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: "Fred L. Templin" <[email-masked]>
Date: Thu, 08 Aug 2002 07:44:42 -0700

Itojun,

> 	- add reference from isatap document to 2893bis to follow it.

ISATAP-04 includes the following text which directly references
the sections of RFC 2893 pertaining to IPv6-in-IPv4 encapsulation:

+ 4.  Transmission of IPv6 Packets on ISATAP Links
+
+    ISATAP links transmit IPv6 packets via automatic tunneling using the
+    site's IPv4 infrastructure as an NBMA link layer. Automatic tunneling
+    for ISATAP uses the same mechanisms specified in [MECH,3.1-3.6],
+    i.e., IPv6 packets are automatically encapsulated in IPv4 using
+    'ip-protocol-41' as the payload type number.

[MECH,3.2] addresses the subject of MTU, and seemed to be the
"best current practice" at the time of our writing, so it would
have been a dubious choice for us to give a detailed discussion
on MTU in the ISATAP draft. We have been following the RFC 2893bis
discussions, and fully intend to update the reference.

Fred
[email-masked]

---

From: [email-masked]
Date: Thu, 08 Aug 2002 23:59:41 +0900

>the sections of RFC 2893 pertaining to IPv6-in-IPv4 encapsulation:

	i see,

>discussions, and fully intend to update the reference.

	[MECH,3.2] does not suggest the default IPv6 MTU value for tunnel
	interfaces, that's the problem.

itojun

---

From: Pekka Savola <[email-masked]>
Date: Thu, 8 Aug 2002 18:41:45 +0300 (EEST)

> 	interfaces, that's the problem.

No, that's not the problem.  The problem is specifying and identifying the 
correct behaviour.

If we specify the default IPv6 MTU value, we will _still_ have nodes which 
don't use that, and will _still_ have problems you're saying.

The only thing the default value does is set the amount of optimization 
per default (how many people have to do PMTUD), AFAICS.

---

From: [email-masked]
Date: Fri, 09 Aug 2002 00:45:03 +0900

>No, that's not the problem.  The problem is specifying and identifying the 

	if you say so, for 6to4 and isatap, you are suggesting that it is sane
	behavior for nodes to use arbitrary MTU value, on a shared (virtual)
	link.  i disagree.

itojun

---

From: Pekka Savola <[email-masked]>
Date: Thu, 8 Aug 2002 18:48:26 +0300 (EEST)

> 	behavior for nodes to use arbitrary MTU value, on a shared (virtual)

Think about existing deployment.  Is it ok to break them too (esp. 6to4) ?

But I don't disagree, shared link should try to achieve same MTU; but how 
that is measured is another thing.

The problem is with (in a smaller case) is with confg'd tunneling too.

---

From: [email-masked]
Date: Fri, 09 Aug 2002 00:57:54 +0900

>Think about existing deployment.  Is it ok to break them too (esp. 6to4) ?

	i have never suggest mandating MTU == MRU, i was just pointing out
	there could be such an implementation.  where did you get that
	impression?

>The problem is with (in a smaller case) is with confg'd tunneling too.

	so we need to talk about inbound processing (MRU) explicitly.  how
	about this in mech-v2?

	  The IPv6 MTU for tunnel interface is 1280 octets (so after
	  encapsulation, IPv4 packet will be 1300 octets long).

	  Due to the way past drafts were speicfied, incoming IPv6 packet may
	  be of size larger than IPv6 MTU.  Implementations SHOULD accept
	  these packets.

	  (configured tunnels)
	  Operators MAY override IPv6 MTU for tunnel interface, however, in
	  such case, both end of the tunnel SHOULD use the same IPv6 MTU value.

itojun

---

From: Pekka Savola <[email-masked]>
Date: Thu, 8 Aug 2002 19:28:53 +0300 (EEST)

> 	there could be such an implementation.  where did you get that

I don't think you tried to mandate it; you just mentioned that's something 
you had to do to (for some reason) avoid blackholes.

> 	so we need to talk about inbound processing (MRU) explicitly.  how

MRU processing is a more general (potential) problem, so it should 
primarily be solved elsewhere.  Especially:

 1) identify whether this is really a problem (how do the specs say about 
this)
 2) after that, see if we need to fix it.

> 	  such case, both end of the tunnel SHOULD use the same IPv6 MTU value.

I don't think this is a good approach, especially for configured tunnels.  
They could use much higher values too.  No use to have the users initiate
PMTUD if MTU could have been e.g. 1500 or 1480.

For 6to4, ISATAP and similar virtual link cases, it may be reasonable to 
set the default MTU to 1280 or some administered value, to avoid a sending 
node having to perform PMTUD so often.  

Whether this is reasonable depends on how many nodes are capable of
processing higher than MTU 1280, and which value would generally be best.  
1500 would be the best, naturally, except for the fact that it would lead
to, in most L2 mediums, fragmented IPv4 packets.  So, something smaller is
ok.  It might very well be just that 1280.

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 9 Aug 2002 14:24:11 +0200 (CEST)

Since this thread was moved from ipngwg, here is the suggested text I
sent reproduced on this list:

How about:
	A node implementing RFC 2893 MUST be able to reassemble IPv4 packets
	of a size up to and including 65353 octets and MUST NOT place
	any limits on the size of the received encapsulated packets.
	In particular, it MUST NOT assume that the maximum size of
	received packets has anything to do with the MTU for the tunnel.

Of course, it might also make sense to add some SHOULD (or MUST?) to
section 3.2 in RFC 2893 to make IPv4 MTU discovery across the tunnel
seem less optional than what the current text says.

 Erik

---

From: [email-masked]
Date: Sat, 10 Aug 2002 11:52:24 +0900

>	received packets has anything to do with the MTU for the tunnel.

	the above is extreme and put unnecessary hurdles to IPv6 deployment.
	think about who will be terminatnig RFC2893 tunnels - they include
	small devices like DSL routers at home.  they won't be able to
	reassemble up to 64k.  from IPv4 host requirement (and RFC791 3.2)
	minimum requirement for IPv4 reassembly is 576 octets.

	i'd suggest far lower number as a requirement, like 1500.

itojun

---

From: [email-masked]
Date: Sat, 10 Aug 2002 11:56:43 +0900

i think we came from the opposite side of the world.

	Erik - monitor IPv4 PMTUD changes, reflect it to IPv6 tunnel link MTU
		(tunnel link MTU is dynamic).
		pros: can utilize large MTU links efficiently
		cons: dynamic MTU change is uncommon and not proven to work,
			state management needed in tunnel endpoint
	itojun - IPv4 PMTUD changes has no impact to IPv6 tunnel link MTU
		(no DF bit set), IPv6 tunnel link MTU is static
		pros: simple and robust, no state management in tunnel endpoint
			lengthy description on "ICMPv4 to ICMPv6 translation"
			is not neeed.
		cons: can be inefficient in terms of the use of MTU size

	from "keep it simple and stupid" principle I think my approach is
	better.

itojun

---

From: Pekka Savola <[email-masked]>
Date: Sat, 10 Aug 2002 08:11:31 +0300 (EEST)

> 	minimum requirement for IPv4 reassembly is 576 octets.

Minimum MTU was also 68 bytes... :-)
 
> 	i'd suggest far lower number as a requirement, like 1500.

What will happen when a node receives larger than that?

---

From: Pekka Savola <[email-masked]>
Date: Sat, 10 Aug 2002 08:13:45 +0300 (EEST)

> 			lengthy description on "ICMPv4 to ICMPv6 translation"

There is a considerable installed base which require this anyway, we just 
cannot get rid of it AFAICS.

---

From: [email-masked]
Date: Sat, 10 Aug 2002 14:22:05 +0900

>cannot get rid of it AFAICS.

	what is "this" in the above text?

itojun

---

From: Pekka Savola <[email-masked]>
Date: Sat, 10 Aug 2002 08:26:51 +0300 (EEST)

> 	what is "this" in the above text?

State management, ICMP translation.

---

From: [email-masked]
Date: Sat, 10 Aug 2002 14:28:50 +0900

>State management, ICMP translation.

	i agree there are installed based which performs these functions,
	but i don't think these items are *required*.  even if one end does it
	and another end doesn't, we won't have interoperability problem.
	(or could you show me an example where interop problem happens?)

itojun

---

From: Pekka Savola <[email-masked]>
Date: Sat, 10 Aug 2002 18:48:15 +0300 (EEST)

> 	(or could you show me an example where interop problem happens?)

Can you define the new rules for fragmentation, ICMP message generation,
translation etc. so we know what to compare against?

---

From: "Fred L. Templin" <[email-masked]>
Date: Thu, 08 Aug 2002 07:44:42 -0700

Itojun,

Fred
[email-masked]

---

From: [email-masked]
Date: Sun, 11 Aug 2002 09:59:08 +0900

>translation etc. so we know what to compare against?

	DF bit - IPv4 DF bit SHOULD be set to 0.
	fragmentation - when encapsulating IPv6 packet into IPv4 packet,
		the encapsulating end may need to fragment it to fit to
		IPv4 path MTU (if smaller than IPv6 tunnel link MTU + 20).
		fragment it as necessary.
		if IPv4 link MTU is large enough, send the encapsulated
		packet as is and let the intermediate IPv4 routers fragment it.
	ICMPv4 -> ICMPv6 meesage translation - none.
	ICMPv6 too big generation - normal rule.  if the packet being forwarded
		does not fit IPv6 tunnel MTU, the router will generate ICMPv6
		too big.

	is it enough?

itojun

---

From: "Fred L. Templin" <[email-masked]>
Date: Thu, 08 Aug 2002 07:44:42 -0700

Itojun,

Fred
[email-masked]

---

From: Pekka Savola <[email-masked]>
Date: Mon, 12 Aug 2002 09:08:04 +0300 (EEST)

> 	DF bit - IPv4 DF bit SHOULD be set to 0.

In this case, if DF is _not_ set to 0 (something that would happen
certainly), there would have to be ICMP message translation at least for
the nodes that don't always set DF to 0.

> 		fragment it as necessary.

You meant IPv4 link MTU, not path MTU above, I assume; there was the 
requirement of no state (including path MTU).

> 		packet as is and let the intermediate IPv4 routers fragment it.

Ok.  You're placing the burden on routers now.  This isn't a big deal with 
MTU of 1280+20, but if something else would be chosen to be a default 
(e.g. 1480), then it could be.

Also, if higher, the decapsulating node would have to perform 
defragmentation -- more work.

> 	ICMPv4 -> ICMPv6 meesage translation - none.

Required (such as it is now) on nodes that may set DF to 1.

The current draft does not say explicitly where to use ICMP message
translation, it only lists packet too big.  But is this something that is 
being done for other packets too (like ICMP unreachables).

A valid problem is that ICMPv4 translation is unreliable as some (most?) 
routers return only 8 bytes; that's why e.g. Linux does not enable 
relaying even though it's implemented (except for a few message types).

> 		does not fit IPv6 tunnel MTU, the router will generate ICMPv6

Ok.

> 	is it enough?

Mostly yes, thanks.

What does a node do if it receives a packet which is bigger than the 
default value?  E.g. the limit in the node is 1280 bytes, but it gets a 
datagram of 1480 bytes.

My take on the pros and cons of this approach:

 + more simplicity w/ no ICMP translation (unreliable as it is)
 + zero fluctuations if IPv4 PTMU changes
 + possibly simpler receive-side implementation

 - 1280 MTU is used even though a bigger one could be used instead in most 
cases
 - routers having to fragment packets if problems with MTU
 - in above case end nodes (with low memory) having to defrag
 - tunneling decreases PMTU of native connections from ~1500 to 1280, thus 
usually forcing PMTUD in end nodes

---

From: Jun-ichiro itojun Hagino <[email-masked]>
Date: Mon, 12 Aug 2002 15:13:15 +0900

>the nodes that don't always set DF to 0.

	then - IPv4 DF bit MUST be set to 0 if the encapsulating node does not
	maintain IPv4 PMTU.  IPv4 DF bit SHOULD be set to 0 otherwise.

>requirement of no state (including path MTU).

	maybe i had *BSD implementation in mind all the time.

	IPv4 PMTU information is maintained in routing table, per-destination
	manner.  therefore, even when tunnel interface itself doesn't manage
	IPv4 PMTU, we automatically consult IPv4 PMTU (if available).

>defragmentation -- more work.

	this is also the case for Erik's version. (if IPv6 tunnel MTU > IPv4
	PMTU).

itojun

---

From: Pekka Savola <[email-masked]>
Date: Mon, 12 Aug 2002 09:50:24 +0300 (EEST)

> 	this is also the case for Erik's version. (if IPv6 tunnel MTU > IPv4

Not so, I think; consider the case with:

IPv6 MTU = 1400, IPv4 PMTU = 1350.
  ==> ICMPv6 pkt too big error is generated with MTU=1330, and IPv6 PMTU 
is adjusted; no fragmentation.

fragmentation would only occur in case IPv4 PMTU is less than 1300 (ie. 
when it's unavoidable).

---

From: Vladislav Yasevich <[email-masked]>
Date: Tue, 13 Aug 2002 16:02:35 -0400

Itojun

I think these are 2 sides of the standard IPv6 approach.

I view it more like, "Does the tunnel pseudo-interface support
Path MTU Discovery?"  If it does not, it follows normal IPv6 interface
requirements and sets its MTU to 1280 (IPv6 minimum link mtu).

If the interface supports PMTU Discovery, it can dynamically change
its MTU as discribed in the draft.

The tunnel mechanisms described in MECH are fine because they are
point-to-point tunnels.

On the other hand, point-to-multipoint tunnels like 6to4 and ISATAP
might have some problems with PMTUD.

-vlad

-- 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Vladislav Yasevich	       Tru64 UNIX - IPv6 Project Lead
Hewlett Packard 	       Tel: (603) 884-1079
Nashua, NH 03062	       ZKO3-3/T07

---

From: Pekka Savola <[email-masked]>
Date: Mon, 12 Aug 2002 09:50:24 +0300 (EEST)

---

From: Erik Nordmark <[email-masked]>
Date: Sun, 18 Aug 2002 11:18:41 +0200 (CEST)

> 	from "keep it simple and stupid" principle I think my approach is

You are forgetting two cons:
 - the (e2e) retransmission unit is not the same as the loss unit
   due to IPv4 fragmentation inside the tunnel.
   The performance implications of this issue is what lead to the need
   for path MTU discovery in the first place.
   Sure, you could argue that 1500 bytes would statistically never cause
   IPv4 fragmentation, but that is different than simply discounting the
   need for PMTU discovery i general.

   I think this issue is counter to your claim that the approach is robust.

 - When IPv4 fragmentation occurs the decapsulating nodes be doing IPv4
   reassembly. The decapsulating nodes are often going to be routers, and 
   AFAIK they are not optimized for reassembly.

So the obvious extension of your KISS approach is
to not do any PMTU discovery at all, since it is simpler, deosn't require MTU
state in the senders, etc. I don't think that is your argument, but
I fail to understand why you think tunnels are different than other
traffic so that PMTUD is good for other traffic and bad for tunnels.
Can you enlighten me on that point?

As I said before, I think the same motivation and tradeoffs for PMTU discovery
apply for traffic whether or not it is encapsulated in multiple IP
headers or only contains one IP header.

  Erik

---

From: [email-masked]
Date: Mon, 19 Aug 2002 10:50:35 +0900

>Can you enlighten me on that point?

	here are a couple of fundamental design requirements i would like
	to keep:
	- link MTU has to be static (rather than dynamically changing) for any
	  kind of link, it applies to IPv6 tunnel link too.  it is necessary
	  for PMTUD to work effectively (from previous message you don't seem
	  to agree)
	- intermediate routers should maintain less state, if any.  by
	  mandating maintainance of PMTU value for IPv4 tunnel portion we
	  do not meet this fundamental design of the Internet.
	- behavior of intermediate routers has to be kept as simple as possible.
	  logic in mech-v2-00 page 12 does not look simple enough for me.
	- mech-v2-00 rule does not work (it seems to me) if you have multiple
	  layer of encapsulation, like IPv6-over-IPv4-over-IPv6 and such.
	  if we remove logic in page 12 to nothing by simpler encapulating
	  rule, we can make mech-v2-00 logic to work with any layers of
	  encapsulation (note: we have nationwide deployment of IPv4-over-L2TP
	  service in production, and we run IPv6 over it.  IPv4-over-L2TP
	  service has more than enough problem with PMTUD, and i guess it will
	  continue to be like this).

	and here are other points i would like to make:
	- PMTUD makes sense only when the node performing PMTUD is the
	  originating node and has ways to retransmit packets.  here, IPv6
	  tunnel ingress router is not the originating node.  it is true that,
	  with mech-v2-00 logic ICMPv4 need fragment message will get translated
	  into ICMPv6 too big messages, however, it is overly complex.
	- ICMPv4 need fragment message do not usually carry the whole original
	  packet, therefore it is difficult (if possible at all) for IPv6
	  originating node to confirm that ICMPv6 too big message is authentic
	  enough (like matches TCP connection table it has).
	- translating ICMPv4 message into ICMPv6 message is bad, as there's no
	  way to authenticate ICMPv4 messages.  any malicious nodes can inject
	  ICMPv4 need fragment messaage to tunnel ingress box and make it
	  generate ICMPv6 too big messages.

>headers or only contains one IP header.

	i understand your goal (to make use of bigger link MTU as much as
	possible), but i don't think the benefit outweigh the above problems.

itojun

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 19 Aug 2002 10:55:05 +0200 (CEST)

> 	this is also the case for Erik's version. (if IPv6 tunnel MTU > IPv4

I think you meant to say "the version in RFC 1933 = RFC 2893".
But your statement isn't very exact (or I don't understand
what the above "if" statement is trying to say).

Only if the IPv4 path MTU is less than 1300 bytes 
is there ever a need to use IPv4 fragmentation causing
reassembly at the decapsulator.

This is the equivalent to an IP-over-foo interface not providing an
MTU of at least 1280 bytes requiring link specific fragmentation and
reassembly.

Extracting from this that it is always the best tradeoff (performance vs. 
implementation complexity) to rely on IPv4 fragmentation seems like
a stretch.

  Erik

---

From: [email-masked]
Date: Mon, 19 Aug 2002 19:38:33 +0900

>	  continue to be like this).

	nitpicking - this IPv4-over-L2TP service is run by NTT east/west,
	not IIJ.

itojun

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 19 Aug 2002 13:16:00 +0200 (CEST)

> 	  for PMTUD to work effectively (from previous message you don't seem

Are these two separate points i.e. is the first sentence a separate point
from the second sentence?

On the "effectively" part:
You'll have nested tunnels in some cases e.g. when combining IPv6-in-IPv4
tunnels with IPsec tunnels. In the worst case such nested tunnels might,
from a path MTU perspective, look like the a set of descending interface
MTUs on a straight path i.e.
	Sender -> R1 -> R2 -> R3 -> Receiver
where Sender sees an interface MTU which is larger than the interface
MTU on R1->R2, which in turn is larger than the interface MTU on R2->R3, etc.

In a case of such descending interface MTUs for N steps, whether tunneling 
is the source of the interface MTUs or not, the effect is that
N ICMP packet too big will be needed for the sender to determine the path
MTU.

If the sender always sends with the minimum MTU (1280 for IPv6, 68 for IPv4)
then by definition we know that there can be no descending interface MTUs.

Now let's look at tunnel specifics.
If you mandate the tunnel interface MTU to be 1280 bytes and
don't set DF, then you might
think that this is always an improvement because you won't have to
deal with IPv4 issues. But in some cases you will actually cause descending
interface MTUs where it wouldn't happen in the if you had a varying 
tunnel interface MTU.
For example,
	Sender -> R1 -> R2 ======== R3
If R1->R3 is an Ethernet (MTU 1500), and the R2->R3 tunnel
is built over fddi, ATM, and/or Sonet (presumably frame relay as well), 
then with your fixed tunnel MTU you are making R2 send back ICMP too big,
when a varying tunnel MTU would not cause such ICMP errors.

A different example,
	Sender -> R1 -> R2 ======== R3
If R1->R3 is an Ethernet (MTU 1500), and the R2->R3 tunnel is built over
only Ethernet, then R2 will see an IPv4 interface MTU which equals the IPv4
path MTU to R3, thus it will send ICMP packet too big with MTU = 1480 whether
a fixed or variable MTU applies to the tunnel.

Finally, 
	Sender -> R1 -> R2 ======== R4
	                       R3
(where R3 is an IPv4 router inside the tunnel between R2 and R4)
If R1->R3 is an Ethernet (MTU 1500), and R2->R3 is fddi, ATM, etc,
and R3->R4 is an Ethernet, then
 - in the fixed MTU case you'll see one ICMP too big (from R2 with MTU=1280)
 - in the variable MTU case R3 will generate one ICMP too big (MTU=1500).
   If R3 generates an ICMPv4 too big with more than 8 bytes of payload
   (as recommended in some specification somewhere) then this
   can be converted to an ICMPv6 packet too big

From the above 3 cases I can't concluded that either fixed or variable
MTU is strictly better in terms of number of ICMP too big messages.
Instead the result of the comparison depends on the assumptions
on which cases (and subcases of the last one) are more likely to
occur in the network.

> 	  do not meet this fundamental design of the Internet.

The current evolution of the internet with QoS, MPLS, multicast, etc
seem to lead to more and more soft-state being maintained in routers.
The important design principle we have for the Internet is to make
all this state be soft, not to try to remove the state.

> 	  logic in mech-v2-00 page 12 does not look simple enough for me.

Hard to quantify this argument.
RFC 1981 doesn't look simple to me either. Neither does OSPF or IS-IS.

> 	  layer of encapsulation, like IPv6-over-IPv4-over-IPv6 and such.

AFAIK it works. There are likely to be implementation issues when there
are multiple layers of encapsulation implemented on the same node.
(e.g. how the layers communicate internally on the node)
When the encapsulations are on separate nodes the ICMP "too big"
is the only necessary communication.

So which case doesn't work?

> 	  continue to be like this).

But this is an argument for PMTUD in general being to complex
hence we should get rid of it.
I don't subscribe to that argument.

> 	  into ICMPv6 too big messages, however, it is overly complex.

No. The performance issue that occurs when the retransmission unit is
not the same as the loss unit occurs independently of where any
PMTUD is performed.

> 	  enough (like matches TCP connection table it has).

It isn't clear how useful such a check would be; presumably such
implementations would attackable by sending forged ICMP errors claiming to be
in response to UDP packets instead of TCP ones. (And with IPv6 there is to IP
identifier which can be verified independent of upper layer protocol).
Thus the utility of such an approach is questionable, and the threat
of attackers just forcing the sender to use the minimum MTU instead
of a larger MTU might not be that significant.

But in any case, if there are nodes which verify the ICMP errors against 
the TCP connection table such nodes would presumably need to ignore ICMP
errors which do not contain enough headers to be verified. Whether such
ICMP errors are generated due to RFC 2893 or for other reasons doesn't
change this need.

And revising RFC 2893 to not convert ICMPv4 errors to ICMPv6 errors 
wouldn't remove that need for such implementations either.

> 	  generate ICMPv6 too big messages.

Yes, but this doesn't add any new threat. Any node can generate
an ICMPv6 too big message with their own source address in the packet
and the receiver has no way to authenticate whether the sender of
the error is who it claims to be, and no way to authorize whether the sender
is indeed allowed to lower the MTU; in particular there is no way for the
receiver to know which are the routers on the path.

> >As I said before, I think the same motivation and tradeoffs for PMTU
> discovery >apply for traffic whether or not it is encapsulated in multiple IP

My goal is and was to solve the "fragmentation considered harmful" issues by
as much as possible making the unit of loss be the same as the unit of
retransmission end-to-end - not make the link MTU bigger.

Based on my rough analysis above there seem to be cases where a fixed
tunnel MTU = 1280 perform better and also cases where it performs worse
than a variable MTU.
And that doesn't take into account any performance difference in being able
to exploit a larger MTU; merely the performance difference relating to
the number of ICMP errors generated.

  Erik

---

From: [email-masked]
Date: Mon, 19 Aug 2002 20:45:37 +0900

this discussion will take a long time, parhaps make a good topic
	for interim - so i would like to make a point separately from the
	discussion.

	if RFC2893/mech-v2 continues to use today's form of MTU handling,
	6to4 and ISATAP must not refer RFC2893 for MTU issues.  these
	documents have to define fixed MTU value for the link, and probably
	MRU value as well.

itojun

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 19 Aug 2002 16:16:43 +0200 (CEST)

> 	documents have to define fixed MTU value for the link, and probably

I agree it makes sense to discuss the two separately, because there are
some differing issues.

But I don't take it for given that 6to4 and ISATAP must not use a variable
MTU; from an implementation perspective the code for a configured pt-pt tunnel
for variable MTU just magically applies to the point-to-multipoint
tunnel interfaces as well.

For a multicast capable tunnel interface the issues are different, since
you can't discover the MTU to use for IPv4 multicast packets using
path MTU discovery hence a fixed MTU is needed at least for multicast on
such tunnels. 

  Erik

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 19 Aug 2002 16:39:26 +0200

I fail to see how 6to4 can define a fixed value.

It seems to work OK today.

  Brian

---

From: [email-masked]
Date: Tue, 20 Aug 2002 00:42:50 +0900

>tunnel interfaces as well.

	normally interfaces have the same (and constant) MTU for any destination
	on the same link - (with 6to4, "link" equals to "2002::/16 cloud").
	for instance, assuming that we model 6to4 as a pseudo interface,
	what value would you put into BSD ifp->if_mtu? (dunno what is the
	equivalent thing on Solaris)

itojun

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 19 Aug 2002 17:59:06 +0200 (CEST)

> 	equivalent thing on Solaris)

A useful value would be the interface MTU for the underlying IPv4
interface minus the added header.
If the 6to4 interface isn't hard bound to use a particular IPv4 interface
underneath it, it would be useful to take the max of the possible
(non-loopback) IPv4 interface MTUs (minus the added header).
Note that if routing changes, and or the set of IPv4 interfaces change on
the running system (hotplugging of NICs etc) then this number can
actually change over time.

And then the IPv4 routing table will record the IPv4 path MTU to each
destination without additional state on a Unix box, and IPv6 can be made
to "see" this MTU "through" the tunneling pseudo-device driver somehow.
This does have *some* additional implementation complexity, but for
the Solaris 6to4 driver I believe this came for free by reusing the code
for the configured tunneling driver.

  Erik

---

From: [email-masked]
Date: Tue, 20 Aug 2002 01:09:00 +0900

i think here we are talking about a different topic than the beginning
	- we are talking about 6to4 interface MTU for a particular node.
	we also need to talk about default 6to4 interface MTU/MRU for all
	6to4-capable nodes in the world.

>for the configured tunneling driver.

	so are you saying that, on Solaris, fragmentation logic is inside
	interface driver, not in L3 output logic like ip6_output()?

	on BSD/KAME, we need a static, constant value in ifp->if_mtu to
	determine if the packet is "too big" to go out from 6to4 interface.
	i don't think it right to put tweaks to look up IPv4 path MTU to
	this part of ip6_output().  6to4 pseudo interface has to be manipulated
	in the same way as other interfaces (like ethernet).

itojun

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 19 Aug 2002 18:45:01 +0200 (CEST)

> 	6to4-capable nodes in the world.

My first paragraph (see below) was about the interface MTU issue.
The second paragraph below is about the per-destination MTU issue with 6to4.

Of course, "MRU" and how it interacts with "be liberal in what you accept"
is a separate topic :-)

> 	interface driver, not in L3 output logic like ip6_output()?

I didn't say that.
The data structure in Solaris which corresponds to the destination cache 
in RFC 2461 contains the path MTU. When that cache entry is created it
effectively looks through the tunneling pseudo-driver.
One way of doing this is to is to use locally generated ICMP too big errors
but that might not be the optimal approach.

> 	in the same way as other interfaces (like ethernet).

See my first paragraph above on what to could be used for if_mtu.

  Erik

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 20 Aug 2002 09:54:43 +0200

Everything here tells me that this is something we absolutely
cannot usefully write into a standard. If we go too far down
this path we will end up with a default MTU for 6to4 of
(576-header) which is hardly what we want.

I believe this should be left as an exercise for the implementor.

(I'm not commenting on ISATAP MTU.)

   Brian

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Brian E Carpenter 
Distinguished Engineer, Internet Standards & Technology, IBM 
On assignment at the IBM Zurich Laboratory, Switzerland

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 20 Aug 2002 09:11:23 -0400

Actually, I think that we want to say that the minimum PMTU for a
6to4 tunnel is 1280, which requires an IPv4 PMTU of at least 1300,
or requires fragmentation and reassembly at the IPv4 level.

If we want to fix a default MTU for 6to4 or ISATAP tunnel interfaces,
we have two choices:

         - Tell implementors to use the MTU of the underlying
                 IPv4 interface, minus 20 or 1280 (whichever is
                 larger)
         - Fix a number like 1280, 1500, 4096...

I much prefer the first option, as a default, with the possibility 
of configuring a different value.

If we expect 6to4 to commonly float over several interfaces, we could
use the "use highest MTU - 20" wording that Erik suggested, as long
as that result is at least 1280.

The tricky part is now what number to pick, but how/if we want to 
use PMTU discovery to avoid fragmentation at the IPv4 layer.  Since
I think that we want 6to4 to work over IPv4 paths that have a PMTU
of less than 1300, I'd err on the side of working and not set the
DF bit in the IPv4 packet headers.  I can see the value of the
other argument, though...

Margaret

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 20 Aug 2002 16:01:33 +0200

I agree with your logic, but my instinct is still to say nothing at all

  Brian

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 20 Aug 2002 18:24:51 +0200 (CEST)

> I believe this should be left as an exercise for the implementor.

Perhaps so, but then we at least need to specify enough to ensure 
interoperability; Itojun has reported interoperability
problems relating to MTU/MRU/PMTU for tunnels.

  Erik

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 21 Aug 2002 09:02:41 +0200

> problems relating to MTU/MRU/PMTU for tunnels.

It may be that all we can do is document those problems, or document
the parameters that need to be configured.

  Brian

---

From: Robert Elz <[email-masked]>
Date: Wed, 21 Aug 2002 14:36:23 +0700

Date:        Wed, 21 Aug 2002 09:02:41 +0200
    From:        Brian E Carpenter <[email-masked]>
    Message-ID:  <[email-masked]>

  | It may be that all we can do is document those problems, or document
  | the parameters that need to be configured.

The issue is that 6to4 is a worldwide common link - just documenting
"configure something" is not a lot of use.  If there's an actual requirement
for a specific value to be configured, that value needs to be specified.

I think I am convinced that at the very least, 6to4 needs to specify a
minimum MRU that a 6to4 node should be prepared to accept.   What value
that is (1280 <= MRU <= 65536) I haven't yet reached a conclusion on,
though I suspect I'd be happiest somewhere in the 4K-8K range.

If that gets done, whether we also need to specify a mandatory MTU for
all nodes, I'm not so sure about.   Clearly, as there is no pre-config
of 6to4 tunnels, the max safe MTU to use would be the MRU just required
above.  Whether all nodes should use that, or whether anything between
1280 and that value would be OK, I'm not yet clear on.

kre

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 21 Aug 2002 11:00:02 +0200

I certainly have no objection to specifying a minmimum MRU.
Pick a number.

   Brian

---

From: Robert Elz <[email-masked]>
Date: Wed, 21 Aug 2002 16:29:06 +0700

Date:        Wed, 21 Aug 2002 11:00:02 +0200
    From:        Brian E Carpenter <[email-masked]>
    Message-ID:  <[email-masked]>

  | I certainly have no objection to specifying a minmimum MRU.
  | Pick a number.

Well, since you asked, and in the tradition of IPv6 using what seem
to be random non power of 2 numbers, though ones for which there's a
justification if you look hard enough, I'm going to suggest 4554 (0x11CA)

I believe that specifying this will resolve the interoperability problem
that itojun mentioned.

  | > Whether all nodes should use that, or whether anything between
  | > 1280 and that value would be OK, I'm not yet clear on.

I have been thinking a bit more about that, and I believe that the
"anything" answer is the right one.  I don't think a standardised
approach is required (the fixed MRU will solve the problems).

Rather, allowing some implementations to pick 1280, and others bigger
numbers (up to the 4554, assuming everyone else accepts that one) will
allow the different (ipv4) fragmentation strategies that people have been
discussing here to be tested - with implementations making use of whatever
happens to be available to them.

kre

---

From: "Fred L. Templin" <[email-masked]>
Date: Fri, 23 Aug 2002 10:13:52 -0700

Hello,

I've been out of town while this discussion has transpired, but I'm mostly
caught up now and have some thoughts to share:

> but that might not be the optimal approach.

I believe Erik's statement above is consistent with section 5.2 of RFC 1981
("Storing PMTU Information"), i.e., when the IPv6 routing table is consulted to
determine the IPv6 next hop toward a particular destination, the PMTU is taken from
the destination cache entry for the next hop. If no such entry exists, one is created
and the PMTU is initialized to the interface MTU of the underlying link layer. In the
case of automatic tunnels, the tunnel pseudo-interface *may* be configured over multiple
physical links used for IPv4 as I believe Margaret Wasserman mentioned in an earlier
message. But, since automatic tunneling mechanisms encode IPv4 information in the IPv6
next hop address, the correct physical interface can be determined by consulting the
IPv4 routing table.

Itojun points out that BSD/KAME simply looks to 'ifp->if_mtu' and uses whatever static
value happens to appear there. Linux provides a routine called 'ipv6_get_mtu()' for
this same purpose. This routine currently does not implement the logic I've described
above, but it could easily be extended to do so (as could BSD/KAME and any other system
that supports dual-stack). So, I don't quite see the IPv6-over-IPv4 tunnel interface as
having a "variable MTU". Rather, the tunnel interface MTU is *transparent*, and the
actual MTU *for a particular next hop* is obtained by drilling down to the IPv4 layer,
i.e., "looking through" the tunnel pseudo-driver as Eric said above. (The MTU so
discovered must be adjusted downward by 20 octets for IPv4 encapsulation, of course.)

Fred
[email-masked]

---

From: Steve Deering <[email-masked]>
Date: Fri, 30 Aug 2002 11:56:34 -0700

>I certainly have no objection to specifying a minmimum MRU.

I just dropped into the middle of this thread, so am missing much of
the context, but 1500 (excluding IP4 tunnel overhead) would seem the
natural number to use for the above, given that all IPv6 nodes are
already required to be able to handle packets of that size, but not
necessarily any larger.  (1500 is IPv6's minimum reassembly buffer
size; see top of page 25 in RFC 2460.)

Steve