Subject: Re: [v6ops] Bar BoF on a 6to4 replacement?

From: Owen DeLong <[email-masked]>
Date: Mon, 1 Dec 2014 12:13:33 -0800

> On Nov 12, 2014, at 12:12 PM, Iljitsch van Beijnum <[email-masked]> wrote:
> 
> I've been talking to a few people and I think there is potential for a draft that lets home gateways (and any other system so inclined) have IPv6 tunneling enabled out of the box (like 6to4) but in a way that works pretty reliably (unlike 6to4).
> 
> My question to (potential) implementers:
> 
> - Is requiring HTTP digest authentication problematic?

Probably

> - Is requiring HTTPS problematic?

Probably not.

> - Is requiring JSON parsing problematic?

No.

>   * Would using plain text similar to HTTP headers to pass parameters be better?

I think this would actually be worse.

> - Is requiring DHCPv6 (the prefix delegation client role) problematic?

Yes. Making it a should with the option for a mechanism that provides a statically configured prefix would be preferable, IMHO.

Owen

---

From: Owen DeLong <[email-masked]>
Date: Mon, 1 Dec 2014 12:27:17 -0800

> On Nov 12, 2014, at 11:45 PM, Doug Barton <[email-masked]> wrote:
> 
> On 11/12/14 1:33 PM, Gert Doering wrote:
>> Hi,
>> 
>> On Wed, Nov 12, 2014 at 10:12:17AM -1000, Iljitsch van Beijnum wrote:
>>> I've been talking to a few people and I think there is potential
>>> for a draft that lets home gateways (and any other system so inclined)
>>> have IPv6 tunneling enabled out of the box (like 6to4) but in a way
>>> that works pretty reliably (unlike 6to4).
>> 
>> I'm not sure where the value of that is.
>> 
>> 10 years ago, I would have agreed that "getting IPv6 out to the people"
>> is a good goal - but when this draft gets to the stage that you'll see
>> devices actually shipping, it's 2016, and we'll have 30-40% native IPv6
>> in many countries already - which should create enough momentum by it's
>> own...
>> 
>> So, call me sceptic :)
> 
> +1
> 
> We don't need this, shouldn't develop it, and definitely shouldn't deploy it.
> 
> Doug

I am inclined to agree. Had we developed it 20 years ago and convinced home gateway manufacturers to deploy it 10 years ago, it would have been great. Today, it’s just a waste of resources.

Owen

---

From: Owen DeLong <[email-masked]>
Date: Mon, 1 Dec 2014 12:35:20 -0800

> On Nov 13, 2014, at 11:56 AM, Gert Doering <[email-masked]> wrote:
> 
> Hi,
> 
> On Thu, Nov 13, 2014 at 09:50:31AM -1000, Iljitsch van Beijnum wrote:
>> There's no reason tunnels have to be so-so. 
> 
> I'm wondering who would provide these tunnels in the future, and what would
> their business model be.
> 
> This is a real question.  Today, you can get tunnels from Sixxs nodes
> (enthusiasts, which might eventually consider the IPv6 deployment issue
> to be "solved" and stop running tunnel nodes), HE.NET (it's good marketing,
> but eventually the costs involved might win over the marketing gain?), 

I’m no longer with HE, but I will say that this is unlikely. While the cost of supporting
the tunnel brokers is not zero, it comes sufficiently close that I’d be very surprised
to see it reach this state at any point prior to ubiquitous native IPv6 and the deprecation
of IPv4 as a common backbone internet protocol. HE has multiple reasons for
providing the tunnelbroker service that go beyond mere marketing.

(In other words, there will probably be pockets of IPv4, but “the internet” will be an
IPv6 internet and IPv4 will exist in islands where it can’t yet be deprecated, but the
hosts that still talk to the internet will do so over IPv6 somehow.)

Owen

---

From: Owen DeLong <[email-masked]>
Date: Mon, 1 Dec 2014 12:42:16 -0800

> On Nov 13, 2014, at 11:59 AM, George Michaelson <[email-masked]> wrote:
> 
> HE.net tunnels cause severe distortion of RTT. They are anything but local termination for much of the world, and whilst I applaud their role in promoting V6 adoption, its a poor fit to the money flow of IPv4 relationships for many people. I should know: I have one in Australia, and my nearest terminations are SG/HK (which is via the US and trombones) or the US (which is 200+ms delay from me)

Why would SG/HK be via the US from AU? That sounds like your provider has failed to provide decent connectivity to Asia more than an issue related directly to tunnelbroker.

Of course, if you can find native IPv6 in AU, why not set up your own tunnelbroker for Aussies or work with one of the existing ones to build a local node?

> I cannot speak to SIXX but I suspect it too distorts the packetflow and causes a disparate RTT compared to native.

Any tunnel or tunnel broker service which involves a detour over a long distance will create RTT disparity. If you can get native, then native will always be superior to tunnels. Tunnels are intended for circumstances where you can’t get native.

> 6rd is homed in your own ISP. its almost 1:1 congruent with your normal RTT for native IPv6 endpoints.

Sure, and that’s great if your ISP supports 6rd. Again, that’s not the target audience for tunnel brokers.

> When we say 'tunnels are bad' its not because GRE randomly corrupts bits: its because the effects on the end-to-end model are bad. The badness varies, but the consequence of a tunnel is usually not what you wanted in reality, unless the sole consideration was connectivity per se, not its quality, compared to V4

Actually, when we say “tunnels are bad”, it’s because we like to oversimplify and we usually develop some tunnel vision in the process.

I’m running entirely on tunnels for both v4 and v6. They don’t significantly distort my RTTs vs. native and in many cases, I’ve discovered that I get better performance over my tunnels that I could have received natively because my tunnels aren’t subject to some of the games that get played to try and manipulate financials between providers to extort payments for access to eyeballs.

Tunnels can be good. Tunnels can be bad. They are a tool like any other tool in the networking toolkit and knowing how to use the tool properly can greatly improve your user experience with said tool.

Owen

> 
> -G
> 
> On Thu, Nov 13, 2014 at 9:56 AM, Gert Doering <[email-masked] <mailto:[email-masked]>> wrote:
> Hi,
> 
> to be "solved" and stop running tunnel nodes), HE.NET <http://he.net/> (it's good marketing,
> and others that run tunnels for various reasons that might not be there
> in 2-3 years...
> 
> 6rd is a very specific tunnel technology that has answers to that question
> ("I want to roll out IPv6 to my customers but part of my infrastructure
> cannot do it.  So I provision my CPEs to do 6rd, and take responsibility
> to run the 6rd relay") - but as with 6to4 relays, what is the incentive to
> run a tunnel broker in 2-3 years from now?  You won't get paid (users do
> not pay for IPv6, otherwise they would go and choose ISPs that provide
> v6)...
> 
> Gert Doering
>         -- NetMaster
> --
> have you enabled IPv6 on something today...?
> 
> SpaceNet AG                        Vorstand: Sebastian v. Bomhard
> Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
> D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
> Tel: +49 (0)89/32356-444 <tel:%2B49%20%280%2989%2F32356-444>           USt-IdNr.: DE813185279
> 
> _______________________________________________
> v6ops mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/v6ops <https://www.ietf.org/mailman/listinfo/v6ops>
> 
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Owen DeLong <[email-masked]>
Date: Mon, 1 Dec 2014 12:46:29 -0800

> On Nov 13, 2014, at 2:22 PM, Alexandru Petrescu <[email-masked]> wrote:
> 
> Le 13/11/2014 22:48, Jeroen Massar a écrit :
>> On 2014-11-13 20:56, Gert Doering wrote:
>>> Hi,
>>> 
>> These services already exists, they are called "VPN Providers".
>> 
>> Typically used for the more shady reasons, but they can in more
>> professional forms be used for what most people tend to use IPv6 for:
>> getting connectivity with a public IP address.
>> 
>> 
>>> to be "solved" and stop running tunnel nodes)
>> 
>> That is the end goal indeed :)
>> 
>> 
>> [..]
>>> but as with 6to4 relays, what is the incentive to
>>> run a tunnel broker in 2-3 years from now?
>> 
>> Getting IPv6 out there, learning from the problems, letting people use it.
>> 
>> But as when closing down the 6bone: we are today already long past the
>> point where these experiences are needed.
>> 
>> We know what is broken and we know how to approach it.
>> 
>> Tunnel Brokers, 6rd etc are transition tech. Hopefully everybody will
>> have native at one point or another.
> 
> Goal - any new replacement of 6to4 must be future proof: avoid renumbering when 6to4-replacement dies out.  (this is what current tunnel brokers dont offer, they force one into a lockin going away from tunnel broker to native IPv6 one has renumber).

Depends on the tunnel broker and how you use it.

My use of HE tunnelbroker will not require me to renumber out of 2620:0:930::/48 when I stop using it.

Owen

> 
> Alex
> 
>> 
>>> You won't get paid (users do
>> VPN providers are proof that people pay for tunneled connectivity.
>> 
>> Heck, most of those are stuffing users behind NAT and not even giving
>> public IP addresses away to "protect the anonymity of the user"...
>> 
>> 
>> And there are tunnel brokers doing it for fun and free ;)
>> It is all magic :)
>> 
>> Greets,
>>  Jeroen
>>

---

From: John Mann <[email-masked]>
Date: Tue, 2 Dec 2014 23:34:59 +1100

Hi,

On 2 December 2014 at 07:42, Owen DeLong <[email-masked]> wrote:

> HE.net tunnels cause severe distortion of RTT. They are anything but
> local termination for much of the world, and whilst I applaud their role in
> promoting V6 adoption, its a poor fit to the money flow of IPv4
> relationships for many people. I should know: I have one in Australia, and
> my nearest terminations are SG/HK (which is via the US and trombones) or
> the US (which is 200+ms delay from me)
>
>
> Why would SG/HK be via the US from AU? That sounds like your provider has
> failed to provide decent connectivity to Asia more than an issue related
> directly to tunnelbroker.
>

Out of interest, I took a look ...

Taking an example, looking at
  https://tunnelbroker.net/status.php
we see a tunnel endpoint at
   tserv19.hkg1 == 216.218.221.2

Also, Monash University is AS56132; we get most of our Internet from AARNet
AS7575.

Looking at
   http://lg.aarnet.edu.au/cgi-bin/lg
  'show bgp ipv4 216.218.221.2' => 216.218.221.0/24 advertised by AS 6969
  'trace 216.218.221.2' goes via nsw, pao, lax, tyo, hkg
And also
  http://bgp.he.net/AS56132#_graph4
  shows AS56132 -> AS7575 -> AS6939
  and also  AS7575 -> AS10026 -> AS6939

So, the problem seems to be that AARNet directly peers with HE at PAO
and doesn't peer with HE at any other IX (or if it does, that peering is
not preferred).
AARNet _could_ also directly peer with HE in Singapore
   http://he.net/HurricaneElectricNetworkMap.pdf

https://www.aarnet.edu.au/images/uploads/resources/International_Map_May_2014.pdf
but care would need to be taken when choosing between the 5 Gbit/s
Australia<->Singapore path v. the 120 Gbit/s Sydney<->USA path

Looking closer at the traceroute
===
lg.aarnet.edu.au> traceroute 216.218.221.2
traceroute to 216.218.221.2 (216.218.221.2), 30 hops max, 40 byte packets
 1  ge-1-0-0.bb1.a.syd.aarnet.net.au (202.158.196.129)  0.390 ms  0.369 ms
 0.359 ms
 2  ae9.pe2.brwy.nsw.aarnet.net.au (113.197.15.56)  0.335 ms  0.327 ms
 0.317 ms
 3  ge-4_0_0.bb1.a.pao.aarnet.net.au (202.158.194.177)  *155.513* ms
 155.508 ms  155.521 ms
 4  30gigabitethernet2-1.core1.pao1.he.net (198.32.176.20)  *252.462* ms
 252.474 ms  252.337 ms
 5  10ge11-6.core1.lax1.he.net (72.52.92.22)  252.252 ms  252.247 ms
 252.236 ms
 6  10ge1-3.core1.lax2.he.net (72.52.92.122)  249.929 ms  249.918 ms
 249.912 ms
 7  10ge3-2.core1.tyo1.he.net (184.105.223.106)  250.008 ms
10ge6-3.core1.hkg1.he.net (184.105.223.170)  255.574 ms
10ge3-2.core1.tyo1.he.net (184.105.223.106)  250.006 ms
 8  10ge5-2.core1.hkg1.he.net (184.105.222.106)  256.108 ms  257.475 ms
 257.460 ms
 9  tserv1.hkg1.he.net (216.218.221.2)  250.165 ms * *
===
The RTT between hops 3 and 4 increases from 155 ms to 252 ms, even though
both routers claim to be in PAO.
Perhaps the return traffic is taking some non-optimal path.

Indeed, a traceroute run on  http://lg.he.net/  from fmt1, hkgq, sin1
to  202.158.196.129 (the hop 1 address from above)
has all three traceroutes routing via asianetcom in Singapore,
*and* the RTTs being about the same (254 ms, 250 ms, 252 ms).

So ... it looks like we have a 250 ms loop for all AARNet <-> HE traffic.

       97ms
   SIN  <-  PAO
76ms v     ^  77ms
       AUS

This is a lot slower than about 155 ms for the direct return path

     AUS <-> PAO

Thanks,
    John

> Of course, if you can find native IPv6 in AU, why not set up your own
> tunnelbroker for Aussies or work with one of the existing ones to build a
> local node?
>
> I cannot speak to SIXX but I suspect it too distorts the packetflow and
> causes a disparate RTT compared to native.
>
>
> Any tunnel or tunnel broker service which involves a detour over a long
> distance will create RTT disparity. If you can get native, then native will
> always be superior to tunnels. Tunnels are intended for circumstances where
> you can’t get native.
>
> When we say 'tunnels are bad' its not because GRE randomly corrupts bits:
> its because the effects on the end-to-end model are bad. The badness
> varies, but the consequence of a tunnel is usually not what you wanted in
> reality, unless the sole consideration was connectivity per se, not its
> quality, compared to V4
>
>
> Actually, when we say “tunnels are bad”, it’s because we like to
> oversimplify and we usually develop some tunnel vision in the process.
>
> I’m running entirely on tunnels for both v4 and v6. They don’t
> significantly distort my RTTs vs. native and in many cases, I’ve discovered
> that I get better performance over my tunnels that I could have received
> natively because my tunnels aren’t subject to some of the games that get
> played to try and manipulate financials between providers to extort
> payments for access to eyeballs.
>
> Tunnels can be good. Tunnels can be bad. They are a tool like any other
> tool in the networking toolkit and knowing how to use the tool properly can
> greatly improve your user experience with said tool.
>
> Owen
>
>
> -G
>
> On Thu, Nov 13, 2014 at 9:56 AM, Gert Doering <[email-masked]> wrote:
>
>> Hi,
>>
>> Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279
>>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 2 Dec 2014 19:44:11 +0100

On 19 Nov 2014, at 19:26, Keith Moore <[email-masked]> wrote:

> If there was actually a bar BOF on a 6to4 replacement, would someone care to summarize?

There was no actual bar BoF.

However, I did talk to a number of people individually about this. I'd say about half of them felt that working on tunnels and/or working on "a 6to4 replacement" was not a good idea. I couldn't tease out whether that was an understandable immediate reaction or also a more considered opinion, and it was also hard to determine whether people mostly objected to the 6to4 part or to any work on tunnels in general.

I explained my proposal to a number of people:

"Something that is as easy to use as 6to4 but as reliable as a tunnel broker tunnel"

The short version:

- NO ANYCASTING, NO ROUTE OPTIMIZATION
- give home gateways a unique login out of the box, such as <macaddress>@<vendor-something>
- user can override this with <user>@<isp> or <user>@<tunnelbroker>
- home gateways connect to servers indicated by the domain part of the login (through SRV records) and state their capabilities
- the server matches their IPv4 address, NAT/CGN status and capabilities with available tunnel options such as brokers, 6rd, public gateways, sends back configuration info
- (ISPs would have to publish their gateway configurations)
- home gateway sets up a tunnel and uses keepalives to make sure it's up, go back to the server if the tunnel goes down for whatever reason
- all of this can of course also be done by individual hosts

I got a generally cautiously positive reception from those that gave me enough time to explain all of this, except that those with skin in the game of course still prefer their own systems.

So I guess I have a draft to write.

Iljitsch

---

From: Ole Troan <[email-masked]>
Date: Tue, 2 Dec 2014 20:15:16 +0100

> So I guess I have a draft to write.

done. RFC5571.

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 2 Dec 2014 19:17:51 +0000

Hi Iljitsch,

Any way you do this, the home gateway needs to be enrolled with the tunnel service
provider. Additionally, it needs security keys that it can use to sign its messages and
get the tunnel service provider to accept them. So, "out of the box" means first
handled by the tunnel service provider and not just straight off the assembly line.

> - NO ANYCASTING, NO ROUTE OPTIMIZATION

I don't mind "no anycasting", but why "no route optimization"? Route optimization
will give better performance and avoid single points of congestion.

Thanks - Fred
[email-masked]

> -----Original Message-----
> From: v6ops [mailto:[email-masked]] On Behalf Of Iljitsch van Beijnum
> Sent: Tuesday, December 02, 2014 10:44 AM
> To: Keith Moore
> Cc: [email-masked]
> Subject: [v6ops] Report: Bar BoF on a 6to4 replacement
> 
> However, I did talk to a number of people individually about this. I'd say about half of them felt that working on tunnels and/or
> working on "a 6to4 replacement" was not a good idea. I couldn't tease out whether that was an understandable immediate reaction or
> also a more considered opinion, and it was also hard to determine whether people mostly objected to the 6to4 part or to any work on
> tunnels in general.
> 
> I got a generally cautiously positive reception from those that gave me enough time to explain all of this, except that those with skin in
> the game of course still prefer their own systems.
>

---

From: George Michaelson <[email-masked]>
Date: Wed, 3 Dec 2014 08:06:16 +1000

And.. we just lost SEA-ME-WE3. So there is no direct to SG and there won't
be for > 2mo judging by the last delay (Indon doesn't like cable ships
operating without a licence)

HK? Its reached via Japan. Japan? its reached via Hawaii, Guam, or LA.

So these apparent optimal routes by ASN, are sometimes significantly less
optimal.

I stand by my words: Tunnels are a mechanism which foist unexpected RTT
outcomes.

-G

On Tue, Dec 2, 2014 at 10:34 PM, John Mann <[email-masked]> wrote:

> Also, Monash University is AS56132; we get most of our Internet from
> AARNet AS7575.
>
> Looking at
>  1  ge-1-0-0.bb1.a.syd.aarnet.net.au (202.158.196.129)  0.390 ms  0.369
> ms  0.359 ms

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Wed, 3 Dec 2014 00:18:13 +0100

> I stand by my words: Tunnels are a mechanism which foist unexpected RTT

The RTT to my colo server in Germany is 10ms *less* over IPv6 HE
tunnel than over the native IPv4 that tunnel runs over.

So it looks like we got the opposite ends of the straw. Which confirms
your words, somewhat, I guess, but not in the always negative
direction :-)

--a

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 09:11:21 +0100

On 2014-12-02 13:34, John Mann wrote:
[..]
> Also, Monash University is AS56132; we get most of our Internet from

Congrats, you have access to an ISP who has been providing native IPv6
for many many years (likely a decade at least).

Please use the native connectivity :)

Greets,
 Jeroen

---

From: John Mann <[email-masked]>
Date: Wed, 3 Dec 2014 21:11:34 +1100

Jeroen,

> Please use the native connectivity :)

I do use native IPv6 connectivity.

WIthin Australia ... see
   http://labs.apnic.net/ipv6-measurement/Economies/AU
     click   "ASN in this Economy" tab
--- (edited)
ASN   AS Name           #samples v6-capable v6-preferred
56132 Monash University   250     63.60%     58.80%
 4739 Internode Pty Ltd  3924      6.27%      6.14%
...
---
Monash (rank #1) is my work Internet provider, and Internode (rank #2) is
my home ISP.

The question I was investigating was why a HE tunnel in SG/HK isn't useful
to Australians [who don't use an IPv6-capable ISP].

===

Back on topic:  Monash's "Anycast 6to4 fail moment" was in 2007 or 2008.
Non-SOE Windows Vista laptops with ICS enabled were advertising
6to4-derived routes to Mac OSX 10.4 machines
and breaking their connectivity to IPv6 sites.  We fixed that using a
RA-filter on all user network ports.

After dual-stack IPv6 was enabled on most subnets, protocol 41 and Teredo
were blocked at Monash's Internet border on 15-Jun-2011.  I don't recall
anyone ever asking for it to be re-enabled.

I assert that the time of "experimental" IPv6 has passed.  IPv6 should be a
"production" service.
If the IPv6 network service isn't almost as good (due to reliability or
RTTs or whatever) as IPv4,
turn it off until you can do it right.

Thanks,
    John

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 12:18:20 +0100

On 2014-12-03 11:11, John Mann wrote:
[..]
> The question I was investigating was why a HE tunnel in SG/HK isn't
> useful to Australians [who don't use an IPv6-capable ISP].

Then doing so from the network they would be using would be a better way
to do so as every ISP has their own routing policies...

[..]
> I assert that the time of "experimental" IPv6 has passed.  IPv6 should
> be a "production" service.

Full ack there.

> turn it off until you can do it right.

Or ask help in fixing the problem. There are lots of folks who will help
one out. [email-masked] is the venue for that (if one is an
operator, not meant for end-users...) Little the IETF can do in
standardizing more stuff to fix that problem.

Greets,
 Jeroen

---

From: "Metzler, Dan J" <[email-masked]>
Date: Wed, 3 Dec 2014 11:41:43 +0000

Hi,

A few thoughts…

‘After dual-stack IPv6 was enabled on most subnets, protocol 41 and Teredo were blocked at Monash's Internet border on 15-Jun-2011.  I don't recall anyone ever asking for it to be re-enabled.’

-          Not necessarily a problem, “IF” Monash does not provide internet access to other ISPs.  If it does, then were they all dual-stack IPv6 enabled?  If not then I would point out that this broke IPv6 service for some downstream customers of those ISPs.  The only reason for an ISP to turn off Teredo and protocol 41 is if all downstream customers have access to native IPv6 connectivity, and even then that decision is questionable.

-          It is unlikely that anyone would ask for it to be re-enabled.  They would more likely incorrectly assume that IPv6 is broken and disable it.  The reason for this is that the vast majority of people, who were affected by this, likely would not have had the technical skill to trace it back to Monash.  Before they ever got to that point someone would likely tell them to disable IPv6, or all tunneling protocols thus completing the breakage of their IPv6 connectivity.

‘I assert that the time of "experimental" IPv6 has passed.  IPv6 should be a "production" service.’

-          100% agreed!

‘If the IPv6 network service isn't almost as good (due to reliability or RTTs or whatever) as IPv4,
turn it off until you can do it right.’

-          Noooooooooooo.

-          If the IPv6 network service isn’t almost as good (due to reliability or RTTs or whatever), and it is under your control to fix it, then please fix it.

-          If it isn’t under your control, then communicate it to your ISP or the offending party and ask them to fix it.

-          “Turn it off” is the wrong answer, though probably the more popular answer.  Turning it off breaks it more completely than it was with poor performance.

Thanks,

  Dan

From: v6ops [mailto:[email-masked]] On Behalf Of John Mann
Sent: Wednesday, December 3, 2014 4:12 AM
To: Jeroen Massar
Cc: [email-masked] WG
Subject: Re: [v6ops] Bar BoF on a 6to4 replacement?

Jeroen,

On 3 December 2014 at 19:11, Jeroen Massar <[email-masked]<mailto:[email-masked]>> wrote:
On 2014-12-02 13:34, John Mann wrote:
[..]

WIthin Australia ... see

...
---
Monash (rank #1) is my work Internet provider, and Internode (rank #2) is my home ISP.

The question I was investigating was why a HE tunnel in SG/HK isn't useful to Australians [who don't use an IPv6-capable ISP].

===

Back on topic:  Monash's "Anycast 6to4 fail moment" was in 2007 or 2008.
Non-SOE Windows Vista laptops with ICS enabled were advertising 6to4-derived routes to Mac OSX 10.4 machines
and breaking their connectivity to IPv6 sites.  We fixed that using a RA-filter on all user network ports.

After dual-stack IPv6 was enabled on most subnets, protocol 41 and Teredo were blocked at Monash's Internet border on 15-Jun-2011.  I don't recall anyone ever asking for it to be re-enabled.

I assert that the time of "experimental" IPv6 has passed.  IPv6 should be a "production" service.
If the IPv6 network service isn't almost as good (due to reliability or RTTs or whatever) as IPv4,
turn it off until you can do it right.

Thanks,
    John

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 3 Dec 2014 13:30:47 +0100

> done. RFC5571.

I guess L2TP would be a way to handle tunneling towards a gateway operated by an ISP or third parties, but it doesn't solve the entire problem, where a home gateway can set up tunnel to an appropriate gateway (i.e., the topologically closest one available) without any configuration. Also, isn't RFC 5571 about IPv4-over-IPv6 tunneling rather than the other way around? And the extra PPP layer could be an unwanted complication.

Iljitsch

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 3 Dec 2014 13:41:37 +0100

Hi Fred,

On 02 Dec 2014, at 20:17, Templin, Fred L <[email-masked]> wrote:

> provider.

Why? That's not necessary to use public 6to4 gateways. By having an out-of-the-box unique login, there is still the possibility to blacklist misbehaving devices. Black/whitelisting based on IP address is also appropriate in many cases, for instance by an ISP providing a tunnel service towards its customers.

> will give better performance and avoid single points of congestion.

Experience with Teredo shows that this can easily backfire: the setup takes long and often fails. The main issue that I have with route optimization is that it renders the whole system non-deterministic. With a tunnel broker tunnel you know with 99%+ certainty that if you can ping the gateway you can reach the entire IPv6 internet. That's an important feature.

However, my goal is not so much to create the final tunneling system that can replace all others, but rather, to allow devices to discover their available tunneling options with very little, if any, user involvement. If some of those options do support route optimization, they could be included, but only if there are sufficient mechanisms in place to make sure that when the route optimization fails, this doesn't break connectivity.

Iljitsch

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Wed, 3 Dec 2014 14:15:48 +0100

On 12/3/14, Iljitsch van Beijnum <[email-masked]> wrote:
>>> - the server matches their IPv4 address, NAT/CGN status and capabilities
>>> with available tunnel options such as brokers, 6rd, public gateways,
>>> sends back configuration info
>>> - home gateway sets up a tunnel and uses keepalives to make sure it's up,
>>> go back to the server if the tunnel goes down for whatever reason
>
>> done. RFC5571.
>
> I guess L2TP would be a way to handle tunneling towards a gateway operated
> by an ISP or third parties, but it doesn't solve the entire problem, where a
> home gateway can set up tunnel to an appropriate gateway (i.e., the
> topologically closest one available) without any configuration.

Just run the LNS on 192.88.99.1.

> Also, isn't
> RFC 5571 about IPv4-over-IPv6 tunneling rather than the other way around?
> And the extra PPP layer could be an unwanted complication.

Designing, implementing, debugging and deploying a completely new
protocol on billions of devices is simpler ?

--a

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 14:25:35 +0100

On 2014-12-03 13:41, Iljitsch van Beijnum wrote:
[..]
> However, my goal is not so much to create the final tunneling system that can replace
> all others, but rather, to allow devices to discover their available
tunneling options
> with very little, if any, user involvement.

Which demographic of users need this?

Please provide a list of requirements and how current solutions do not
match those requirements or how those requirements could be met.

Noting also that an ISP that is willing to provide any kind of tunnel
service can do so perfectly with 6rd, the discovery being solved with a
mere DHCP option.

Greets,
 Jeroen

---

From: Keith Moore <[email-masked]>
Date: Wed, 03 Dec 2014 10:46:47 -0500

> turn it off until you can do it right.

The problem with that statement is that IPv4 is rapidly getting more and 
more "experimental", and less and less predictable and robust, as 
providers install more and more middleboxes (including LSNs) that 
implement more and more kludges, more and more layering violations, 
etc.   There seems to be a strong tendency for providers to say "if the 
web works and email works, that's good enough for most people; anything 
else is a special case, but by default it's okay to screw those 
users".   Which just makes IPv4 more and more hostile to applications.   
>From the point of view of some kinds of applications developers, IPv6 is 
starting to look much saner than IPv4, even if the IPv6 access often 
relies on some tunneling mechanism.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 03 Dec 2014 10:50:08 -0500

> I guess L2TP would be a way to handle tunneling towards a gateway operated by an ISP or third parties, but it doesn't solve the entire problem, where a home gateway can set up tunnel to an appropri…
I haven't evaluated RFC 5571 specifically yet.   But every other L2TP 
solution I've looked at has been a disaster, precisely because of too 
many protocol layers, too many protocol headers with too much 
overhead.   Layering things over PPP just to leverage its authentication 
is horrendous.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 03 Dec 2014 10:54:00 -0500

> Which demographic of users need this?

Every user who doesn't have native IPv6 access at every one of the 
locations from which he accesses the Internet.

Because IPv6 won't be a viable replacement for IPv4 until users can 
access it everywhere.   And in the near term, that means being able to 
access it from anywhere that currently has IPv4.

Keith

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:01:18 +0000

Hi Iljitsch,

> From: Iljitsch van Beijnum [mailto:[email-masked]]
> Sent: Wednesday, December 03, 2014 4:42 AM
> To: Templin, Fred L
> Cc: [email-masked]
> Subject: Re: [v6ops] Report: Bar BoF on a 6to4 replacement
> 
> Hi Fred,
> 
> Why? That's not necessary to use public 6to4 gateways. By having an out-of-the-box unique login, there is still the possibility to
> blacklist misbehaving devices. Black/whitelisting based on IP address is also appropriate in many cases, for instance by an ISP providing
> a tunnel service towards its customers.

I am interested in an ISP-independent tunnel service solution. So, the tunnel service
would accept tunneled packets coming from an arbitrary ISP even though it is not in
any way associated with that ISP. For that, the device needs to be enrolled with the
tunnel service provider.

That said, AERO can certainly be used in an ISP-specific deployment as well that does
not accept tunneled packets from other ISPs. In that case, I guess you are right that
an out-of-the-box solution is OK.

> Experience with Teredo shows that this can easily backfire: the setup takes long and often fails. The main issue that I have with route
> optimization is that it renders the whole system non-deterministic. With a tunnel broker tunnel you know with 99%+ certainty that if
> you can ping the gateway you can reach the entire IPv6 internet. That's an important feature.

With AERO, the route optimization setup is conducted in parallel with actual
data packet transmissions and without diverting to the route optimized path
until everything has been verified to be working correctly. So, if route
optimization fails data packet transmissions continue without disruptions via
the dogleg path. And, after route optimization is completed the system can
always fall back to using the dogleg path in case something breaks.

> However, my goal is not so much to create the final tunneling system that can replace all others, but rather, to allow devices to
> discover their available tunneling options with very little, if any, user involvement. If some of those options do support route
> optimization, they could be included, but only if there are sufficient mechanisms in place to make sure that when the route
> optimization fails, this doesn't break connectivity.

Right; AERO provides that. And, it can be seen as a final tunneling system
that can replace all others. Thanks for bringing up the concept of an ISP
specific AERO deployment, which I guess could be used instead of 6rd
and others.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:04:01 +0000

> From: v6ops [mailto:[email-masked]] On Behalf Of Jeroen Massar
> Sent: Wednesday, December 03, 2014 5:26 AM
> To: Iljitsch van Beijnum
> Cc: [email-masked]

6rd is not perfect, because it clamps the MTU to something less than 1500.
Tunnels should not clamp the MTU, such that 1500s make it through
deterministically and 1501+ are allowed to sink or swim on their own.

Thanks - Fred

---

From: Tim Chown <[email-masked]>
Date: Wed, 3 Dec 2014 16:05:27 +0000

> any way associated with that ISP. For that, the device needs to be enrolled with the

What will this give functionally that a tunnel broker does not?

Tim

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:13:39 +0000

Hi Tim,

> From: Tim Chown [mailto:[email-masked]]
> Sent: Wednesday, December 03, 2014 8:05 AM
> To: Templin, Fred L
> Cc: Iljitsch van Beijnum; [email-masked]

Route optimization, distributed mobility management, traffic engineering, security,
multi-access handovers (e.g., WiFi/Cellular), routing control, etc.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 17:15:12 +0100

> access it from anywhere that currently has IPv4.

That demographic of users are signing up to tunnel brokers and VPN
providers. As such, I still do not see a need for any other service
where those folks cannot use this solution that is in place for years
already.

One of the many reasons we (SixXS) ask for a "signup reason" is to
measure the reason why people sign up.

Number one for the last year and a bit:
  My ISP put me behind DSLite and I cannot access my home network
  anymore, getting an IPv6 tunnel solves that.

That solution works and clearly is not too complicated.
Note that a lot of folks seems to deploy both TSP tunnels and AYIYA
tunnels on their Android devices using the android clients for that
platform.

As such, please list the requirements that those folks need and how
these requirements are not being met with existing solutions.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 17:16:27 +0100

On 2014-12-03 17:01, Templin, Fred L wrote:
[..]
> any way associated with that ISP.

The tunnel service becomes the ISP.

And no ISP should ever accept 'arbitrary' packets.

Something about source filtering and somebody needs to pay for those
packets...

Greets,
 Jeroen

---

From: Tim Chown <[email-masked]>
Date: Wed, 3 Dec 2014 16:17:30 +0000

> multi-access handovers (e.g., WiFi/Cellular), routing control, etc.

Seems to be a plan to cure all ills with one pill :)

Is this proposal also including 6-in-6?

Tim

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 17:17:56 +0100

On 2014-12-03 17:04, Templin, Fred L wrote:
[..]
> deterministically and 1501+ are allowed to sink or swim on their own.

I do still do not see why that a MTU of < 1500 but >1280 is a limitation
in any way.

Note that most 6rd deployments have a MTU of 1480 (just the IPv4
overhead) and thus even if you would have 10 layers of IPv6 tunneling
happening you would be completely fine.

Greets,
 Jeroen

---

From: Keith Moore <[email-masked]>
Date: Wed, 03 Dec 2014 11:20:01 -0500

On 12/03/2014 11:15 AM, Jeroen Massar wrote:
> providers.
No, they're not, because it's just too obscure and painful for most 
people.    We need something that can be easily configured, that OS 
vendors can ship as part of their products, that works with the local 
ISP when it provides that service.    In short we need something that 
ordinary users can have enabled by default and it "just works".   Adding 
additional hoops that users have to jump through is effectively just a 
form of hostility to IPv6 deployment.

Keith

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:20:49 +0000

Hi Tim,

> Sent: Wednesday, December 03, 2014 8:18 AM
> To: Templin, Fred L

Yes.

> Is this proposal also including 6-in-6?

Yes. Any IP-in-IP protocol version combination is fine.

Thanks - Fred

---

From: John Mann <[email-masked]>
Date: Thu, 4 Dec 2014 03:21:16 +1100

Hi,

On 3 December 2014 at 22:41, Metzler, Dan J <[email-masked]> wrote:

>  Hi,
>
>
>
> A few thoughts…
>
>
>
> ‘After dual-stack IPv6 was enabled on most subnets, protocol 41 and
> Teredo were blocked at Monash's Internet border on 15-Jun-2011.  I don't
> recall anyone ever asking for it to be re-enabled.’
>
> -          Not necessarily a problem, “IF” Monash does not provide
> internet access to other ISPs.  If it does, then were they all dual-stack
> IPv6 enabled?  If not then I would point out that this broke IPv6 service
> for some downstream customers of those ISPs.  The only reason for an ISP to
> turn off Teredo and protocol 41 is if all downstream customers have access
> to native IPv6 connectivity, and even then that decision is questionable.
>

Yes, you are right.  Monash does provide Internet transit for some
downstream organisations.
I only blocked protocol 41 and Teredo for Monash's IP address ranges.

 -          It is unlikely that anyone would ask for it to be re-enabled.
> They would more likely incorrectly assume that IPv6 is broken and disable
> it.  The reason for this is that the vast majority of people, who were
> affected by this, likely would not have had the technical skill to trace it
> back to Monash.  Before they ever got to that point someone would likely
> tell them to disable IPv6, or all tunneling protocols thus completing the
> breakage of their IPv6 connectivity.
>

I don't see how blocking tunnel protocols at Monash's border impacts any
user or server outside Monash ...
Users/servers outside Monash can still use 6to4 or Teredo tunnels -- if
those tunnels terminate outside Monash somewhere.

Monash users with native IPv6 and would not have needed any tunneling for
general Internet access.
Only Monash users could have been affected by the block, called our
Helpdesk with "The Internet is slow",
been triaged, and got through to people with technical skills.

The people who could have wanted tunnels re-enabled might be e.g.
- a student with a NAT home router in their dorm room who wants to get to
some IPv6 service (over Teredo)
  [ Mind you, a PC at Monash behind NAT -> Teredo gateway in Seattle -> Web
server at Monash on public IPv6 and back  was about 400ms RTT. ]

> -          If the IPv6 network service isn’t almost as good (due to
> reliability or RTTs or whatever), and it is under your control to fix it,
> then please fix it.
>
> -          If it isn’t under your control, then communicate it to your
> ISP or the offending party and ask them to fix it.
>
> -          “Turn it off” is the wrong answer, though probably the more
> popular answer.  Turning it off breaks it more completely than it was with
> poor performance.
>

Perhaps I wasn't clear.  I was talking as the operator of the network
service.
By "turn it off" I meant "disable IPv6 on their local subnet router
interface; don't let IPv6 RAs get to the user's devices".

We do actively work on fixing IPv6 problems.
But some vendors can take many years and many attempts before they get IPv6
support up to approximate feature and performance parity with IPv4.

In the meantime, we disable the "IPv6 network service" so that users don't
have a bad network experience and then have to disable IPv6 in their
devices to get stuff done.  And the users never learn that disabling IPv6
can sometimes fix problems.

Thanks,
    John

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 17:24:00 +0100

On 2014-12-03 17:13, Templin, Fred L wrote:
> Route optimization,

This seems to be listed, but your "AERO" documents do not describe it at
all how this could ever work in practice let alone what the positive
outcome would be.

If any type of it will work, it will still be limited to the single ASN
unless you convince other ASNs to be part of your network policy, at
which point that is a single ASN.

> distributed mobility management,

As every node in such a network would need to know about it, when will
this collapse? Something about scalability, which also goes for the
previous point.

> traffic engineering

But you also have "route optimization", who controls this then?

> security,

What kind of "security"?

> multi-access handovers (e.g., WiFi/Cellular)

Most VPN protocols handle this perfectly already.

> routing control, etc.

Is that not the same or in contrast to your "Route optimization" point?

Greets,
 Jeroen

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 3 Dec 2014 17:23:57 +0100

>  anymore, getting an IPv6 tunnel solves that.

???

DS-lite is IPv4 over IPv6, so DS-lite users would already have native IPv6...

> these requirements are not being met with existing solutions.

Anyone who is behind a (CG) NAT that doesn't allow for incoming connections for peer-to-peer applications such as VoIP and video chat.

Yes, those users could use an existing tunnel broker. But most of those users have no idea how to install a tun0 device.

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:28:10 +0000

Hi Jeroen,

> From: Jeroen Massar [mailto:[email-masked]]

Hosts have become conditioned to expect 1500. So, if they send a
1500 and it gets dropped but no ICMP PTB message comes back it
black holes.

> happening you would be completely fine.

How do the other layers of tunnels know what to clamp their MTUs
to?  Since it would be tunnels over IPv6, the first inner tunnel would
have to know that it needs to clamp is MTU to 1440. Then, the next
inner tunnel would have to know that it needs to clamp its MTU to
1400, etc. How can nested tunnels so carefully clamp their MTUs when
ICMPs may be lost? Unless some single administrative authority is very
carefully managing each tunnel, I think they can't.

On the other hand, tunnels that always provide an assured 1500 recurse
indefinitely. Those that clamp the MTU cannot.

Thanks - Fred

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 17:34:18 +0100

On 2014-12-03 17:20, Keith Moore wrote:
> people.

"Most people". Interesting choice of words.

Please note that in a lot of countries due to 'legal restrictions' and
other paranoia a lot of people are signing up for VPN services.
It happens a lot and they just click them together.

Also please describe in detail how tunnel brokers are "obscure".
google(I want IPv6) gives you a LOT of details.
Even http://cs.brown.edu/~adf/cerf/ pops up ;)
and of course https://en.wikipedia.org/wiki/List_of_IPv6_tunnel_brokers

Also note that most of the better tech magazines have detailed point by
point instructions on how to get IPv6 connectivity for well over a
decade. I have a stack of copies of these kind of articles where SixXS
was featured in and where other Tunnel Brokers where described in too.

And describe how they are "painful" for most people.

If you actually want to resolve those points, you'll have to come with a
lot more detail for anybody to be able to look at your problem.

> We need something that can be easily configured, that OS
> ISP when it provides that service.

That is called native IPv6.

And otherwise IPv6 over PPP(oE) or things like 6rd.
Anything else would require further cooperation.

If those ISPs wanted to deliver IPv6 they would be doing so. As they are
obviously not, choose with your money and go elsewhere if you want it
from them.

> In short we need something that
> ordinary users can have enabled by default and it "just works".

Please also note that "just works" does not exist.

> Adding

Simple question: how did you sign up for your IPv4 service?

Indeed, you needed to sign up for that, likely sign some paper work, get
a device installed etc.

Oh, and remember the hell that was to get it all working when it was
still Trumpet Winsock over a modem connection? :)

As that ISP is not providing your with IPv6, you will need to do the
same if you want IPv6 from another service provider.

Greets,
 Jeroen
   (who did do a IPv6 tunnel over 28k8 back then ;)

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 17:37:51 +0100

> DS-lite is IPv4 over IPv6, so DS-lite users would already have native

Yes. Exactly why I recently had to write:

https://www.sixxs.net/news/2014/#tunnelsanddslite-1008

to explain the nice misconceptions a lot of folks have.
(they think they get IPv4 back with an IPv6 tunnel...)

Note that people want an IPv6 tunnel for *OTHER* locations.
Eg their parents who do have IPv4-only (possibly behind (CG)NAT) or
their Android phone.

>> As such, please list the requirements that those folks need and
>> how these requirements are not being met with existing solutions.
> 
> Anyone who is behind a (CG) NAT that doesn't allow for incoming
> connections for peer-to-peer applications such as VoIP and video
> chat.
> 
> Yes, those users could use an existing tunnel broker. But most of
> those users have no idea how to install a tun0 device.

They do not need to know about that...

apt-get install aiccu ?

or:

https://play.google.com/store/apps/details?id=de.flyingsnail.ipv6droid

Greets,
 Jeroen
   (yes, the windows edition does suck installer wise...)

---

From: Gert Doering <[email-masked]>
Date: Wed, 3 Dec 2014 17:39:47 +0100

Hi,

> 6rd is not perfect, because it clamps the MTU to something less than 1500.

It doesn't have to.  If the transport network providing the 6rd service
(remember that this is inside a well-controlled ISP network, not 
"arbitrary Internet") provides larger-than-1500 MTU, there is no reason 
why 6rd cannot have 1500.

Gert Doering
        -- NetMaster

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 17:40:08 +0100

On 2014-12-03 17:28, Templin, Fred L wrote:
> Hosts have become conditioned to expect 1500.

Which hosts? Most IPv4 nodes are connected with a LOT less...

> So, if they send a

Then configure your network properly.

Also, do not create tunnels over paths that do not pass the packet size
your are sending.

I think I mentioned that in another thread before...

> to? 

By configuring them properly?

> Since it would be tunnels over IPv6, the first inner tunnel would

As you are so set on having tunnels inside tunnels, you are creating
that problem for yourself.

Word of advise: do not do that...

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:43:44 +0000

Hi Jeroen,

> Sent: Wednesday, December 03, 2014 8:24 AM
> To: Templin, Fred L; Tim Chown
> Cc: [email-masked]
> Subject: AERO again (Was: Report: Bar BoF on a 6to4 replacement)
> 

You must not be reading the documents I am writing. "Route Optimization"
is hard-coded in the name AERO, and well specified in the spec.

> which point that is a single ASN.

The ASN for intra-ASN route optimization is the tunnel serve provider's
ASN; not the ASN (or ASNs) from which the node receives its basic
Internet services. But, AERO also supports inter-ASN route optimization
such that an AERO node associated with ASN1 can perform route
optimization to talk directly to an AERO node associated with ASN2. 

> this collapse? Something about scalability, which also goes for the

The AERO tunnel service provider can deploy hundreds or even thousands of
AERO Servers, and AERO Clients will receive the same service whichever
Server they associate with. But, with mobility always comes deaggregation,
and the AERO Relays (running an ASN-internal instance of BGP) can handle
probably up to about 1M deaggregated prefixes. That is a lot of mobile nodes. 

> But you also have "route optimization", who controls this then?

AERO Clients can register multiple underlying ISP connections with their
AERO Servers. So, the AERO Server can accept packets coming from more
than one of the Client's ISPs. The Client can then do outbound traffic
engineering by sending certain packet types over certain ISP connections,
and it can tell the Server how to do inbound route optimization over
the multiple ISP connections.

> What kind of "security"?

Many kinds. Data origin authentication is all that is needed for nodes within
the same enterprise network trust domain. Authenticated route optimizations
in the same spirit as MIPv6 return routability. Cryptographic signatures for
environments where such are required. IPsec where we need really strong
security, etc.

> Most VPN protocols handle this perfectly already.

While still retaining a single, stable mobile network prefix for the Client? 

> Is that not the same or in contrast to your "Route optimization" point?

Routing control in order to facilitate Route optimization. Routing control is
needed by route optimization, but the converse is not true.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:48:02 +0000

Hi Gert,

> From: Gert Doering [mailto:[email-masked]]
> Sent: Wednesday, December 03, 2014 8:40 AM
> To: Templin, Fred L
> Cc: Jeroen Massar; Iljitsch van Beijnum; [email-masked]

We are now talking about IPv4 link MTUs being perfectly managed. If they
are, then great. But, all it takes is one mis-configured link MTU to bring
down the whole house of cards. And, anything that can be configured
can be mis-configured.

thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 16:53:53 +0000

Hi Jeroen,

> Which hosts? Most IPv4 nodes are connected with a LOT less...

Not on my networks they aren't. Go to any host on WiFi or Ethernet
and you will see 1500 (either IPv4 or IPv6).

> Then configure your network properly.

Again, a house of cards.

> Also, do not create tunnels over paths that do not pass the packet size

You create tunnels over whatever path with no pre-conceived notion
of what MTUs the links in those paths support. All you can be reasonably
assured of is 1280 bytes for IPv6 and 68 bytes for IPv4.

> By configuring them properly?

See below.

> Word of advise: do not do that...

Nested mobile networks need tunnels-within-tunnels. Anyone who VPNs
into their corporate network from their home network (supported by a
tunnel) needs tunnels-within-tunnels.

We need support for tunnels that recurse indefinitely, because sure as
dawn breaking someone is going to do it.

Thanks - Fred

---

From: Gert Doering <[email-masked]>
Date: Wed, 3 Dec 2014 17:59:00 +0100

Hi,

> down the whole house of cards. And, anything that can be configured

Different can of worms.  Of course misconfiguration happens, but that doesn't
make the statement "6rd clamps the MTU to something less than 1500" any
more valid.  It doesn't.  

"6rd over infrastructure with IPv4 MTU smaller than 1520 clamps the MTU to 
less than 1500", that is obviously true, but a very different statement.

Gert Doering
        -- NetMaster

---

From: Keith Moore <[email-masked]>
Date: Wed, 03 Dec 2014 12:01:20 -0500

> "Most people". Interesting choice of words.

It's only interesting because the solutions you seem to propose are not 
suitable for most people, even though we want everyone (not just "most 
people") to migrate to IPv6.

> Also please describe in detail how tunnel brokers are "obscure".

I'm not going to play that game.   But if you want to collect data 
yourself, poll N randomly selected Internet users and ask them if they 
understand what "tunnel broker" means or how to set one up.

> decade.

Most users do not read "better tech magazines".

You seem to have this strange idea that IPv6 is for techies only.

> And describe how they are "painful" for most people.

If the option to enable v6 connectivity doesn't appear in the user's 
computer's or router's Settings / Control Panel / whatever screen, and 
it can't be automatically configured using information from PPP or DHCP, 
it's too painful for most people.   Current options other than 6to4 
typically require users to install new software and learn new things, 
perhaps multiple layers of new things, in order to use them.

> lot more detail for anybody to be able to look at your problem.

I don't need to get other people to look at "my" problem.   (which is 
the Internet's problem even if some people don't realize it) I'm quite 
capable of doing the protocol design work or assisting with it and 
willing to make that effort.   What I'm looking for now is just 
assistance in defining the problem.

> That is called native IPv6.

Native IPv6 is the desired end point.   But the idea that people are 
going to use IPv6 on a widespread basis before it's effectively 
available /everywhere/ is just delusional.   So your presumed deployment 
model assumes that all providers of IP service will just upgrade to IPv6 
voluntarily.   Even if most ISPs do that, I don't think most hotels, 
coffee shops, airports, etc. that provide wifi access are going to 
eagerly do that.   What's much more likely is that they'll just replace 
their equipment as it fails, and if the new equipment supports IPv6, and 
their ISP supports IPv6, it will work.   And these are vital parts of 
Internet infrastructure.  And until IPv6 access is ubiquitous via some 
means, applications will still be stuck using the same extremely 
dysfunctional (and constantly getting worse) IPv4 Internet.

> Please also note that "just works" does not exist.

Nope.  It only results from good design, good network implementation, 
good operation, etc.   "just works" is only a user perception, but it's 
absolutely the perception that users should have.
> Simple question: how did you sign up for your IPv4 service?

On the most recent occasion, I spent weeks trying to find a provider 
that would give me a static IP address and not filter my traffic, more 
weeks trying to get the link and their router configured properly to 
deliver the service that they promised me.   Why do you ask?

> still Trumpet Winsock over a modem connection? :)

Even then I knew better than to use Windows, but I do remember those 
days.  What is your point?   That getting Internet service is supposed 
to be painful?   Or that users should have to go through every bit as 
much pain to enable IPv6 as they did to get IPv4 access?

Keith

---

From: Tore Anderson <[email-masked]>
Date: Wed, 3 Dec 2014 18:03:30 +0100

* Gert Doering

> It doesn't have to.  If the transport network providing the 6rd
> service (remember that this is inside a well-controlled ISP network,
> not "arbitrary Internet") provides larger-than-1500 MTU, there is no
> reason why 6rd cannot have 1500.

Actually, since 6RD is stateless, the BRs have no way of knowing if any
given CE will support 1520 octets long mini-jumbos. So unless the ISP
somehow forbinds the user from bringing his own CE, it pretty much has
to assume 1500 bytes is the largest that can be relied upon to work.

In any case, the use case for 6RD is typically an ISP with a bunch of
rusty old DSLAMs or whatever that just can't support IPv6, and those
probably don't support mini-jumbos any better...

FWIW, the only ISP in Norway that has 6RD does TCP MSS clamping at the
BR as well as ensuring their own CEs advertise a LAN MTU of 1480 to
limit the inevitable PMTUD problems. That makes it work sufficiently
well.

Tore

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 18:05:39 +0100

On 2014-12-03 17:48, Templin, Fred L wrote:
[..]
> are, then great.

In a provider network this is the case.
There is no other case where 6rd can be deployed.

> But, all it takes is one mis-configured link MTU to bring

There is no magic bullet for those situations unless you do probing.

And if you on misconfigure things then even probing will not solve that
problem.

[..]
> and you will see 1500 (either IPv4 or IPv6).

And then check one hop further and the hop behind that and bingo, not
1500 any more, the reason why it is called Path MTU.

But lets take an actual number from what SixXS users configure by
looking at the MTU statistics:

https://www.sixxs.net/misc/mtu/

Note that most people (94.78%) do not bother to change it as 1280 works
perfectly fine for them. The rest has a range of anything in between
with 47.70% of the folks who do change it forcing it to 1480, which is
the maximum they can as they indeed have a 1500 MTU link. The 1472 is
likely all the tunnels with AYIYA overhead.

As shown though some people chose to set it as low as 1281; no idea why
they do that.

We even have a nice FAQ article about how to determine the MTU:
 https://www.sixxs.net/faq/connectivity/?faq=mtu

Note that this is indeed manual, but as the numbers show, only few
people see a big need to change their MTU.

>> Then configure your network properly.

Well, your network.

And really, stating "house of cards" but wanting to do multiple tunnels
inside tunnels, ouch...

> assured of is 1280 bytes for IPv6 and 68 bytes for IPv4.

But that is the fun thing with Tunnel Brokers.

They default at 1280 and allow users to configure them higher when the
path allows.

And 1280 always works it seems: never heard anybody complain in the near
15 years that we got SixXS (+IPng.nl) up and running about that.

[..]
> Nested mobile networks need tunnels-within-tunnels.

They only need them because you are chosing that.

I can only suggest to chose something different that is a better solution.

> Anyone who VPNs

No. They only need 1 tunnel over their existing IPv4 infrastructure.

I really do not see a second tunnel there.

Though you could do a IPv6 tunnel over that of course if your VPN still
does not support IPv6 "natively" in the tunnel.

> dawn breaking someone is going to do it.

It seems you are determined to do so.

My position remains though: avoid it!

Something about a house of cards and especially the fun of debugging all
of that.

Greets,
 Jeroen

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Wed, 03 Dec 2014 15:07:56 -0200

Apologies for top posting, but

I don't think the demographics is limited to those who are using tunnel
brokers. It's offering operators who are stuck with non-IPv6 compliant
equipment in their networks an alternative with basically the strengths
of 6rd and addresses its weaknesses.

regards

Carlos.

On 12/3/2014 3:01 PM, Keith Moore wrote:
> the Internet's problem even if some people don't realize it)    I'm
> quite capable of doing the protocol design work or assisting with it and

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 18:21:08 +0100

> people") to migrate to IPv6.

These folks will move when they need to. They do not care if their
"Internet" is IPv4 or IPv6 as they just use port 80/443, that is they
just use "Facebook" and "Google".

> I'm not going to play that game.

Which "game"?

The "no answer to a statement you are making" one?

> But if you want to collect data

I can state that 99% of the users of SixXS will not know how to set up a
Tunnel Broker.

Even though 100% of them clearly know how to create a tunnel; and this
funnily while a tunnel broker is just the automated version of that ;)

The "normal internet users" you are speaking of do not care about
IPv4/IPv6, they care about "cat pictures"...

Hence, they will not "need IPv6".

> Most users do not read "better tech magazines".

And thus that demographic is not interested in getting IPv6 either.

There is a big reason why I asked for the demographic you are talking
about. Clearly you are talking about people who Google for Facebook.

These folks really can't care less about this whole IP thing or whatever
we do in the IETF.

They will love these kind of threads though, wondering why we waste our
time discussing them.

> You seem to have this strange idea that IPv6 is for techies only. 

I for sure do not seem to think that, having helped close to 45.000
people getting IPv6 at locations they do not have it. Most who
definitely are not 'techies' but just normal users at home who did
something fun to get to learn something new./

> it's too painful for most people.

IPv6 is enabled per default on most (unfortunately by far all) devices
being kicked out of the door today.

> Current options other than 6to4

Your demographic of users do not care about these things and thus will
not bother with them.

> I don't need to get other people to look at "my" problem.

Then why are you discussing it here and are you so opposed to
deprecation of 6to4?

> (which is

But you have that definition of the problem (which is far from clearly
described).

Unfortunately it seems the demographic for your problem does not care
about your problem.

> available /everywhere/ is just delusional.

Nobody is going that route.

6bone and tunnel brokers where a great way of bootstrapping IPv6 and
getting it in the hands of the people who needed/wanted to test and play
with it.

6bone was decommissioned when we knew it worked.

Tunnel Brokers are still there to let people actually use IPv6 for the
situations where the user wants it.

> So your presumed deployment
> voluntarily.

They will over the next 50 years.

> Even if most ISPs do that, I don't think most hotels,
> eagerly do that.

Those are restricted environments that will not easily allow tunneling
anyway. Thus you won't be able to bolt IPv6 there even if your life
depends on it (though AYIYA typically works great, heck even worked from
many hotels in China and even the subway :)

> What's much more likely is that they'll just replace

Absolutely. But what can the IETF do about that?

> absolutely the perception that users should have.

That is a fairy dream outcome. Not even IPv4 works in that way.

> deliver the service that they promised me.   Why do you ask?

Because it answers the point completely:
  Do not expect IPv6 yo magically 'just work'.

Also, as you spent weeks on it, did you get IPv6 with that? :)

> much pain to enable IPv6 as they did to get IPv4 access?  

That it *IS* painful and that you should not expect magic from something
that is going to run above it.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 18:23:34 +0100

On 2014-12-03 18:07, Carlos M. Martinez wrote:
> brokers.

Please define the exact demographics that you are talking about.
As they are not at all matching with Keith's edition of "everybody"

> It's offering operators who are stuck with non-IPv6 compliant
> equipment in their networks

As "operators" knew that this little IPv6 thing was coming over the last
20 years. How did they end up there? How did they not upgrade their
network over time and prepared for this?

> an alternative with basically the strengths

Which 'weaknesses'? Please define.

Greets,
 Jeroen

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Wed, 03 Dec 2014 16:10:07 -0200

Hello Jeroen,

> As they are not at all matching with Keith's edition of "everybody"
It's definitely not everybody, but it's certainly bigger than just the
guys who know what tunnel broker means.
> network over time and prepared for this?
We can talk for hours how they ended up in this situation. But its true.
I know first hand examples of FTTH deployments done in 2013 who are
stuck with OTNs that have hardware bugs that prevent them from
transporting native IPv6 (if you want the details, the OTNs blindly drop
anything multicast). This particular ISP has close to a million users,
and they are willing to do IPv6, but they are, well, stuck.

These are just a few examples, but i'm pretty sure that there are many more.
> Which 'weaknesses'? Please define.
OAM, mostly.

Carlos

---

From: Keith Moore <[email-masked]>
Date: Wed, 03 Dec 2014 13:14:58 -0500

I followed up to Jeroen in detail in private mail.

Executive summary:  He seems to think there's no need for an 
easy-to-configure way to enable v6 connectivity for arbitrary hosts or 
consumer networks because those people who can't be bothered to install 
and configure SixXS just want to look at pictures of cats anyway.   I 
disagree, and think that large numbers of ordinary users benefit from 
availability of applications other than the web, which run better over 
v6 than with ad hoc mechanisms for NAT traversal. I also think that the 
widespread support of anycast 6to4 in hosts and consumer routers was 
evidence of at least a perception of a need for easy v6 connectivity 
over v4.   Since anycast 6to4 has been demonstrated to not work well and 
6to4 with an explicitly configured router still has other problems, I 
want to define a different mechanism that works better but still has 
6to4's ease of configuration.   I am however happy if all we need for 
that mechanism is a way that networks can advertise, and hosts and 
consumer routers can discover, available mechanisms for tunneling v6 
over v4, and if the hosts/routers can be configured to use a fallback 
tunnel service if no other means is available.  The advertising 
mechanism should be able to advertise 6rd, AYIYA, and anything else that 
works well.

Keith

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 19:18:20 +0100

> guys who know what tunnel broker means.

That is not very specific at all.

> anything multicast).

Clearly they need to ask their money back from that vendor...

> This particular ISP has close to a million users,

6rd does not rely on multicast, hence there is one solution that is very
well tested.

>> Which 'weaknesses'? Please define.

Please be a lot more specific.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 19:40:53 +0100

> I followed up to Jeroen in detail in private mail.

Why not keep it on list?

> and configure SixXS just want to look at pictures of cats anyway.

Clearly that is what you interpreting from what I am trying to get out
of you. But that is not what I have stated at all and is actually gross
misrepresentation.

First of all, I do not limit to SixXS (which is just an implementation
to solve a particilar set of problem with a specific demographic
audience), let alone to Tunnel Brokers for finding a solution for the
problem.

One of the major problems with your mails is that the demographic that
you want is completely unclear. As you say "most people", the only
answer can be "folks using HTTP" (thus watching cats, facebook, gmail
and the like). These folks do not even care about SMTP as their email is
a website and they do not know better.

Those people really do not care about IPv4/IPv6 etc as for them "The
Internet" (as they know it) just works fine, even if they are behind
fifty layers of NATs or heck are getting native IPv6-only already.

Hence, please define your demographic and then define the problem they
are having and where they are having that problem.

As only then will we be able to determine where the problems are and
then go for a solution to solve that problem.

> I disagree, and think that large numbers of ordinary users benefit from
> v6 than with ad hoc mechanisms for NAT traversal.

Like you I love and want that people can use the FULL Internet even
though most people will never know the full possibilities that it offers.

Hence, why I have been deploying IPv6 for so long already: to get people
to be able to deploy things. (of course I wanted to do that myself too,
just happened to end up doing it for others too ;)

But the people who specifically need these kind of setups either read
magazines like the paper editions of www.heise.de/ct/ and
www.heise.de/ix/ or PC Magazin etc who have been covering IPv6 for years
already; or those people google/yahoo/bing and find a solution to their
problem, typically not even a IPv6 tunnel or other form of connectivity
but things like getting a VPS, TeamViewer, Hamachi or a form of a
peer-to-peer network (and then I mean bittorrent not true p2p which is
what one gets with unfirewalled IPv6).

Any other demographic simply does not care about IP.

If you have examples of situations of people who do care about having
proper connectivity to the Internet, then please name them and let us
define them and then see what kind of problems these people have.

> I also think that the
> over v4.

Yes, there was this. But we learned that Anycast 6to4 causes problems
that we cannot solve as there are too many layers of operation involved.

Also, since 6to4 came out there have been better solutions that solve
that problem much better. Remember that when 6to4 was developed the only
NAT that existed was the one in the computer of the user, not in the
network of the ISP. There where no mobile networks either and most folks
had dial-up.

> Since anycast 6to4 has been demonstrated to not work well and
> 6to4's ease of configuration.

But why define something if you cannot even define who will be using it
let alone the problems they currently have?

I'll ask again:
 Which exact gap are you trying to solve that existing methods do not
 solve? And of course, please do specify the demographic.

> I am however happy if all we need for
> tunnel service if no other means is available.

6rd uses DHCP for 'advertising', this works great on the local media to
which an ISP has access.

Anything else will require somebody to run a service where possibly
millions of people can get coordinated access to that service.
That system will not happen as that has the same problem as 6to4
anycast: other entities are paying for that service.

Note that the 'advertising' of tunnel brokers happens in printed media /
blogs and maybe strangely more importantly in the devices that people buy.

Signup reasons that I see regularly when going through the SixXS signup
queue prove this:
 "My Sophos UTM pointed out to this"
 "UnityMedia helpdesk sent me here"
 "My ZyXEL supports SixXS"
 "I saw AICCU support in OpenWRT"
 "I installed the aiccu package"
etc

(Oh, fun side-note: SixXS is just a hobby project that takes too much
time.... hence no real advertising in any way)

> The advertising

Any such "advertising" mechanism would require central coordination,
such a mechanism does not exist on the Internet and cannot be
distributed easily around the Internet in a scalable method.

Somebody has to run it, there is no way to make something run unattended
and unmanaged and without it costing anybody anything.

Greets,
 Jeroen

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 3 Dec 2014 18:47:57 +0000

> So I guess I have a draft to write.

You and I didn't have that conversation. Yes, I would agree that you have a draft to write.

The big question I would ask is about diversion of effort. It's not like 6to4 is universally deployed in any real sense, or for that matter Teredo (it's in many, perhaps all, Microsoft products, but not everything comes from Microsoft). With actual use plummeting (Brian indicates that worldwide statistics suggest a grand total of 100K people are using 6to4 anycast and that number is falling), I'm not sure there is a market for a replacement. Software is typically updated when the underlying hardware is changed; the deployment model becomes "when product X is replaced, the user simultaneously gets IPv6 and 6to4bis". If a given operator has to do something to deploy 6to4bis, why would they not enable IPv6 instead? If the choice is to spend $$$ to deploy IPv6 xor 6to4bis, do we want to encourage them to choose 6to4bis?

I think I see where you're going, and if you want to do it I’m not going to object, but to me it's a dead end.

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 18:53:17 +0000

Hi Gert,

> Sent: Wednesday, December 03, 2014 8:59 AM
> To: Templin, Fred L
> Cc: Gert Doering; Jeroen Massar; Iljitsch van Beijnum; [email-masked]

6rd says:

   "If the MTU is well-managed such that the IPv4 MTU on the CE WAN side
   interface is set so that no fragmentation occurs within the boundary
   of the SP, then the 6rd Tunnel MTU should be set to the known IPv4
   MTU minus the size of the encapsulating IPv4 header (20 bytes).  For
   example, if the IPv4 MTU is known to be 1500 bytes, the 6rd Tunnel
   MTU might be set to 1480 bytes.  Absent more specific information,
   the 6rd Tunnel MTU SHOULD default to 1280 bytes."

I do not know, but I suspect many 6rd installations clamp the tunnel MTU
to 1480 because "1500 everywhere" is assumed and for the most part that
assumption would be valid. To get packets between 1480-1500 through,
though, 6rd would need to use fragmentation unless and until the path
tests positive for passing unfragmented 1500 byte packets. Then, the
fragmentation can be turned off.

So yes, hope for well-managed large link MTUs. But, test for them first
before blindly assuming they are well managed.

Thanks - Fred

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 03 Dec 2014 18:57:24 +0000

On 03/12/2014 18:14, Keith Moore wrote:
> He seems to think there's no need for an easy-to-configure way to enable v6
> connectivity for arbitrary hosts or consumer networks because those people
> who can't be bothered to install and configure SixXS just want to look at
> pictures of cats anyway.

that is largely true, but maybe he missed an important point: if you want
to define a new tunnelling mechanism, you could begin to expect real world
deployment 5-8 years down the road, assuming you get vendors on board.

So the question is not whether we should to start discussing new transition
/ tunnelling mechanisms, but whether the IETF think it's more appropriate
to aim to start pushing yet another new tunnelling mechanism 5-8 years from
now, or whether it would be more useful to push for more native v6 now.

> I want to define a different mechanism that works better but still has

This is a monumental waste of time.

Nick

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 19:10:31 +0000

Hi Jeroen,

> Sent: Wednesday, December 03, 2014 9:06 AM
> To: Templin, Fred L
> Cc: [email-masked]

Even in a provider network, well-managed link MTUs still need to be tested.

> There is no magic bullet for those situations unless you do probing.

Probing to test, yes.

> And if you on misconfigure things then even probing will not solve that

No, the probing I am talking about only returns success if things are
not misconfigured. If they are misconfigured, then you need to do
a little bit of fragmentation to get the 1500's through.

> 1500 any more, the reason why it is called Path MTU.

Most networks I am aware of pass 1500's.

> As shown though some people chose to set it as low as 1281; no idea why

If you set a 1280 MTU on the AYIYA tunnel, then there is no way to run an
IPv6 VPN from within the end user site since the IPv6 layer inside the VPN
will only see 1240 minus the IPsec overhead (while it needs to see 1280).
I see plenty of use cases for wanting to create an IPsec tunnel that rides
within the AYIYA tunnel, but they won't work if they can't get a 1280 MTU
to the inner IPv6 layer.

> And really, stating "house of cards" but wanting to do multiple tunnels

No, the AERO handling of nested tunnels plows through any path MTU
limitations. That is because AERO tunnel MTUs recurse indefinitely.

> They default at 1280 and allow users to configure them higher when the

And when the path changes due to a routing change?

> 15 years that we got SixXS (+IPng.nl) up and running about that.

Again, 1280 for the AYIYA tunnel means 1240 or less for other nested
IPv6 tunnels, which doesn't work.
 
> I can only suggest to chose something different that is a better solution.

When a mobile router A comes onto a link offered by a mobile router B,
which comes onto a link offered by a mobile router C, you have nested
tunnels within tunnels. Not an avoidable situation.

> No. They only need 1 tunnel over their existing IPv4 infrastructure.

One tunnel over the existing IPv4 infrastructure to provide IPv6 access
to the site. A second tunnel over the IPv6 site to get to the VPN gateway.
Means an IPv6-in-(foo)-in-IPv6 tunnel within an IPv6-in-IPv4 tunnel.

> I really do not see a second tunnel there.

See above.

> My position remains though: avoid it!

Can't. Not without loss of generality, and not without excluding real use cases.

> Something about a house of cards and especially the fun of debugging all

I don't know what that means. There is nothing to "debug" with AERO tunnel
MTU handling, and it all just works.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 19:35:06 +0000

More on tunnel MTU clamping. Tunnels should be able to pass any IP packet size
up to the IP protocol maximum (minus encapsulation overhead) as long as the
underlying path MTU is sufficient. Clamping the tunnel MTU to any size smaller
than that will preclude moving to larger sizes (e.g., 8KB or larger).

Thanks - Fred

---

From: David Farmer <[email-masked]>
Date: Wed, 03 Dec 2014 14:20:06 -0600

On 12/3/14, 04:11 , John Mann wrote:
...
> After dual-stack IPv6 was enabled on most subnets, protocol 41 and
> recall anyone ever asking for it to be re-enabled.
...

I like to bring us back to the discussion of of filtering the route for 
192.88.99.0/24 relating to draft-ietf-v6ops-6to4-to-historic;

I would like to see some discussion around this in the draft, is it BCP, 
or at least acceptable, to block 6to4 once dual stack is deployed to end 
user?

What is the best and least destructive way to do so?  What of the 
trade-offs for the available options?  Security implications?

Block protocol 41?
Filter route to 192.88.99.0/24?

Many security people are twitchy and would like to filter protocol 41 
for security reasons.

Finally, filtering Teredo is an interesting question too, but out of 
scope for the draft in question, IMHO.

Thanks.

-- 
================================================
David Farmer               Email: [email-masked]
Office of Information Technology
University of Minnesota
2218 University Ave SE     Phone: 1-612-626-0815
Minneapolis, MN 55414-3029  Cell: 1-612-812-9952
================================================

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Wed, 3 Dec 2014 20:52:58 +0000

> evidence of at least a perception of a need for easy v6 connectivity

Thankfully, support of 6to4 and Teredo directly in consumer routers was not widespread. 

For grins and giggles I went on amazon.com and looked at the "network routers" with sort on "new and popular". Looking at just the first 9 listed to me, 4 indicated support for "IPv6". Of those 4, it looks like 2 do 6rd (and 1 does 6to4), but that was really hard to figure out. This is 3.5 years after the publication of RFC 6204 and a year after RFC 7084 (which didn't change much from 6204 related to native IPv6 and added optional 6rd requirements). And figuring this out was like pulling teeth. It's buried way down in the gory technical details of what's supported on these routers -- not in the top list of features that are prominently advertised.

For more laughs, I went looking at IPv6 usage on Comcast and AT&T. Comcast has stated that 100% of its network is IPv6 (native) enabled, and AT&T has a lot of native out there, but all customers have access to 6rd. According to http://www.worldipv6launch.org/measurements/, both are around 28% of traffic to IPv6-enabled sites using IPv6. What accounts for the difference between 100% and 28% is the number of customers who don't have IPv6-enabled (or 6rd-enabled and configured) CE routers. And I'd be willing to bet the *vast* majority of the 28% is using ISP-provided CE routers -- not retail ones. As another example, Charter Communications (who I use as one of my 2 ISPs) has a 6rd service that any of their customers can use; but almost no-one uses it. 

I have 6rd working for both my AT&T and Charter ISP connections. For both I had to get my own router and manually configure 6rd (no DHCP config from either ISP). Finding the right router was really hard and painful. Getting configuration info and inputting it (once I got the right router) was easy. Both connections work great -- no complaints from anyone in my family.

IMO
1. The only proven way to get truly widespread IPv6 usage is through ISP deployments and ISP mechanisms that upgrade ISP-provided routers in a free-to-the-user and user-does-nothing manner. Beyond upgrades to ISP-provided routers, you're depending on CE router churn to replace the embedded base.
2. We need for the "new and popular" CE routers being shipped today all to support native IPv6. Distracting the manufacturers with yet another tunnel protocol (hey, wait, vendors -- don't do IPv6 yet because we're not done with IPv6 standards!) will not be helpful in reaching this goal.
3. 4 years after publication of a new tunnel standard, you might be lucky enough to get maybe 20% of newly shipping CE routers to include support of it; and discovering which CE routers have that support is likely to be really hard. 
4. ISPs won't be upgrading the routers they provide to support the new standard (because ISPs won't be the ones deploying it since they can do 6rd if not native). You would be depending totally on users buying new CE routers who happen to stumble across one with support (since it sounds like the target demographic is people who can't spell "IPv6").
5. I was talking to one of the people at IETF from Nigeria. The concern she expressed to me about IPv6 was how to get the embedded base of CE routers (in Nigeria) replaced. Most can't be upgraded. The cost will be considerable, and there's no-one offering to pay for it.

Please don't create yet-another-tunnel protocol. Please do whatever you can to convince CE router vendors to implement native IPv6 and 6rd today.

Barbara

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 03 Dec 2014 22:50:27 +0100

> or at least acceptable, to block 6to4 once dual stack is deployed to end

IMHO one should never block anything in a network that is supposed to
give "Internet access".

> Filter route to 192.88.99.0/24?

Filtering the anycast route might be a reasonable thing to do.

But the primary reason for doing so will be to avoid helpdesk calls and
by filtering it you might actually generate them.

Though !N is a pretty good indicator that there is no route and most
stacks will disable 6to4 based on that.

> Many security people are twitchy and would like to filter protocol 41

Please do not call such people "security people". Just put them in the
'paranoia consultants' corner.

These folks will also filter anything not 80/443 because all of it is
evil and drop ICMPv6. Their network, let them mess it up.

There are several RFCs out there covering this.

> scope for the draft in question, IMHO.

Why would you want to filter it?

Teredo is a last-resort and in current Windows editions not even enabled.

Also Teredo is required for Xbox One, thus better not touch it, their
sales department has it hard enough already with that PS4 that just does
IPv4. (guess we'll have to wait another 20 years for PS6 there; this
while there where PS2s that did IPv6...).

Greets,
 Jeroen

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Dec 2014 21:50:38 +0000

> Please don't create yet-another-tunnel protocol.

Just so this comment is not taken out of context, please note that AERO is about
much more than just IPv6-over-IPv4 tunneling in ISP networks. That would be
only one domain of applicability; AERO applies across many other domains.

Thanks - Fred

> From: v6ops [mailto:[email-masked]] On Behalf Of STARK, BARBARA H
> Sent: Wednesday, December 03, 2014 12:53 PM
> To: Keith Moore; [email-masked]
> For grins and giggles I went on amazon.com and looked at the "network routers" with sort on "new and popular". Looking at just the
> first 9 listed to me, 4 indicated support for "IPv6". Of those 4, it looks like 2 do 6rd (and 1 does 6to4), but that was really hard to figure
> out. This is 3.5 years after the publication of RFC 6204 and a year after RFC 7084 (which didn't change much from 6204 related to native
> IPv6 and added optional 6rd requirements). And figuring this out was like pulling teeth. It's buried way down in the gory technical
> details of what's supported on these routers -- not in the top list of features that are prominently advertised.
> 
> For more laughs, I went looking at IPv6 usage on Comcast and AT&T. Comcast has stated that 100% of its network is IPv6 (native)
> enabled, and AT&T has a lot of native out there, but all customers have access to 6rd. According to
> http://www.worldipv6launch.org/measurements/, both are around 28% of traffic to IPv6-enabled sites using IPv6. What accounts for
> the difference between 100% and 28% is the number of customers who don't have IPv6-enabled (or 6rd-enabled and configured) CE
> routers. And I'd be willing to bet the *vast* majority of the 28% is using ISP-provided CE routers -- not retail ones. As another
> example, Charter Communications (who I use as one of my 2 ISPs) has a 6rd service that any of their customers can use; but almost
> no-one uses it.
> 
> I have 6rd working for both my AT&T and Charter ISP connections. For both I had to get my own router and manually configure 6rd (no
> DHCP config from either ISP). Finding the right router was really hard and painful. Getting configuration info and inputting it (once I got
> the right router) was easy. Both connections work great -- no complaints from anyone in my family.
> 
> IMO
> 1. The only proven way to get truly widespread IPv6 usage is through ISP deployments and ISP mechanisms that upgrade ISP-provided
> routers in a free-to-the-user and user-does-nothing manner. Beyond upgrades to ISP-provided routers, you're depending on CE
> router churn to replace the embedded base.
> 2. We need for the "new and popular" CE routers being shipped today all to support native IPv6. Distracting the manufacturers with
> yet another tunnel protocol (hey, wait, vendors -- don't do IPv6 yet because we're not done with IPv6 standards!) will not be helpful
> in reaching this goal.
> 3. 4 years after publication of a new tunnel standard, you might be lucky enough to get maybe 20% of newly shipping CE routers to
> include support of it; and discovering which CE routers have that support is likely to be really hard.
> 4. ISPs won't be upgrading the routers they provide to support the new standard (because ISPs won't be the ones deploying it since
> they can do 6rd if not native). You would be depending totally on users buying new CE routers who happen to stumble across one
> with support (since it sounds like the target demographic is people who can't spell "IPv6").
> 5. I was talking to one of the people at IETF from Nigeria. The concern she expressed to me about IPv6 was how to get the embedded
> base of CE routers (in Nigeria) replaced. Most can't be upgraded. The cost will be considerable, and there's no-one offering to pay for
> it.
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 3 Dec 2014 22:51:57 +0100

On 03 Dec 2014, at 21:20, David Farmer <[email-masked]> wrote:

> is it BCP, or at least acceptable, to block 6to4 once dual stack is deployed to end user?

I find it completely unacceptable. ISPs have no business filtering legitimate packets.

> What is the best and least destructive way to do so?  What of the trade-offs for the available options?  Security implications?

> Block protocol 41?

This would be the worst way to do it because it also blocks "manual" (non-6to4) tunnels.

> Filter route to 192.88.99.0/24?

This is also a bad way to do it because now 6to4 systems send packets into a black hole. The best way to handle 6to4 is simply to let it do it do whatever it's going to do and succeed or fail accordingly. But if you want to filter it then the least destructive way would be to run a 6to4 gateway on 192.88.99.1 for your users and have that gateway return ICMPv6 unreachables to those users.

> Many security people are twitchy and would like to filter protocol 41 for security reasons.

That's nice, but ISPs have no business making those decisions for their customers.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 04 Dec 2014 10:58:20 +1300

David,

The WGLC just concluded, and I need to update the draft accordingly.
It's clear to me (and I think to the WG Chairs) that the recommendation
to filter the anycast root does not have consenses and will be removed.
I suggest waiting until we get a new draft out before resuming the
debate.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 04 Dec 2014 11:00:00 +1300

OK, I typed too fast but I guess the typos are obvious...

Regards
   Brian

---

From: [email-masked]
Date: Thu, 04 Dec 2014 09:58:09 +0100 (CET)

> Which hosts? Most IPv4 nodes are connected with a LOT less...

I would actually like to see some statistics on this. I'm afraid I
regard the SixXS user base (and MTU info) as somewhat skewed in
comparison to the general public.

As a counterexample, I work for one of the larger ISPs in Norway. We
offer full 1500 byte MTU to all customers, including residential/DSL,
and also for IPv6. Business customers can get larger MTU on request.

Steinar Haug, AS 2116

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 04 Dec 2014 10:49:17 +0100

[Geoff, any measurements you know of?]

> comparison to the general public.

Yes, it is a quite limited view. Though at least it covers hundreds of
different ISPs where the endusers are.

The primary thing I think though that one can take away from those
numbers is that people do not really care about MTU and that they are
mostly fine with 1280. There is then also no real perceived benefit for
them as long as their stuff is 'fast'.

I don't even think a speedtest will show the difference too much between
1280 and the 1480 that a proto-41 tunnel could do, though in theory it
is a lot of overhead.

As such shoving the people who kept it to 1280 as in a "do not care"
category, only few actual changers remain.

Possibly this is a better report to look at:
https://labs.ripe.net/Members/emileaben/ripe-atlas-packet-size-matters

at least takes data from where Atlas probes are located, but those tend
to be reasonably managed networks again. Those numbers are for ICMP.

It is hard to get a good estimate of this as it involves people to
provide the data or doing some kind of magic probing around the world;

There is ISI's tr-643.pdf but unfortunately ftp.isi.edu is down and
can't seem to find a cached copy anywhere else (except for mackeeper
malware spreading sites it seems...)

Geoff wrote this a few years back:
http://www.potaroo.net/ispcol/2009-02/mtu.pdf

but that does not cover the general MTU sizes available. Of course on
the IPv6 level we do not care there it is a minimum of 1280 or pMTU.

> and also for IPv6.

Most networks that I am aware of do indeed do 1500. Though quite a few
do deploy PPPoE or similar systems and end up with 1480 there already
unless they use a larger size for the Ethernet that that runs over
(trick also employed by 6rd folks).

> Business customers can get larger MTU on request.

That is great, but are they able to use it in a meaningful way?

I know a couple of networks that run at a MTU of 9000 in the core, but
it is rare to see that intra-AS. Similar to this mystical Multicast
thing. Though I think there is an actual business case to be made for
larger MTUs unlike the business case for intra-AS-Multicast.

Greets,
 Jeroen

---

From: [email-masked]
Date: Thu, 04 Dec 2014 11:18:20 +0100 (CET)

> them as long as their stuff is 'fast'.

"People" as in residential users may not really care about MTU as long
as their Facebook and Youtube works.

I can assure you that our business users *do* care: 1500 byte MTU (or
more) is a *hard* requirement for a lot of the contracts we bid on
("hard" meaning: If we didn't offer at least 1500 byte MTU, we would
automatically be excluded from that bid).

Thus we have concluded that it's far simpler to just offer 1500 bytes
to all customers. Note that we don't use PPPoE except for a few limited
circumstances.

Steinar Haug, AS 2116

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 04 Dec 2014 11:36:15 +0100

> as their Facebook and Youtube works.

I cannot disagree at all ;)

> automatically be excluded from that bid).

Interesting. Any background on why they have that requirement?

Though likely likely don't want to rely on pMTU for at least the cases
close to home (as one cannot control that for the rest of the Internet).

> to all customers. Note that we don't use PPPoE except for a few limited

I don't see why one would offer anything else actually unless the tech
really does not support it.

Greets,
 Jeroen

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 4 Dec 2014 14:36:00 +0100

I did some tcpdumping on my server. It's not a very busy server, so I got 927 incoming TCP SYNs in two hours, mostly on port 80. 866 of them included the maximum segment size (MSS) option, 584 IPv4 and 282 IPv6.

I also looked at ICMP(v6) too bigs. I got 4 for IPv6, which looks like they were all for my own stuff (they are from the HE tunnel broker) and not a single one for IPv4. The MSS size distribution is:

IPv6 (MSS = MTU - 60 (sizeof(IPv6) + sizeof(TCP)):

1440: 47
1420: 207
1386: 1
1220: 27

Three of these sizes make total sense: native ethernet (1440), ethernet with IPv6-in-IPv4 (1420) and the minimum IPv6 MTU (1220). The only surprising thing is that tunneled is so much more than native. Note that in these cases, either the source host terminates the tunnel, the router terminating the tunnel advertises the reduced tunnel MTU or a device along the way rewrites the MSS (which is common with IPv4 but not AFAIK with IPv6).

IPv4 is a different story:

IPv4 (MSS = MTU - 40 (sizeof(IPv4) + sizeof(TCP)):

1460: 364
1452: 36
1448: 1
1440: 53
1430: 19
1420: 11
1414: 13
1410: 3
1400: 11
1392: 2
1380: 43
1360: 18
1260: 5
1240: 5

"Native IPv4" over ethernet is only 62%. Another 6% must be PPPoE. 1440 and 1240 could be IPv4-in-IPv6, which seems a lot at 10%. And then there's a whole bunch of strange values, probably numbers people picked manually.

(Note that I didn't de-duplicate IP addresses, so 10% could be 60 sessions from the same IP address.)

Note that in both cases there was nothing below an MTU of 1280 and nothing above an MTU of 1500.

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 4 Dec 2014 15:46:23 +0000

Hi Jeroen,

> Sent: Thursday, December 04, 2014 2:36 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [v6ops] MTUs on the general Internet
> 

Principal of least surprise? 1500 has become the "Internet Cell Size" that
hosts have come to expect will always succeed.

> really does not support it.

A minimum 1500 is required, whether tunneled or non-tunneled.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 4 Dec 2014 15:49:40 +0000

Hi Iljitsch,

Nice analysis. You are making a good case for the existence of the
1500 Internet Cell Size as well as MTU diversity. Now we just need
to fix the tunnels that can't do 1500.

Thanks - Fred

> Sent: Thursday, December 04, 2014 5:36 AM
> To: [email-masked] WG
> Subject: [v6ops] Quick measurement: MTUs on the general Internet
> 
> I did some tcpdumping on my server. It's not a very busy server, so I got 927 incoming TCP SYNs in two hours, mostly on port 80. 866 of
> them included the maximum segment size (MSS) option, 584 IPv4 and 282 IPv6.
> 
> I also looked at ICMP(v6) too bigs. I got 4 for IPv6, which looks like they were all for my own stuff (they are from the HE tunnel broker)
> and not a single one for IPv4. The MSS size distribution is:
> 
> Three of these sizes make total sense: native ethernet (1440), ethernet with IPv6-in-IPv4 (1420) and the minimum IPv6 MTU (1220).
> The only surprising thing is that tunneled is so much more than native. Note that in these cases, either the source host terminates the
> tunnel, the router terminating the tunnel advertises the reduced tunnel MTU or a device along the way rewrites the MSS (which is
> common with IPv4 but not AFAIK with IPv6).
> 
> "Native IPv4" over ethernet is only 62%. Another 6% must be PPPoE. 1440 and 1240 could be IPv4-in-IPv6, which seems a lot at 10%.
> And then there's a whole bunch of strange values, probably numbers people picked manually.
>

---

From: Geoff Huston <[email-masked]>
Date: Fri, 5 Dec 2014 07:15:00 +1100

On 4 Dec 2014, at 8:49 pm, Jeroen Massar <[email-masked]> wrote:

Measurement exercises for Path MTU in IPv6? I'm aware of work by Matthew Luckie and Ben Stasiewicz some years ago (http://www.caida.org/~mjl/pubs/measuring-pmtud.pdf).

In IPv6 the problem is at its worst in a situation where the large packet sender is using a 1500 octet MTU and close to the sender is a packet filter that discards IPv6 packet too big massages. If there is some form of encapsulating tunnel on the path to the receiver and the tunnel operates with an effective MTU of less than 1500 then its possible to encounter a wedged TCP session:

   the sender sends a 1500 octet packet
   the tunnel ingress cannot accept the packet and passes back IPv6 ICMP PTB message toward the sender
   the packet filter discards the IPv6 ICMP PTB message
   the sender times out and retransmits the 1500 octet

   rinse and repeat forever

This is not a receiver-side problem but an outcome of path MTU-adjusting tunnels and aggressive ICMP packet filters

As Luckie and Stasiewicz comment "A third strategy, which is arguably the most challenging, is to educate system administrators about the necessity of forwarding PTB messages"

A simpler approach is for the sender to take the marginal hit of peak throughput rate and use an initial MTU that is lower that 1500. 1280 is an obvious choice, and 1400 is probably safe as well.

  regards,

   Geoff

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 4 Dec 2014 20:24:46 +0000

Hi Geoff,

> From: v6ops [mailto:[email-masked]] On Behalf Of Geoff Huston
> Sent: Thursday, December 04, 2014 12:15 PM
> To: Jeroen Massar
> Cc: [email-masked]
> Subject: Re: [v6ops] MTUs on the general Internet (Was: Report: Bar BoF on a 6to4 replacement)
> 
> Measurement exercises for Path MTU in IPv6? I'm aware of work by Matthew Luckie and Ben Stasiewicz some years ago
> (http://www.caida.org/~mjl/pubs/measuring-pmtud.pdf).
> 
> In IPv6 the problem is at its worst in a situation where the large packet sender is using a 1500 octet MTU and close to the sender is a
> packet filter that discards IPv6 packet too big massages. If there is some form of encapsulating tunnel on the path to the receiver and
> the tunnel operates with an effective MTU of less than 1500 then its possible to encounter a wedged TCP session:
> 
> As Luckie and Stasiewicz comment "A third strategy, which is arguably the most challenging, is to educate system administrators about
> the necessity of forwarding PTB messages"
> 
> A simpler approach is for the sender to take the marginal hit of peak throughput rate and use an initial MTU that is lower that 1500.
> 1280 is an obvious choice, and 1400 is probably safe as well.

Or (since tunnels seem to be the culprit) make tunnels pass packets of at least
1500 bytes. That can be done independently of getting hosts to do something.

Thanks - Fred

---

From: "Metzler, Dan J" <[email-masked]>
Date: Thu, 4 Dec 2014 21:00:53 +0000

Hi,

> Sent: Thursday, December 4, 2014 2:15 PM
> To: Jeroen Massar
> Cc: [email-masked]
> In IPv6 the problem is at its worst in a situation where the large packet sender
> is using a 1500 octet MTU and close to the sender is a packet filter that
> discards IPv6 packet too big massages. If there is some form of encapsulating
> tunnel on the path to the receiver and the tunnel operates with an effective
> MTU of less than 1500 then its possible to encounter a wedged TCP session:
> 

Wouldn't this be working as designed?
As it stands now we can even probe for this by sending a purposely large packet, and monitor for the response or PTB reply.  If it doesn't come back, that network is broken.

Is it really worth masking the problem?

> 
> 
> This is not a receiver-side problem but an outcome of path MTU-adjusting
> tunnels and aggressive ICMP packet filters
> 
> As Luckie and Stasiewicz comment "A third strategy, which is arguably the
> most challenging, is to educate system administrators about the necessity of
> forwarding PTB messages"
> 
> A simpler approach is for the sender to take the marginal hit of peak
> throughput rate and use an initial MTU that is lower that 1500. 1280 is an
> obvious choice, and 1400 is probably safe as well.
> 
>   regards,
> 
>    Geoff
> 
>

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 4 Dec 2014 21:17:50 +0000

On Dec 4, 2014, at 2:18 AM, [email-masked] wrote:

> "People" as in residential users may not really care about MTU as long as their Facebook and Youtube works.

More to the point, it’s going to be about delivery rate using a protocol such as TCP, and the total time to deliver a quantum of information like a page or the initial load of a video file. The difference between a TCP transfer of a median-sized file (about 20 data packets) using IPv4 and IPv6 is 20 bytes per packet, which means that the last segment might be 19*20=380 bytes longer, or there might be one additional segment. If RFC 6928 (initial window = 10 segments) is in force, that’s not even an additional RTT. The difference for a long file is an additional segment every 76 segments.

I should think a change in MTU has a similar effect. We send a few more segments. The question is not about the size of a segment, its the rate of segment flow in a protocol that keeps a number of segments simultaneously in flight. There is a hill of beans; it’s not very tall.

The way you say that implies “for people that aren’t very knowledgeable”. Honestly, the places that PMTU materially affects have to do with the interrupt rate to computers moving large files across direct multi-gigabit interconnects. Using RFC 5690 to reduce the interrupts from every-other-segment to every-twelfth-segment, or equivalently using 9K MTUs, reportedly materially changes the behavior of such a file transfer, or so the researchers tell me. At 1G or below, it makes a lot less of a difference.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 05 Dec 2014 08:22:38 +1100

In message <[email-masked]>, Geoff Huston write
s:
> Measurement exercises for Path MTU in IPv6? I'm aware of work by Matthew Luck
> ie and Ben Stasiewicz some years ago (http://www.caida.org/~mjl/pubs/measurin
> g-pmtud.pdf).
> 
> In IPv6 the problem is at its worst in a situation where the large packet sen
> der is using a 1500 octet MTU and close to the sender is a packet filter that
>  discards IPv6 packet too big massages. If there is some form of encapsulatin
> g tunnel on the path to the receiver and the tunnel operates with an effectiv
> e MTU of less than 1500 then its possible to encounter a wedged TCP session:
> 
> As Luckie and Stasiewicz comment "A third strategy, which is arguably the mos
> t challenging, is to educate system administrators about the necessity of for
> warding PTB messages"
> 
> A simpler approach is for the sender to take the marginal hit of peak through
> put rate and use an initial MTU that is lower that 1500. 1280 is an obvious c
> hoice, and 1400 is probably safe as well.
> 
>   regards,
> 
>    Geoff

Which is why named sets IPV6_USE_MIN_MTU to 1 on all IPv6 sockets.
Yes, this results in fragmented TCP packets on some platforms despite
it being set before connect/accept due to the TCP layer not paying
attention to it.  It should cause the MSS negotiation to be adjusted
down (yes the MTU for the socket is 1280 not 1500 so do the MSS
calculation based on 1280).

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Keith Moore <[email-masked]>
Date: Thu, 04 Dec 2014 17:54:31 -0500

> I find it completely unacceptable. ISPs have no business filtering legitimate packets.
+1.

Keith

---

From: Geoff Huston <[email-masked]>
Date: Fri, 5 Dec 2014 12:51:22 +1100

> On 5 Dec 2014, at 8:00 am, Metzler, Dan J <[email-masked]> wrote:
> 
>> 

The problem is here that there is potentially no single "network" that is broken. At one point on the path from the sender to the receiver there is a constrained MTU. At one point on the path from the constrained MTU to the sender (which is not necessarily the same path as the forward path) there is an ICMP filter.

So who is "broken"? The tunnel ingress that is performing IP-over-IP and needs to down-adjust the MTU to factor in the additional header? Or the filter rule set that is saying that it is unprepared to admit ICMP packets from arbitrary sources? Its not necessarily a single point of "failure" here, but the interaction of two different items of middleware, and they may not be located within a single locus of admin/operational control.

Geoff

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 09:03:34 +0100

> So who is "broken"?

It doesn't say so explicitly in RFC 2460, but you can't filter ICMP too bigs _and_ expect to send packets larger than 1280 bytes.

So any network that wants to filter ICMP too bigs (which I see no legitimate reason for) needs ot make sure that it only accepts 1280-byte packets, so PMTUD or MSS advertisement of 1220 happens before the packets enter the non-PMTUD-capable stretch of network.

Anyone who suggests otherwise is not part of the solution but part of the problem.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 09:03:34 +0100

---

From: Tore Anderson <[email-masked]>
Date: Fri, 5 Dec 2014 09:18:12 +0100

* Geoff Huston <[email-masked]>

> ICMP PTB message toward the sender

I would add to this that a number of very common router implementations
will rate-limit their local ICMP error generation (or at least not give
it very high priority), and this isn't always user-configurable. That
means that once the tunnel ingress router are receiving sufficient
amounts of packets sized 1481 or larger, PMTUD starts breaking down.
Clients will then start retransmitting their large packets, which
obviously just exacerbates the problem.

This is a fairly common problem with IPv6 tunnels, in my experience.
TCP MSS clamping seems to be the most pragmatic way to deal with it (at
least if you don't have control of all the CPEs and can set the LAN RA
MTU), as reduces the amount of too-large packets to a managable level
that's more unlikely to trip any rate-limiter.

Tore

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 09:29:45 +0100

> amounts of packets sized 1481 or larger, PMTUD starts breaking down.

I've never seen this. And in measurements using a traceroute implementation that sends 10 packets at line rate I usually get ICMPs for all all 10 packets, so it seems rate limiting isn't as common as we are lead to believe.

Rewriting the MSS seems more resource intensive than returning too bigs, so I don't see how that is a pragmatic choice. Also, it's architectually problematic: boxes in the middle of the network have no business modifying layer 4. I hope we can stay away from this in IPv6.

Despite the discussion here there doesn't seem to be a widespread PMTUD black hole problem in IPv6 like there is in IPv4, let's keep it that way by keeping the message very clear that you MUST either generate/accept too bigs OR stick to an MTU of 1280.

Don't forget that unlike in IPv4, in IPv6 a router that connects to an MTU-limited hop can easily advertise a reduced MTU to local hosts in RAs, so the case where MSS clamping is used in IPv4 can be solved more cleanly in IPv6.

---

From: Tore Anderson <[email-masked]>
Date: Fri, 5 Dec 2014 09:43:41 +0100

* Iljitsch van Beijnum <[email-masked]>

> I've never seen this. And in measurements using a traceroute
> implementation that sends 10 packets at line rate I usually get ICMPs
> for all all 10 packets, so it seems rate limiting isn't as common as
> we are lead to believe.

10 packets is nothing. Try 1M or so. Anyway, perhaps you find RAS more
trustworthy than me, see slides 35-36:

https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf

This presentation discusses traceroutes so it focuses on TTL/HLIM
exceeded, but PTB is ICMP too.

Tore

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 09:53:01 +0100

On 05 Dec 2014, at 9:43, Tore Anderson <[email-masked]> wrote:

> 10 packets is nothing. Try 1M or so.

Are you saying that there are rate limiting implementations that don't kick in until the 9 millionth packet??

Observations of TTL exceeded rate limiting also don't tell us much about too big rate limiting as ALL routers are in the position to generate TTL too bigs and not doing so is only a minor inconvenience (stars in traceroute) while only routers that sit at the border between two links with different MTUs need to generate too bigs and failing to do so has serious connectivity implications.

But please let's keep our eyes on the ball here: with IPv4, PMTUD is dead because there is too much ICMP filtering. With IPv6, PMTUD on the whole works, so we shouldn't give up on it and deploy ugly hacks or hardcode 1400 and be stuck with that for the next few decades.

---

From: [email-masked]
Date: Fri, 05 Dec 2014 10:05:02 +0100 (CET)

> exceeded, but PTB is ICMP too.

I can confirm that such rate limiting exists, and some of us like to
run it with stricter policing (lower limits) than the vendor defaults.

Steinar Haug, AS 2116

---

From: [email-masked]
Date: Fri, 05 Dec 2014 10:12:51 +0100 (CET)

> Are you saying that there are rate limiting implementations that don't kick in until the 9 millionth packet??

The ratelimiting is often of the type "max X Mbps of Internet traffic
is allowed to hit the control plane". It is irrelevant whether this
traffic is explit ping of one of the router addresses, or for instance
TTL expired or packet too big.

Thus if you send 10 packets you may not hit the rate limiting. If you
(really: all senders aggregated) send a sufficient number of packets
in a sufficiently small time interval, you will hit the rate limiting
and packets will be dropped. There is nothing magic about this...

Steinar Haug, AS 2116

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 10:17:59 +0100

> and packets will be dropped. There is nothing magic about this...

Indeed. And whatever the magic event is that causes that to happen:
 That ICMP PTB is gone.

And that is just one of many reasons why they might go missing and why
PMTUD is not reliable; though it mostly does just work.

Note that with TCP the packet will be resent a bit later and thus there
is a chance in such a scenario that that ICMP PTB does come back for the
next packet. but there is then already a weird delay which happened.

And that is ignoring the networks that just filter or the hardware that
ignores ICMPv6 parsing at all. (eg those line-rate loadbalancers that
started this thread)...

Greets,
 Jeroen

---

From: Tore Anderson <[email-masked]>
Date: Fri, 5 Dec 2014 10:18:50 +0100

> Are you saying that there are rate limiting implementations that
> don't kick in until the 9 millionth packet??

Obviously not, but that a burst of 10 packets are extremely unlikely to
trip any ICMP rate-limiter on its own, unless the router is operating
close to the rate-limiter's upper limit to begin with.

As you'll see in the presentation, most of those rate-limiters cap the
ICMP generation rate at a few hundred pps. That's hardly enough for
real traffic levels (just a measly 1Gb of traffic is *at least* 83Kpps,
and a fair chunk of those packets are going to be >1480).

That's why I said «try 1M», because if you are planning on providing
internet service over IPv6 tunnels, you had better prepare for dealing
with *M*pps of oversized packets, or if that's too much for your tunnel
ingress routers, do something else like TCP MSS clamping and/or reduced
RA MTUs to get the too-big rate down to manageable levels.

> Observations of TTL exceeded rate limiting also don't tell us much
> about too big rate limiting as ALL routers are in the position to
> generate TTL too bigs and not doing so is only a minor inconvenience
> (stars in traceroute) while only routers that sit at the border
> between two links with different MTUs need to generate too bigs and
> failing to do so has serious connectivity implications.

The thing is, both HLIM exceed and PTB are both ICMPv6 errors the
router has to generate locally. So both types typically end up in the
same rate-limiter or being downprioritised in the same way.

You should really read the presentation that I linked to, and actually
don't limit yourself to pages 35-36, start at 31 and continue to 37.
Most of the information applies equally well to PTB generation as it
does to TTL/HLIM exceeded.

Tore

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 10:19:26 +0100

> to fix the tunnels that can't do 1500.

As Iljitsch stated " It's not a very busy server, so I got 927 incoming
TCP SYNs in two hours"

Might want to check a larger network... before making any big statement.

Let say looking at at least a million packets from several thousand
sources and not 927 packets possibly all from the same source.

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 10:22:44 +0100

On 2014-12-05 10:18, Tore Anderson wrote:
[..]
> RA MTUs to get the too-big rate down to manageable levels.

Indeed. The defaults on for instance Linux & BSD used to/still have ICMP
ratelimits configured too. And even turning those up to high numbers
will cause ICMP packets (be that echo/request/pt/etcb) to be dropped all
the time.

Disabling them is one of the ways to resolve it, the other is to not use
Linux/BSD kernel itself for doing the handing of the tunnels.

Greets,
 Jeroen

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 10:37:46 +0100

> and a fair chunk of those packets are going to be >1480).

No. If a router sits behind a link with an MTU < 1500 (and especially < 1480) then obviously it will see many TCP sessions that will attempt to send packets that are too large to pass and thus too bigs need to be generated. But due to TCP's congestion control mechanisms, such a router is never going to see those packets at line rate or anything close to it.

Sure, it's possible that if the factory defaults for ICMP load balancing are too tight or the operator lowers the limit that some fraction of those full size packets early in the TCP session are going to be dropped, but that doesn't mean there's a PMTUD black hole, only that there will be some number of TCP retransmits before a too big is returned and the session continues at the lower packet size.

Also, hopefully the operators in question will realize there is a problem and start using a bigger MTU and/or get rid of that restrictive rate limiting.

I don't see the possibility of this scenario as a reason for the IETF or the operator community to start implementing ugly hacks. I know it's always tempting to try to engineer something new that can handle a corner case that the existing stuff doesn't handle, but life is so much easier if we just consistently implement best practices rather than try to make our protocols completely foolproof. Those fools are just so damn ingenious.

> if you are planning on providing
> with *M*pps of oversized packets

I'd be interested to hear from tunnel operators on how many oversized packets they receive as a fraction of total packets / 1480 byte packets / line capacity.

---

From: Tore Anderson <[email-masked]>
Date: Fri, 5 Dec 2014 11:05:46 +0100

> No. If a router sits behind a link with an MTU < 1500 (and especially
> < 1480) then obviously it will see many TCP sessions that will
> attempt to send packets that are too large to pass and thus too bigs
> need to be generated. But due to TCP's congestion control mechanisms,
> such a router is never going to see those packets at line rate or
> anything close to it.
> 
> Sure, it's possible that if the factory defaults for ICMP load
> balancing are too tight or the operator lowers the limit that some
> fraction of those full size packets early in the TCP session are
> going to be dropped, but that doesn't mean there's a PMTUD black
> hole, only that there will be some number of TCP retransmits before a
> too big is returned and the session continues at the lower packet
> size.

Unless we're talking about your own single-user personal tunnel here,
you're not dealing with "the session" singular, you're probably dealing
with tens of thousands of new sessions being created every second.
And unless there is TCP MSS clamping in effect, a majority of those will
have an initial burst of oversized (1500 octets) packets right after the
three-way handshake, which require PTBs to be generated.

So yes, you *will* end up with a PMTUD black hole, as the continued
influx of new sessions that require PTBs will keep the rate-limiter
permanently clogged (and the existing sessions will sit there
retransmitting their too-big packets, clogging it up even further). Only
a few very sessions will get the required PTB; most of them will just
time out before a single byte of application data has been successfully
transmitted.

> Also, hopefully the operators in question will realize there is a
> problem and start using a bigger MTU and/or get rid of that
> restrictive rate limiting.

Easier said than done, considering that the ratelimiters often aren't
tunable or possible to disable. (Again, read RAS' presentation.)

Also, increasing the outer MTU to 1520+ is impossible in most cases,
because you have to be certain that *every* device in the network
(including any user-owned/managed HGWs) supports mini-jumbos.

Tore

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 11:19:00 +0100

>time out before a single byte of application data has been successfully

There are two things surprising here (well not really we know how it works),
one is that people design and deploy devices that basically just don't work in the
context where they are deployed. I.e. a tunnel endpoint will have to be able to
send loads of packet too big ICMPs. That's part of the required feature set.
Same thing for loadbalancers that can't deliver ICMPs to the right hosts.

The other confusing part is that commonly used operating systems still ship with
PMTU blackhole detection disabled.

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 11:38:04 +0100

>(including any user-owned/managed HGWs) supports mini-jumbos.

Another way forward is to have to 'internet facing' side of the tunnel forward
large packets fragmented. Assuming that the 'small' other endpoint can reassemble
them. Traffic toward 'the internet' can be considered local and can be fixed to
properly deal with PMTU detection.

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 11:56:01 +0100

On 2014-12-05 11:38, Philip Homburg wrote:
> large packets fragmented.

Only end hosts do fragmentation in IPv6.

Also, hosts that are causing these problems are not going to perform an
even heavier task.

Greets,
 Jeroen

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 12:00:50 +0100

>Only end hosts do fragmentation in IPv6.

I was talking about tunnels. In general, tunnel endpoints are hosts.

In case it wasn't clear, I meant fragmenting the outer (IPv4) packet,
not the inner (IPv6) packet.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 12:45:47 +0100

> I was talking about tunnels. In general, tunnel endpoints are hosts.

Not sure if that's true. But if it is, there's no problem anyway, as the host will advertise an MSS that reflects the path MTU. I.e., tunnel on host: IPv4 MTU = 1500, IPv6 MTU = 1480, IPv6 MSS = 1480 - 60 = 1420 and no too bigs are required.

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 13:14:40 +0100

> not the inner (IPv6) packet.

Not only tunnels have a lower MTU. Just adding an extension header
somewhere and you already lower the payload size.

Doing fragmentation means sending at least two packets, maybe one gets
lost, what to do, resend, wait? Note that most tunneling protocols do
not resend data (unless they run over TCP and then TCP handles it).

It also pushes the burden of refragmentation towards the receiver of the
packet, which suddenly becomes open to a fragmentation attack, which is
likely not what they will want to see happening (enough crap gets send
to those boxes already because someone is misbehaving on IRC).

Remember that the minimum MTU for IPv6 is 1280 for a reason: very safe
to use for tunneling and other constructs where one does not have a full
Ethernet MTU.

Just sticking everywhere to 1500 is not a solution. Unless one wants to
define the minimum MTU for IPv6 as 1500. It is still a minimum and there
are speakers out there that do 9000 with ease, just not intra-AS, and
intra-AS is where these kind of PTB and ICMP ratelimiting problems
exist: in somebody elses network.

As for implementations doing fragging of tunneled packets, check:
http://backreference.org/2013/07/23/gre-bridging-ipsec-and-nfqueue/

Which gives a nice overview why that further is a bad idea and why it
typically will not work anyway unless getting very hacky.

Oh, and do apply that to every deployed host on the Internet please.

Search also for frag/mtu/df/PMTUDISC in:
 https://github.com/torvalds/linux/blob/master/net/ipv6/sit.c

yes there is a magic 'pmtudisc' option for tunnels, but not enabled per
default and it does not work as expected.

For some fun reason, only custom TTLs get a frag_off filled and thus
possibility to not frag. Thus if one does not specify a TTL at creation
time (advised anyway as the default used to be 255 or actually 'TTL from
the tunnel device') the DF flag does not get set...

Also check how often the MTU is forced to the IPv6 minimum MTU. Fun also
that that code is so well commented on why certain pieces of code are
there let alone what they do.

These:
https://github.com/freebsd/freebsd/blob/master/sys/netinet/in_gif.c
https://github.com/freebsd/freebsd/blob/master/sys/netinet/ip_output.c
read sooo much better than the Linux equivalent... comments, what a
useful mechanism for documenting why things are done.

As gif does not set ip_off, DF is not set and thus ip_output will
fragment here.

Or summary on those two platforms:
 FreeBSD	fragments packets (DF not set)
 Linux 		fragments normally (DF not set), but most configs
		define TTL and thus DF gets set and packets get dropped
                need to optionally add hack to drop DF bit.

I am quite sure that at least in the SixXS userbase nobody uses this
feature, as sixxsd does not do reassembly for proto-41 packets...
(and the default MTU is 1280 as mentioned before).

Greets,
 Jeroen

---

From: Gert Doering <[email-masked]>
Date: Fri, 5 Dec 2014 13:19:47 +0100

Hi,

> Only end hosts do fragmentation in IPv6.

A tunnel ingress *is* an end-host as far as the outer packet is concerned.

Gert Doering
        -- NetMaster

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 13:20:41 +0100

In your letter dated Fri, 5 Dec 2014 12:45:47 +0100 you wrote:
>Not sure if that's true. But if it is, there's no problem anyway, as the 
>host will advertise an MSS that reflects the path MTU. I.e., tunnel on 
>host: IPv4 MTU =3D 1500, IPv6 MTU =3D 1480, IPv6 MSS =3D 1480 - 60  =
>1420 and no too bigs are required.

The host? My local tunnel endpoint is in the CPE. All IPv6 hosts are connected over
ethernet. So those hosts will advertise an MSS based on a 1500 MTU.

When a remote host sends a 1500 octet IPv6 packet then the tunnel endpoint I 
normally use will just send a fragmented IPv4 packet, so my tunnel link has an
MTU of 1500 avoiding an PTMU problems.

One other tunnel only forwards IPv6 packets where the resulting IPv4 packet is
1500 octets or less, so this has a potential for PMTU problems.

Yet another tunnel seems to limit IPv6 packets to 1280 for some non-obvious reason.

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 13:22:21 +0100

On 2014-12-05 12:45, Iljitsch van Beijnum wrote:
> Not sure if that's true. But if it is, there's no problem anyway, as the host will
>
> advertise an MSS that reflects the path MTU.

psst... MSS only applies to TCP.

We need a new term likely though for what you want to say:
 MPL = Max Payload Length

As that is what we are talking about, not the MTU.

> I.e., tunnel on host: IPv4 MTU = 1500, IPv6 MTU = 1480,
> IPv6 MSS = 1480 - 60 = 1420 and no too bigs are required.

Yep, you definitely mean MPL there.
(which might match MSS, but is not MSS)

The problem some folks in this discussion seem to want to avoid though
is that tunneled MTU deviates off 1500.

And they thus want to force tunnels to always have a MTU of 1500 and
thus force the lower layer to fragment packets instead.

While indeed then for that section you might not have any PMTU issues
anymore, you will create a whole bunch of overhead by fragmenting
packets (multiple packets, more possible packet loss, more traffic needs
to be sent etc). Next to the deployed base not support it anyway (see
other mail checking Linux code that is currently on github, not the
stuff that is even out on the Internet which can be ancient)...

Greets,
 Jeroen

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 13:31:19 +0100

On 05 Dec 2014, at 13:20, Philip Homburg <[email-masked]> wrote:

>>> In general, tunnel endpoints are hosts.

> My local tunnel endpoint is in the CPE.

Ok, so which is it?

> IPv6 hosts are connected over

Yes. So here you need PMTUD (unless the tunnel supports 1500-byte inner packets).

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 13:39:20 +0100

On 2014-12-05 13:20, Philip Homburg wrote:
> The host? My local tunnel endpoint is in the CPE.

A node terminating a tunnel is also a host, but on IPv4.
While it is a forwarding node in IPv6.

> All IPv6 hosts are connected over

Which means you need proper PMTUD or for TCP clamp the MSS.
Though MSS-clamping does not solve any non-TCP problems.
(or have google etc force a guessed low MSS)

> normally use

You mean the remote side of the tunnel (thus the other end of your CPE).

> will just send a fragmented IPv4 packet, so my tunnel link has an

And in sending 1 Gig of traffic you end up sending a lot more than that.
Let alone the extra latency you get (at least double) for full packets.
But if one can live/work with that, fine of course.

Latency might be 4x the norm if you are sending full packets in both
directions, fun(tm).

While it is an interesting 'solution' (fragging on a lower level and
thus having a 1500 MTU path). The overhead/latency and attacks that are
then possible on the infrastructure kind of removes the positive side:
that a few nodes on the Internet have broken ICMPv6 handling.

> Yet another tunnel seems to limit IPv6 packets to 1280 for some non-obvious reason.

Which "another tunnel"

Note that 1280 is the minimum IPv6 MTU which is likely why that is
chosen. Most tunnel protocols allow changing MTU though.

Greets,
 Jeroen

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 13:44:18 +0100

In your letter dated Fri, 5 Dec 2014 13:31:19 +0100 you wrote:
>On 05 Dec 2014, at 13:20, Philip Homburg <[email-masked]> =
>wrote:
>

For the outer (IPv4) packet, the CPE is a host. For the inner (IPv6) packet, the 
CPE is a router. 

>Yes. So here you need PMTUD (unless the tunnel supports 1500-byte inner 

So, make the tunnel support 1500 octet inner packets until PMTU is actually fixed.

Of course, as a content provider you can just restrict the link MTU on your hosts
and have happy users. Essentially to avoid problems, you want to receive packets
as big as possible and send packets that are smaller than commonly used paths.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 14:01:42 +0100

> For the outer (IPv4) packet, the CPE is a host. For the inner (IPv6) packet, the 

Please don't (ab)use terminology like that. A host is a host, a CPE is not a host in normal conversation.

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 14:08:35 +0100

> Please don't (ab)use terminology like that. A host is a host, a CPE is not a host in normal conversation.

While the terms used in his previous message where indeed a bit
confusing, I see nothing wrong with the above.

Every[1] IP capable node is a host[2].

It is just that some nodes are also forwarding packets and thus become
routers.

CPEs definitely are hosts for most people, be that normal users (they go
to http://myrouter/ and for sysadmins: they admin those hosts.

And in the above sentence that CPE is definitely a host acting as one
side of a tunnel endpoint.

Greets,
 Jeroen

[1] = do unmanageable forward-only nodes exist? the moment a packet is
sent to the IP address configured though it is acting as a host. Thus I
don't think this is possible unless they do not comply with IP and are
effectively L2 bridges.

[2] Even https://en.wikipedia.org/wiki/Host_%28network%29 states:
    "A network host is a computer or other device connected to
     a computer network."

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 14:32:24 +0100

>somewhere and you already lower the payload size.

In the current internet, PMTU is quite broken. Not completely broken, but enough
to be noticeable. Extension headers are also routinely filtered. 

So if you don't want to much pain, you avoid PMTU and extension headers. 

>not resend data (unless they run over TCP and then TCP handles it).

Locally, i.e. between my CPE and my tunnel endpoints the loss rate of IPv4 packets
in general is way lower than losses caused by PMTU discovery.

So this is good trade off. By the time PMTU discovery is essentially perfect
it may be time to reconsider.

>to those boxes already because someone is misbehaving on IRC).

If somebody wants to DoS my CPE, they can just do that. No need to try a 
fragmentation attack. 

>to use for tunneling and other constructs where one does not have a full

And then somebody starts a VPN that goes over a tunnel somewhere and we are back at
square one.

Furthermore, unless you can somehow get all hosts on the internet to use a 1280 MTU,
your tunnel will cause PMTU problems.

Way more than a decade after ethernet devices first started supporting jumbograms,
most ethernets are still at 1500. So the most sane way to avoid PMTU problems is
to run all links at a 1500 octet MTU.

>exist: in somebody elses network.

The only practical MTU on the internet is currently 1500. Locally you can go
higher or lower.

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 05 Dec 2014 14:40:24 +0100

Please doc this 'frag on the layer below' idea with the pros and cons it
has, would be a good informational document; but likely that will see
little global/large deployment due to the cons.

> So if you don't want to much pain, you avoid PMTU and extension headers. 

I've been using PMTU successfuly for over 15 years.

Only recently there where two events where a middle box was broken as
they do not comply to the IPv6 specification (they ignore ICMPv6).

All other instances of PMTU issues are easily classified as either:
 - misconfigurations
 - rate limitting

Yes. These are problematic. Not much one can do about as these involve
people and physical limits.

> it may be time to reconsider.

PMTU will never be 'perfect' as well, humans...

> If somebody wants to DoS my CPE, they can just do that. No need to try a 

The CPE is not the typical target, but the endpoint where your tunnel
terminates likely is.

And of course we are not talking about a single user setup here, or even
10 people using the same setup. Scale up to a few thousand.

> And then somebody starts a VPN that goes over a tunnel somewhere and we are back at

Then they should not do that or let that VPN support the fragmentation.

> your tunnel will cause PMTU problems.

Forcing a singular MTU is not a proper solution. Your proposal forces
everybody to 1500.

And re-read your sentence above "then somebody starts a VPN that goes
over a tunnel"... ;)

> to run all links at a 1500 octet MTU.

Exactly a major reason for not having a fixed MTU: you will fix the
whole network to that MTU and it will never be able to change.

Thus while your proposal is a reasonable solution it does not solve
anything, it just avoids it (and with big penalties at that).

> The only practical MTU on the internet is currently 1500. Locally you can go

By fixing the MTU like you propose we will be stuck with an 1500 MTU
forever. Lets not go there.

Greets,
 Jeroen

---

From: "Metzler, Dan J" <[email-masked]>
Date: Fri, 5 Dec 2014 14:42:30 +0000

Hi Geoff,

> From: Geoff Huston [mailto:[email-masked]]
> Sent: Thursday, December 4, 2014 7:51 PM
> To: Metzler, Dan J
> Cc: Jeroen Massar; [email-masked]
> >> In IPv6 the problem is at its worst in a situation where the large
> >> packet sender is using a 1500 octet MTU and close to the sender is a
> >> packet filter that discards IPv6 packet too big massages. If there is
> >> some form of encapsulating tunnel on the path to the receiver and the
> >> tunnel operates with an effective MTU of less than 1500 then its possible to
> encounter a wedged TCP session:
> >>
> > As it stands now we can even probe for this by sending a purposely large
> packet, and monitor for the response or PTB reply.  If it doesn't come back,
> that network is broken.
> >
> The problem is here that there is potentially no single "network" that is
> broken. 

I respectfully disagree. :-)

> At one point on the path from the sender to the receiver there is a
> constrained MTU. 

Yes there generally is an MTU constraint of some type.  It's broken if below 1280.  I know of no requirement that MTU be below 1500, 4000, or even 9000.  It must be above 1280 which will allow the ICMPv6 error messages to pass.

> At one point on the path from the constrained MTU to the
> sender (which is not necessarily the same path as the forward path) there is an
> ICMP filter.

That's the one that is broken.

> 
> So who is "broken"? The tunnel ingress that is performing IP-over-IP and needs
> to down-adjust the MTU to factor in the additional header? Or the filter rule
> set that is saying that it is unprepared to admit ICMP packets from arbitrary
> sources? Its not necessarily a single point of "failure" here, but the interaction
> of two different items of middleware, and they may not be located within a
> single locus of admin/operational control.

The network that is filtering Packet Too Big is broken.  The owner of the network should be pointed at RFC 4890 (Filtering Recommendations).  There are a number of error messages that Must not be dropped.  (Packet Too Big is one of them.)

Perhaps this would be better stated in RFC 2463, or RFC 2460.
In any case, don't we at some point just have to tell people, "hey your network is broken".
Some ICMPv6 are required for proper functioning of IPv6.

There will always be people who do things wrong.  Some people mistakenly block port 80 to a web server, but we don't go recommending that the standard http port move to something else.  Even though IPv6 has been around for a long time, many are not yet even thinking about it, or are in the early stages.  Consequently, people do dumb things; like block all ICMPv6 traffic because what they really want to do is stop IPv4 style ping probes (echo).  (The problem is not MTU size, but inadequate knowledge.)

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Dec 2014 14:44:49 +0000

Hi Philip,

> From: v6ops [mailto:[email-masked]] On Behalf Of Philip Homburg
> Sent: Friday, December 05, 2014 4:44 AM
> To: Iljitsch van Beijnum
> Cc: [email-masked] WG
> Subject: Re: [v6ops] PMTUD forever, was: MTUs on the general Internet
> 

Correct, but remember that 1500 is the minimum size all tunnels should support.
Tunnels should not place any hard-coded limits on the maximum size, i.e., they
should not clamp the MTU.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Dec 2014 14:49:24 +0000

Hi Philip,

> Sent: Friday, December 05, 2014 5:32 AM
> To: Jeroen Massar
> Cc: [email-masked] WG

Right, but I would say a 1500 octet MTU *or larger*. Then in a BCP we say:

  "Hosts that send packets larger than 1500 bytes SHOULD use RFC4821."

At the same time, let's do away with tunnel MTU clamping so we can get
out of this jam we're in.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Dec 2014 14:59:09 +0000

> Sent: Friday, December 05, 2014 5:40 AM
> To: Philip Homburg
> Cc: [email-masked] WG

You are misunderstaing; we are talking about a little bit of fragmentation
that should be turned off as soon as possible. That will happen as more
and more links set MTUs larger than 1500.

> I've been using PMTU successfuly for over 15 years.

I'm surprised no one has mentioned this yet, but any node on the Internet
can send a forged PTB message; it doesn't even need to spoof the source
address. Fernando Gont raises this concern in his "atomic fragments" doc.

> Forcing a singular MTU is not a proper solution. Your proposal forces

No, the idea is to make sure *all* 1500s get through while at the same time
allowing larger packets through as long as they can make it without any
fragmentation. No more tunnel MTU clamping.

> And re-read your sentence above "then somebody starts a VPN that goes

Tunnels within tunnels, yes. That is why the solution needs to work recursively.

> whole network to that MTU and it will never be able to change.

Correct - no fixed upper bound. But, a 1500 lower bound.

> forever. Lets not go there.

No more tunnel MTU clamping, as above. 1500 is only the *minimum*.

Thanks - Fred

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 16:32:14 +0100

I think RFC 4821 tries to be too general. Implementing some kind of PMTU
blackhole detection is relatively easy for TCP. But the way forward is to have
clear RFC that details how exactly it should be done for TCP. Not a relatively
vague description of all possibilities. One important choice that needs to be made
is whether to deprecate packet too big ICMPs or not. 

Going beyond TCP, I don't think these techniques would work at all for a root DNS
server (or any other big DNS service).

But maybe it is sufficient to say that any protocol that cannot reliably do PMTU
blackhole detection should assume a link MTU of 1280.

---

From: "Metzler, Dan J" <[email-masked]>
Date: Fri, 5 Dec 2014 15:43:20 +0000

> From: v6ops [mailto:[email-masked]] On Behalf Of Templin, Fred L
> Sent: Friday, December 5, 2014 8:59 AM
> To: Jeroen Massar; Philip Homburg
> Cc: [email-masked] WG
> > Please doc this 'frag on the layer below' idea with the pros and cons
> > it has, would be a good informational document; but likely that will
> > see little global/large deployment due to the cons.
> 
> You are misunderstaing; we are talking about a little bit of fragmentation that
> should be turned off as soon as possible. That will happen as more and more
> links set MTUs larger than 1500.
> 
> > >>> In case it wasn't clear, I meant fragmenting the outer (IPv4)
> > >>> packet, not the inner (IPv6) packet.
> > >>
> > > In the current internet, PMTU is quite broken. Not completely
> > > broken, but enough to be noticeable. Extension headers are also routinely
> filtered.
> > >
> I'm surprised no one has mentioned this yet, but any node on the Internet can
> send a forged PTB message; it doesn't even need to spoof the source address.
> Fernando Gont raises this concern in his "atomic fragments" doc.

So the point being that somebody might be able to spoof you down to 1280, if the timing is right, and they target the right address?

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 5 Dec 2014 15:51:31 +0000

On Dec 5, 2014, at 1:37 AM, Iljitsch van Beijnum <[email-masked]> wrote:

> I don't see the possibility of this scenario as a reason for the IETF or the operator community to start implementing ugly hacks. 

For the record, I live behind an IPsec tunnel, and it solves the problem another way. Since it can see traffic in the clear (it is adding the IPsec ESP), it can see the TCP MSS option, and set it to a smaller value. That’s usually 1400, not 1280, but it could be 1280.

I might be tempted to refer to that as an "ugly hack". It does predictably work.

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Dec 2014 18:01:56 +0000

Hi Philip,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Philip Homburg
> Sent: Friday, December 05, 2014 7:32 AM
> To: Templin, Fred L
> Cc: [email-masked] WG

Indeed. IMHO, PTBs are nice to have when you can get them and *if* they can
somehow be authenticated as originating at an on-path router. Otherwise, the
best course of action may be to ignore them.

> server (or any other big DNS service).

OK.

> blackhole detection should assume a link MTU of 1280.

That is a slightly different statement than the one I have been contemplating,
which is "hosts that send packets larger than 1500 bytes MUST use RFC4821
(or something else very much like it)". That said, most hosts expect 1500 to
work but the only real guarantee of course is 1280.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Dec 2014 18:05:39 +0000

> So the point being that somebody might be able to spoof you down to 1280, if the timing is right, and they target the right address?

Yes, but Fernando took it one step further and said that a bogus PTB that reports
a size smaller than 1280 might cause the recipient to start including a Fragment
Header in future IPv6 packets, where they might be clobbered if the network
is configured to drop fragments.

Thanks - Fred

---

From: Philip Homburg <[email-masked]>
Date: Fri, 05 Dec 2014 19:54:52 +0100

In your letter dated Fri, 5 Dec 2014 18:01:56 +0000 you wrote:
>That is a slightly different statement than the one I have been contemplati=
>ng,
>(or something else very much like it)". That said, most hosts expect 1500 t=
>o

There are many links with an MTU lower than 1500. So on the current Internet, 
any protocol that cannot do PMTU blackhole detection should default to 1280.

(This of course results in a catch 22 for most tunnels)

Implementations that do support PMTU blackhole detection can start at the interface
MTU, though for TCP you would also limit yourself to the MSS.

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Dec 2014 19:21:46 +0000

Hi Philip,

> Sent: Friday, December 05, 2014 10:55 AM
> To: Templin, Fred L
> Cc: [email-masked] WG
> There are many links with an MTU lower than 1500.

Yes, I am aware of that.

> So on the current Internet,

Regardless of what they should do, many try for 1500.

> (This of course results in a catch 22 for most tunnels)

Why catch 22? Tunnels should simply do what they need to do to get 1500s
through and should not clamp the MTU so that even larger packets can be
admitted.

Thanks - Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 5 Dec 2014 21:10:32 +0100

> Note that in both cases there was nothing below an MTU of 1280 and nothing above an MTU of 1500.

I've kept this running for the past 30 hours, and now I have some 9k (MSS = 8960) MTUs.

But I have no idea what on earth is going on inside Amazon's datacenters. Maybe they love prime numbers?

17:11:24.578439 IP 54.174.148.205.41221 > 83.149.65.1.80: S 2882684603:2882684603(0) win 26883 <mss 8961,sackOK,timestamp 401910 0,nop,wscale 7>
18:15:40.824912 IP 54.173.217.66.50098 > 83.149.65.1.80: S 3904598406:3904598406(0) win 26883 <mss 8961,sackOK,timestamp 4294946346 0,nop,wscale 7>
18:34:22.194809 IP 54.172.213.42.49088 > 83.149.65.1.80: S 1482929797:1482929797(0) win 26883 <mss 8961,sackOK,timestamp 652387 0,nop,wscale 7>
18:40:09.912631 IP 54.86.224.246.46138 > 83.149.65.1.80: S 2156593579:2156593579(0) win 26883 <mss 8961,sackOK,timestamp 740533 0,nop,wscale 7>
20:05:47.713870 IP 54.174.140.22.36348 > 83.149.65.1.80: S 3029198432:3029198432(0) win 26883 <mss 8961,sackOK,timestamp 1564561 0,nop,wscale 7>

Still not a single IPv4 ICMP too big.

---

From: Jeroen Massar <[email-masked]>
Date: Sat, 06 Dec 2014 11:34:01 +0100

> I might be tempted to refer to that as an "ugly hack". It does predictably work.

And then you send something that is not TCP....

Greets,
 Jeroen

---

From: Jeroen Massar <[email-masked]>
Date: Sat, 06 Dec 2014 12:03:54 +0100

> You are misunderstaing; we are talking about a little bit of fragmentation

A little bit indeed, but because of that you are sending 2 packets
instead of one, and that happens in both directions.

> and more links set MTUs larger than 1500.

IPv6 should also have been deployed globally as soon as possible...

And ICMPv6 MUST not be dropped, but that is happening too.

Greets,
 Jeroen

---

From: [email-masked]
Date: Sat, 06 Dec 2014 12:20:39 +0100 (CET)

> IPv6 should also have been deployed globally as soon as possible...

Speaking only for myself: We run with a suitable jumbo MTU on most
links *within our AS*. I see no sign of such an increased MTU becoming
more available on peering and transit links.

Steinar Haug, AS 2116

---

From: Jeroen Massar <[email-masked]>
Date: Sat, 06 Dec 2014 13:26:25 +0100

> more available on peering and transit links.

Agreed.

IMHO one big reason of which is that there are cases where PMTU does not
work 100% and thus there is no real benefit of enabling larger MTUs as
it will only give problems (as it is something not well known) and thus
higher support cost, than creating faster transfers.

Hence, a status quo.

Greets,
 Jeroen

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 6 Dec 2014 13:50:26 +0100

On 06 Dec 2014, at 13:26, Jeroen Massar <[email-masked]> wrote:

> there are cases where PMTU does not

You are assuming that if PMTUD fails to detect the correct path MTU on paths with a < 1500 MTU that PMTUD will also fail to detect the correct path MTU on paths with a >= 1500 MTU. These are very different cases, so that is an incorrect conclusion. In the one case, you are shorter than everyone else, in the other case you're taller than everyone else. That's not the same experience.

It's too bad that my home gateway doesn't support an IP MTU above 1500 (although its LAN ports will forward packets as large as 1982 bytes) so I can't try this myself at home...

But as I've said several times now: IPv4 is a lost cause, but with IPv6 PMTUD works well enough that we don't have to write it off. We just need to educate people that not generating / excessively rate limiting / filtering too bigs is incompatible with running IPv6 properly.

---

From: "Templin, Fred L" <[email-masked]>
Date: Sat, 6 Dec 2014 16:17:46 +0000

Hi Jeroen,

> Sent: Saturday, December 06, 2014 4:26 AM
> To: [email-masked]
> Cc: [email-masked]

Yes, the status has been quo for a long time, but we now have an opportunity
to move beyond that. If tunnels stop clamping the MTU and end hosts start
using RFC4821 larger MTUs will be possible. The good news is that fixing the
tunnels and fixing the end hosts can be done independently of one another.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Dec 2014 16:23:53 +0000

More on what I said the other day, there are two magic numbers that tunnels
need to concern themselves with: 1280 and 1500. Accommodating all packets
within that size range while placing no explicit upper bound for accommodating
larger packets is what is being proposed. In other words, no MTU clamping.

Thanks - Fred

> Sent: Saturday, December 06, 2014 8:18 AM
> To: Jeroen Massar; [email-masked]
> Cc: [email-masked]

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Dec 2014 16:10:44 +0000

> larger packets is what is being proposed. In other words, no MTU clamping.

Have we reached conclusion on this, and can it now be considered "case closed"?

Remember - "take care of the smalls, and let the bigs take care of themselves":

https://datatracker.ietf.org/doc/draft-templin-aerolink/
https://datatracker.ietf.org/doc/draft-templin-aeromin/

Thanks - Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 18 Dec 2014 15:57:04 +0100

I left the tcpdumps running for almost a week and did a more thorough analysis of the results, see:

Maximum packet sizes on the internet 
http://www.bgpexpert.com/article.php?article=151

Internet packet sizes part 2: IPv4 path MTU discovery is dead
http://www.bgpexpert.com/article.php?article=152

Quick summary: all IPv6 hosts and 99.7% of IPv4 hosts advertise they can handle at least 1280 bytes in their TCP MSS and about 65% can handle 1500.

In almost a week, I didn't get a single IPv4 too big, so it seems that in practice, IPv4 PMTUD is no longer used. (Please confirm with a larger dataset before taking any actions based on this!) However, depending on PMTUD doesn't lead to obvious problems with IPv6.

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 18 Dec 2014 16:28:44 +0100

> http://www.bgpexpert.com/article.php?article=151

"my server received 41753 incoming TCP SYN"
..
"Turns out, most of the IPv6 traffic on my server is from bots that check "

Sample size too small to represent the Internet which also does not
consists out of bots....

...
"1280 and 1480 are probably IPv6-in-IPv4 tunnels and 1428 AYIYA tunnels."

If you have the addresses I can tell you exactly what type of tunnels
they are if coming from SixXS space[1], though likely it does not matter
too much what type it is.

As you are looking at MSS, most clients are configured that the tunnel
has a MTU of 1280 while the local network announces a MTU of 1500 and
thus a MSS of 1500-overhead.

Hence, you will see an MSS for 1500 and then get a PTB from the PoP the
tunnel terminates on for the first reply packet.

MSS clamping on tunnel PoPs is not being done as that would hide
problems with any other protocol than TCP and thus would make debugging
a real hell "TCP works, but hey, UDP does not WTF!" etc...

Greets,
 Jeroen

[1] https://www.sixxs.net/pops/prefixes/

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 18 Dec 2014 16:04:49 +0000

Hi Iljitsch,

Thanks for this analysis - it correlates with what I have been saying all along.
There are two magic numbers in the Internet today: 1280 and 1500. For all
sizes in between, it is likely attributed to tunnels that clamp the MTU to
some degenerate size.

Take care of the smalls, and let the bigs take care of themselves:

Thanks - Fred

> Sent: Thursday, December 18, 2014 6:57 AM
> To: [email-masked] WG
> Subject: Re: [v6ops] Quick measurement: MTUs on the general Internet
> 
> In almost a week, I didn't get a single IPv4 too big, so it seems that in practice, IPv4 PMTUD is no longer used. (Please confirm with a
> larger dataset before taking any actions based on this!) However, depending on PMTUD doesn't lead to obvious problems with IPv6.

---

From: Mark ZZZ Smith <[email-masked]>
Date: Fri, 19 Dec 2014 07:34:11 +0000 (UTC)

----- Original Message -----
> From: Iljitsch van Beijnum <[email-masked]>
> To: "[email-masked] WG" <[email-masked]>
> Cc: 
> Sent: Friday, 19 December 2014, 1:57
> Quick summary: all IPv6 hosts and 99.7% of IPv4 hosts advertise they can handle 
> at least 1280 bytes in their TCP MSS and about 65% can handle 1500.
> 
> In almost a week, I didn't get a single IPv4 too big, so it seems that in 
> practice, IPv4 PMTUD is no longer used. (Please confirm with a larger dataset 
> before taking any actions based on this!)

I wonder if RFC4821 'probing' based PMTUD is being used instead. A number of years ago I know Windows both supported and enabled it under certain circumstances, so perhaps in more recent versions of Windows it has been enabled by default or in more situatins.

> However, depending on PMTUD 
> doesn't lead to obvious problems with IPv6.
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 19 Dec 2014 11:42:00 +0100

> I wonder if RFC4821 'probing' based PMTUD is being used instead. A number of years ago I know Windows both supported and enabled it under certain circumstances

I believe RFC 4821 is used for blackhole detection in Windows and possibly other OSes.

My measurements were on a FreeBSD machine, though.

However, from the point of view of the network there is no difference between RFC 1191 style or RFC 4821 style PMTUD: in both cases packets as large as the MSS are sent with DF set, so if the path MTU is smaller than the MSS value, there should be too bigs. The only difference is that if no too bigs make it back to the sending host, RFC 1191 is dead in the water while RFC 4821 recovers.