Subject: Re: FYI: DNSOPS presentation

From: Brian E Carpenter <[email-masked]>
Date: Sat, 03 Apr 2010 04:16:28 +1300

As every successful IPv6 user on Windows XP knows, the hack
would break all Windows XP IPv6 usage (since it can't resolve
DNS over IPv6 at all).

I prefer the current situation, where occasionally I have to
disable IPv6 manually due to being on an IPv6-broken network
where DNS serves up AAAA records and my regular tunnel is too
slow. This seems OK, since people using v6 on XP are normally doing
so as conscious early adopters.

I do think that Google's solution to this, i.e. being selective
about who gets the AAAA records in the first place, is much less
of hack with less harmful side effects.

Surely a better hack would be for recursive resolvers in IPv6-broken
networks not to serve up AAAA records at all? Tunnel users could
always find another resolver.

Regards
   Brian

---

From: Rémi Després <[email-masked]>
Date: Fri, 2 Apr 2010 18:09:53 +0200

Le 2 avr. 2010 à 17:16, Brian E Carpenter a écrit :

> where DNS serves up AAAA records

So do I.

> of hack with less harmful side effects.

+1

> networks not to serve up AAAA records at all?

Agreed.

RD

---

From: [email-masked]
Date: Fri, 2 Apr 2010 16:55:52 +0000

> > networks not to serve up AAAA records at all?

	aren't you making the assumption that the recursive resolver is
	on/in the same broadcast domain as the stub which it is serving?
	how can you ensure that the IP reachability of the stub is "semantically
	simialar"** to the IP reachability of the recursive resolver?

	the DNS has no hooks into topological reachability. 

** identical, equivalent, etc... See IDN equivalence

--bill

---

From: james woodyatt <[email-masked]>
Date: Fri, 2 Apr 2010 11:11:03 -0700

On Apr 2, 2010, at 08:16, Brian E Carpenter wrote:
> of a hack, with less harmful side effects.

While everyone else was in the 6MAN session, I was actually jumping back and forth between 6MAN and DNSOP because I didn't want to miss this presentation.  As I pointed out at the microphone in DNSOP, the proposed hack is absolutely broken because it relies on a pair of TOTALLY INOPERATIVE ASSUMPTIONS:

  A) good IPv6 connectivity between the DNS resolving server and the DNS content server implies good IPv6 connectivity between the client application host and the application server host.

  B) good IPv6 connectivity between the client application host and the application server host requires good IPv6 connectivity between the DNS resolving server and the DNS content server.

The obvious illustration that comes to mind is a host behind an Apple AirPort Extreme base station at a public IPv4 address and configured in 6to4 tunnel mode, but in a domain where the 6to4 relay service is unreliable, badly-impaired or black-holed, and also configured to forward its DNS queries to resolving servers that are dual-stack IPv4/IPv6 capable.  This configuration is sufficiently common that any proposed solution to the problem must have an answer for it, and the proposed solution doesn't deal with it at all.

Shorter james: the proposed hack cannot solve the problem it intends to solve, and it breaks other networks that work fine today.  It should not be endorsed.

--
james woodyatt <[email-masked]>
member of technical staff, communications engineering

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 03 Apr 2010 07:45:44 +1300

Bill,

> 	the DNS has no hooks into topological reachability. 

Correct, but a hack in this area by definition can't cover all cases.

Er, much better to mend the IPv6 brokenness, of course.

   Brian

---

From: Joel Jaeggli <[email-masked]>
Date: Fri, 02 Apr 2010 12:05:00 -0700

On 04/02/2010 11:11 AM, james woodyatt wrote:
>> about who gets the AAAA records in the first place, is much less of
>> a hack, with less harmful side effects.
> 
> While everyone else was in the 6MAN session, I was actually jumping
> back and forth between 6MAN and DNSOP because I didn't want to miss
> this presentation.  As I pointed out at the microphone in DNSOP, the
> proposed hack is absolutely broken because it relies on a pair of
> TOTALLY INOPERATIVE ASSUMPTIONS:
> 
> A) good IPv6 connectivity between the DNS resolving server and the
> DNS content server implies good IPv6 connectivity between the client
> application host and the application server host.
>
> B) good IPv6 connectivity between the client application host and the
> application server host requires good IPv6 connectivity between the
> DNS resolving server and the DNS content server.

actually they don't rest on those assumptions.

the assumption being made is that queriers who are customers of a given
set of name-servers have available to them a particular kind of
connectivity...

this is exactly the same kind of assumption made when selectively
returning different A records in an ipv4 global loading balancing setup.

The consequences of getting it wrong in ipv4 are you go to a more
distant instance of a service.

The consequence of getting it wrong in ipv6 are you hand a AAAA record
to a host that doesn't use it (this causes no harm) or you hand a AAAA
record to host that shouldn't use it (because soemthing about their
connectivity is broken). presumably the decision to make a particular
set of queriers as blessed for the purpose of reciveing AAAA records is
based on a judgement about the likelyhood of the hosts behind that
resolver being non-broken.

> The obvious illustration that comes to mind is a host behind an Apple
> AirPort Extreme base station at a public IPv4 address and configured
> in 6to4 tunnel mode, but in a domain where the 6to4 relay service is
> unreliable, badly-impaired or black-holed, and also configured to
> forward its DNS queries to resolving servers that are dual-stack
> IPv4/IPv6 capable.  This configuration is sufficiently common that
> any proposed solution to the problem must have an answer for it, and
> the proposed solution doesn't deal with it at all.
> 
> Shorter james: the proposed hack cannot solve the problem it intends
> to solve, and it breaks other networks that work fine today.  It
> should not be endorsed.
> 
> 
> -- james woodyatt <[email-masked]> member of technical staff,
> communications engineering
> 
> 
>

---

From: Jason Livingood <[email-masked]>
Date: Fri, 02 Apr 2010 15:17:09 -0400

I agree with what James just posted.

Since this has hopped over to v6ops from dnsops, I thought I'd post here
what I wrote over there, as I think it's important as we're working on
solutions w/o a clear agreement on the problem:

It seems that have the cart before the horse, so to speak.  IMHO, we need to
do the following (and there's no reason they cannot occur rapidly):

1 - Develop a clear problem statement that outlines (1) how "broken" users
are defined and (2) what effects this "brokenness" has on these end users
(or other parts of the Internet).

2 - Describe all the various methods and tactics by which end user
"brokenness" can be detected.  This may include website-based detection,
DNS-query-based detection, or a variety of other methods.

3 - Then, after we have agreement on Problem Definition and Problem
Detection, we can measure the problem to understand what the scope or scale
of the problem is. 

4 - After we understand Problem Definition, Problem Detection, and Problem
Scope, then you can arrive at possible solutions.  Seems like in this case
we sort of *started* here, which concerns me and I think we need to be
careful that discussion thus far does not constrain development of a full
list of Solution Options.  I further believe we will need to encourage the
pursuit of multiple solutions simultaneously.  Lastly, just because end user
software upgrades may be difficult doesn't mean we shouldn't do them and
shouldn't focus just as much energy on those than other options.

Jason

---

From: james woodyatt <[email-masked]>
Date: Fri, 2 Apr 2010 12:26:18 -0700

> actually they don't rest on those assumptions.

Actually, I spoke at length with Igor Gashinsky on Tuesday night at IETF 77 before the presentation in DNSOP, and I'm pretty sure I accurately characterized the assumptions behind his proposal.

Where he and I disagree, I think, is on the question of whether the assumptions are *operative*.  He believes they are; I believe they are not.  It's *his* train set, though, so I suppose he can break it however he wants.

---

From: [email-masked]
Date: Fri, 2 Apr 2010 20:35:08 +0000

> Correct, but a hack in this area by definition can't cover all cases.

	does it even cover a statistically significant fraction of cases?
> Er, much better to mend the IPv6 brokenness, of course.

	then why not spend the energy/resouce on ensuring v6 transport
	works instead of wasting this on what is clearly a DNS haq for 
	a temporary condition?  Or is this a case of "when all we have is
	the DNS, everything looks like a query/response"?

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 03 Apr 2010 10:30:10 +1300

On 2010-04-03 09:57, Igor Gashinsky wrote:
...
> :: Surely a better hack would be for recursive resolvers in IPv6-broken
> :: networks not to serve up AAAA records at all? Tunnel users could
> :: always find another resolver.
> 
> It is the same hack. 

Not if applied administratively, and reversed administratively
when the v6 brokenness is fixed. From a network ops point of view,
this is much cleaner than automated detection, and will also
put the responsibility in the right place: the network manager
who has not deployed v6.

And early adopters who want to bypass the hack can simply
configure a DNS server that does deliver AAAA.

    Brian

    Brian

You would use that very same feature that was
> developed to accomplish this (filter-aaaa) -- what's more, when your 
> network stops being v6-broken, you get the extra benefit that the users 
> who can query your dns resolvers over ipv6 can now get AAAA's as first 
> step of rollout...
> 
> -igor
>

---

From: Olivier MJ Crepin-Leblond <[email-masked]>
Date: Sat, 03 Apr 2010 11:01:54 +0200

On 02/04/2010 21:17, Jason Livingood wrote :
> Scope, then you can arrive at possible solutions.

Completely in agreement, although I'd like to add "...and whether such
solutions are required."

I feel uneasy messing around with the DNS by introducing "ugly hacks" to
resolve a problem with "0.078% of people", thus affecting 99.922% of
people who had no problem and who might then end up having a problem due
to the hack.
Agreed, 470K people is no small number, but we have to look at the
bigger picture with 600M people. If we start designing Internet techno
to look at the lowest common denominator, we're in for a very bad ride.

Warmest regards,

Olivier

-- 
Olivier MJ Crépin-Leblond, PhD
http://www.gih.com/ocl.html

---

From: Rémi Després <[email-masked]>
Date: Sat, 3 Apr 2010 15:45:35 +0200

> Shorter james: the proposed hack cannot solve the problem it intends to solve, and it breaks other networks that work fine today.  It should not be endorsed.

+1
An approach that would break what works today in IPv6 MUST be rejected.

It remains that a clear description of which problem is encountered, and by which organization, might help to find a acceptable solution.

RD

---

From: Tore Anderson <[email-masked]>
Date: Sat, 03 Apr 2010 17:11:44 +0200

Hi,

* Olivier MJ Crepin-Leblond

> Completely in agreement, although I'd like to add "...and whether 
> such solutions are required."

Currently, the number of end users who can connect to a single-stack
IPv4 web site is greater than the number of end users who can connect
to a dual-stack web site.  This situation must change, or you won't be
seeing much IPv6 content appearing on the internet anytime soon.

> I feel uneasy messing around with the DNS by introducing "ugly hacks"
> to resolve a problem with "0.078% of people", thus affecting 99.922%
> of people who had no problem and who might then end up having a
> problem due to the hack.

The obvious "solution" to this is currently to _not_ deploy IPv6 on
existing content.  For now, that will keep ~100% of users happy, and
doesn't require any DNS hacks/whitelists to be maintained.

But I still would like to deploy IPv6 somehow.  Suggestions on how
exactly I can go about doing that, without messing around with DNS or
breaking connectivity for users who up until now have had no problems,
would be greatly appreciated.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

---

From: Tore Anderson <[email-masked]>
Date: Sat, 03 Apr 2010 20:06:27 +0200

Hi Mark,

* Mark Andrews

> You turn IPv6 on a service at a time.  Turn it on for delay tolerent 
> services.  e.g. SMTP.  Turn it of for services which try different 
> servers quickly.  e.g. DNS.  Turn it on for the client side.

Sure thing, I've already done some of those, but it won't really change
much.  There's no real reason for a end-user to contact my SMTP or DNS
servers directly, as he will normally relay e-mail/DNS queries via his
own ISPs servers anyway.  And the fact that all the workstations at my
office have native IPv6 connectivity doesn't have any effect on all the
broken end users in other networks.

The problematic service is without question the web sites.

> There may be few place to connect to now but the ratio of good to
> bad will get better as more people turn up IPv6 as a production
> service on the client side.

Absolutely.  I'm sure that at some point in the future there'll be more
end users able to connect to a dualstacked web site than there is end
users able to connect to a IPv4-only site.  At that point you'll see
lots of IPv6 content showing up.  No problem.

However, when will that point in time be?  The way it's going right now
it seems to me that it will be well after IPv4 pool is exhausted, when
the IPv4 internet is starting really to crumble under CGNs and other
life-extending band-aids.  I'm more impatient than that though, and
don't want to just sit around and wait until then.  It appears other
content providers like Yahoo and Google don't either.  But short of
resorting to DNS tricks, or attempting to persuade all the end users'
ISPs and/or software vendors to fix their problems, I really don't see
what else can be done.

By the way I actively pursue the option of asking problematic networks
and software vendors to fix their problems.  It's a very slow process,
though.  Even when software patches are released, it takes lots of time
before a sufficient amount of end-users have applied them.

Best regards,

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 04 Apr 2010 11:37:09 +1200

On 2010-04-04 06:06, Tore Anderson wrote:
...
> before a sufficient amount of end-users have applied them.

We used to do a lot of that about 20 years ago when IPv4 connectivity
and applications support was patchy. The Internet only works today because
the message eventually got through (or the providers and vendors who
didn't get the message vanished).

There's no short cut.

  Brian

---

From: Tore Anderson <[email-masked]>
Date: Mon, 05 Apr 2010 11:45:50 +0200

Hi Mark,

* Mark Andrews

> The first thing is to publish a complete description of the testing 
> methodology and envionment.  So far I havn't see it.

I've posted descriptions a couple of times before:

http://lists.cluenet.de/pipermail/ipv6-ops/2009-November/002658.html
http://www.ietf.org/mail-archive/web/dnsop/current/msg08406.html

Have a look, and don't hesitate to let me know if you're still wondering
about something.

> Most of the transition technology involves encapsulation so was the
> mtu set to prevent pmtud being triggered due to encapsulation in
> IPv4?

I haven't set the MTU explicitly, but the objects that are being served
are (intentionally) very small, so I never transmit any packets that are
anywhere near 1280 octets.  However I see that I occasionally receive
incoming packets that exceed 1280 (HTTP requests with large amounts of
header information).  I will try to reduce the advertised TCP MSS to
prevent PMTUD in the inbound direction from ever becoming an issue.
These jumbo requests occur very seldom though, so I'll be very surprised
if the lowered MSS will make a noticeable impact on the measured client
loss.

Thank you for your feedback!

BR,

---

From: Gert Doering <[email-masked]>
Date: Mon, 5 Apr 2010 12:29:48 +0200

Hi,

> Shorter james: the proposed hack cannot solve the problem it intends to solve, and it breaks other networks that work fine today.  It should not be endorsed.

Seconded.

(I can't remember how many times I have written in the last 10 years that
the transport protocol used for DNS queries coming from recursor servers
has nothing whatsoever to do with the transport protocol capabilities of
the client sending the DNS query - even without any caching involved)

Gert Doering
        -- NetMaster
-- 
Total number of prefixes smaller than registry allocations:  150584

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: Olivier MJ Crepin-Leblond <[email-masked]>
Date: Mon, 05 Apr 2010 18:20:35 +0200

Le 03/04/2010 17:44, Mark Andrews a écrit :
> servers quickly.  e.g. DNS.  Turn it on for the client side.  There
> may be few place to connect to now but the ratio of good to bad
> will get better as more people turn up IPv6 as a production service
> on the client side.
>
>   

This works exactly in line with what I have been suggesting my corporate
clients should do when they are looking at implementing IPv6.
A copy of my presentation slides can be found on:
http://www.slideshare.net/ocl999/suggestion-for-an-ipv6-roll-out

Turning the delay tolerant services on, migrating such services to dual
stack, enables personnel to gain real IPv6 traffic & admin experience
with minimal potential disruption - at low cost.
I realise that I am digressing from the matter in the subject, so I'll
stop here.

Kindest regards,

Olivier

---

From: Jason Livingood <[email-masked]>
Date: Mon, 19 Apr 2010 12:31:18 -0400

This thread seems to have died out (on dnsops as well).  But I will note
that it appears to be on the agenda at the upcoming ISOC IPv6 workshop (from
16:30 - 18:00 - see
https://www.isoc.org/isoc/conferences/registration/index.php?id=7ce20e4c88b7
e328).  It is also addressed in a short whitepaper some colleagues and I put
together in advance of that meeting (available at
http://www.comcast6.net/IPv6_DNS_Whitelisting_Concerns_20100416.pdf)

Jason

On 4/2/10 3:17 PM, "Jason Livingood" <[email-masked]>
wrote:

> Since this has hopped over to v6ops from dnsops, I thought I'd post here what
> I wrote over there, as I think it's important as we're working on solutions
> w/o a clear agreement on the problem:
>

---

From: Doug Barton <[email-masked]>
Date: Mon, 19 Apr 2010 21:34:16 -0700

On 4/19/2010 9:31 AM, Jason Livingood wrote:
> This thread seems to have died out (on dnsops as well).

My understanding is that the Yahoo! folks are gathering data and will
report back when they have results.

Doug

-- 

	... and that's just a little bit of history repeating.
			-- Propellerheads

	Improve the effectiveness of your Internet presence with
	a domain name makeover!    http://SupersetSolutions.com/

---

From: Tore Anderson <[email-masked]>
Date: Tue, 20 Apr 2010 09:49:03 +0200

* Jason Livingood

> [...] a short whitepaper some colleagues and I put together in 
> advance of that meeting (available at 

Hello, Jason.  I have a few comments.  On the 1st of April, you posted
the following text to the dnsop mailing list, quoting another message
sent to the same list by me:

> - Total client loss to the dualstacked host is at 0.074%.
> - (At least) 95% of the client loss is due to clients choosing to use
>   inherently unreliable transitional IPv6 (6to4/Teredo) instead of IPv4.
> - I've identifed three groups of clients that behave in this way:
>   1) Users of Opera <10.50, which did not use the RFC 3484-compliant
>      getaddrinfo() call, at least on Windows.  It would prefer any form
>      of IPv6 connectivity above IPv4.  This is especially a problem on
>      Windows Vista and 7, where 6to4 and/or Teredo are enabled by
>      default.  This accounts for about half of the 95%.
>   2) Dualstacked Mac OS X users with RFC 1918 IPv4 addresses (using
>      NAT) and transitional IPv6 addresses.  In this case, getaddrinfo()
>      will sort the IPv6 address above the IPv4 address, causing the
>      transitional connectivity to be used.  This accounts for the other
>      half of the 95%.
>   3) Dualstacked Linux users with RFC 1918 IPv4 addresses and
>      transitional IPv6 addresses, as GNU libc's getaddrinfo()
>      implementation behaves exactly like the one in Mac OS X.  However,
>      the overall client loss caused by this is miniscule compared to
>      #1 and #2 (I have problems measuring any at all).
> - When disregarding all hits from users in problem groups #1 and #2,
>   the total client loss is at 0.003% - this is, in my opinion, low
>   enough to accept.  (Of course, other content providers might feel
>   differently.)

I find it quite puzzling that you, just a few weeks later, claim that no
data about this problem has been shared with the community.  That was,
after all, exactly what I intended to do with the above message as well
as all the reports I've sent to the ipv6-ops mailing list in the last
few months.  Why did you choose to disregard the data from my
measurements completely?  Do you feel that they are flawed or
incomplete?  If so, I would appreciate it very much if you could tell me
what exactly is the problem.  I promise I will do my best to address
your concerns, and also publicly share any updated reports.

Others are doing measurements, too, and are publishing results:

http://www.ripe.net/ripe/meetings/ripe-57/presentations/Colitti-Global_IPv6_statistics_-_Measuring_the_current_state_of_IPv6_for_ordinary_users_.7gzD.pdf
http://labs.ripe.net/content/ipv6-measurement-compilation

I believe that I'm the one that's been trying the most to look behind
the numbers and identify (and publish) the actual underlying causes of
client loss, though.

Also, in the document, you write:

> [...] 0.078% of users are "broken." That is a trivial percentage of users to
> most ISPs. For example, in the Comcast network, this represents fewer
> than 12,000 users. If the problem could be solved with a new home
> gateway device, to use one possible example, Comcast could rapidly
> ship replacement devices to affected users. We believe, given the
> small number claimed, that most ISPs could rapidly solve this problem
> (if it can be precisely defined) on their own, without content
> providers having to resort to DNS whitelisting.

I doubt any content provider feels that breaking access to 12,000 users
from Comcast alone is «trivial».  For a large provider, 0.078% of all
users could very well mean millions of users world-wide.  In the end, it
all boils down to the content providers asking themselves the following
question:

«Do we want to service 999 or 1,000 users today?»
Or:  «Do we want to make €999 or €1,000 today?»

I fail to see why any commercial content provider would choose the first
option - which currently represents dual-stack.  The DNS whitelisting
approach is just a workaround in order to allow for some IPv6/dual-stack
operation to take place without at the same time reducing the number of
potential customers.

At some point in the future, when the number of users able to access a
dual-stack site equals or exceeds the number of users able to access a
IPv4-only site, there's no longer any need for DNS whitelists and I'm
sure they will disappear overnight (potentially replaced with blacklists
though), and since having only IPv4 will be a competitive disadvantage,
you'll see IPv6 deployment accelerating rapidly on the content side.

You in Comcast can help making that point in time be reached sooner
rather than later, by providing production-quality native IPv6 service
to all users.  You obviously are working on it, and I applaud that.
However your efforts alone are not enough - we need other ISPs to do the
same, too.

Almost all of the client loss I see is caused by end-user software
preferring unreliable transitional IPv6 connectivity, 6to4 in
particular.  I've not been able to identify faulty CPE devices as a
cause of client loss, so this is out of your control.  (It could very
well be that faulty CPE devices are a problem with certain ISPs in
countries that simply are not accessing my Norwegian-language content,
though.)

In any case, you can help alleviate this problem somewhat by placing
6to4 relays (192.88.99.1/24) in your network as close to the end users
as possible, and then content providers can do the reverse (relay to
2002::/16) in their data centres.  This will ensure that 6to4 will be
working almost as reliably as regular IPv4.

But as before, Comcast can not solve this problem alone, and broken 6to4
connectivity is especially a problem in enterprise/managed network
environments that are filtering proto-41 outright.  I've had discussions
with some of these networks, some do it knowingly and have no intention
of changing it, some have no idea of what I'm talking about, and finally
some removed their filters.  Identifying and contacting all these
«6to4-hostile» networks, and persuading them to change their practise,
is simply not a scalable way of handling the problem, though.
Especially if you're a global content provider.

The best way I can think of to fix the problems I'm observing, is to
have software vendors release updates, and for those updates to make
their way onto most end users' systems.  I've been pursuing that goal
for a while, so far this has resulted in the release of a RFC
3484-compliant version of the Opera web browser, which has caused a very
significant drop in overall client loss.  Also several operating system
vendors have applied my patches to GNU libc implementing section 2.7 in
draft-arifumi-6man-rfc3484-revise-02 to their developement branches.  So
far these are Fedora, Ubuntu, Gentoo, and Mandriva. Debian, openSUSE,
and last but not least Apple, have not yet responded.

The Linux distributions will not have a noticeable impact on the overall
client loss numbers, but if Apple were to apply the proposed change to
Mac OS X and push updates to their customers, it would have a enormous
impact.  Glancing briefly at my numbers this far in April, some 60-70%
of client loss is attributable to Mac OS X, or 85-95% if I first
disregard older known broken versions of the Opera web browser. The GNU
libc maintainers declined to apply the change because RFC 3484 is still
the current standard, perhaps Apple feels the same way.  If so it would
be fantastic if an updated RFC 3484 was published really soon, as it
takes time to get software updates onto most end users' computers.  And
time is something we do not have too much of at this point.

Best regards,

---

From: Philip Homburg <[email-masked]>
Date: Tue, 20 Apr 2010 11:03:44 +0200

>Or:  «Do we want to make €999 or €1,000 today?»

I don't if it would fly, but why not white-list ISPs that have their
web-servers, MXen, pop and imap servers all listed in DNS with IPv6 addresses?
(I wonder how many ISPs would qualify at the moment)

Then you can be reasonably sure that any customers with broken IPv6
connectivity will complain to the ISP as well.

>Especially if you're a global content provider.

The thing that IMHO went wrong here is that the first thing everybody does
is disable neighbor discovery, and therefore also neighbor unreachability
detection.

Eventhough it is perfectly possible to find out that a connection doesn't work,
in practice nobody does anything with that option.

To some extent that is also due to lack of standards. For example, if source
address selection had an option to skip source addresses if the route to
the destination is down, then a host with a broken IPv6 configuration would
fall back to just IPv4.

There is a lot that can be done with accurate reports on unreachability, but
hardly any RFC mandates that. And ISP are all to happy to disable this 
becuase of the load it generates on their routers/networks.

---

From: Tore Anderson <[email-masked]>
Date: Tue, 20 Apr 2010 11:44:20 +0200

Hi,

* Philip Homburg

>> For a large provider, 0.078% of all users could very well mean
>> millions of users world-wide.

After sending my previous message I did the math on this statement, and
I realise it's an exaggeration - so I'd like to moderate «millions» to
«hundreds of thousands».   (Not that it changes any of my points.)

> web-servers, MXen, pop and imap servers all listed in DNS with IPv6
> addresses? (I wonder how many ISPs would qualify at the moment)

Not many, I think.  Such a requirement would probably cause Google's
current whitelist to shrink considerably - Comcast, for instance, would
no longer be eligible.  Personally I will not deploy a whitelist at all
due to the overhead with maintaining it, instead I'll stay IPv4-only
until my customers feel the client loss is low enough to deploy
dual-stack (with global visibility).

> The thing that IMHO went wrong here is that the first thing everybody
> does is disable neighbor discovery, and therefore also neighbor
> unreachability detection.

Are you talking about ICMPv6 ND here?  If so, that is not end-to-end, it
only works on a link, similar to IPv4 ARP.  It is end-to-end
connectivity that is broken, so I fail to understand what you feel could
be done better here?

> Eventhough it is perfectly possible to find out that a connection
> doesn't work, in practice nobody does anything with that option.

I think Windows XP did some form of reachability testing and would
disable the 6to4 interface entirely if it failed.  They removed it in
Vista, but I've heard they're now considering putting back in.  But
Windows doesn't cause much problems anyway, since it appears their RFC
3484 implementation already implements section 2.7 from
draft-arifumi-6man-rfc3484-revise-02.  (You can argue that the Windows
implementation is the reference implementation, as it was Microsoft who
authored RFC 3484 to begin with.)

> To some extent that is also due to lack of standards. For example, if
> source address selection had an option to skip source addresses if
> the route to the destination is down, then a host with a broken IPv6
> configuration would fall back to just IPv4.

Actually, this is part of RFC 3484 already:

>    Rule 1:  Avoid unusable destinations.
>    If DB is known to be unreachable or if Source(DB) is undefined, then
>    prefer DA.  Similarly, if DA is known to be unreachable or if
>    Source(DA) is undefined, then prefer DB.

Applications will also fall back to IPv4 if the initial IPv6 connection
failed.  However for latency-sensitive applications that make use of
many short-lived connections (HTTP is a good example), this timeout is
so long that preferring defective IPv6 above working IPv4 will
essentially render the application useless.

By the way:  A curious effect of the current RFC 3484 algorithm is that
if you have both default IPv4 and IPv6 routes, RFC 1918-based IPv4
addresses, and only link-local IPv6 addresses, IPv6 will be preferred.
I'm not sure if this situation is commonly found in the wild though.

Best regards,

---

From: Havard Eidnes <[email-masked]>
Date: Tue, 20 Apr 2010 12:13:37 +0200 (CEST)

> I don't if it would fly, but why not white-list ISPs that have
> their web-servers, MXen, pop and imap servers all listed in DNS
> with IPv6 addresses?  (I wonder how many ISPs would qualify at
> the moment)
>

The failure mode experienced by the users is typically ... not
obvious, so whether there pops out a report to the provider from the
customer about broken IPv6 connectivity as a result of the failure
is far from given.  I suspect that the site administrators would not
expect IPv6 to be the problem source, since that's typically not
something they've enabled on their local network (yet).

Regards,

- Håvard

---

From: Philip Homburg <[email-masked]>
Date: Tue, 20 Apr 2010 12:52:18 +0200

>connectivity that is broken, so I fail to understand what you feel could

In discussion about 6to4, and other automatic tunneling protocols, the
big problem seems to be the first hop. I think most of the problems would
be solved if a 6to4 router would verify the performance of the tunnel.

>authored RFC 3484 to begin with.)

Does that mean that this policy will prefer a NAT44 IPv4 over a configured
tunnel?

>essentially render the application useless.

That's the problem. At the moment the host usually doesn't know.

A 6to4 router (like an Airport) could stop advertising the prefix when the
6ro4 default router is unreachable, but nobody does that.

In theory, web browser could configure faster timeout in TCP connection 
setups when destination unreachable ICMPs are received, but usually the
ICMPs are not there (and the API is probably not there either).

---

From: Philip Homburg <[email-masked]>
Date: Tue, 20 Apr 2010 13:06:44 +0200

>something they've enabled on their local network (yet).

If customers are not using any application level service offered by the
ISP (i.e. read mail through gmail/hotmail/yahoo, etc, don't visit the ISP's
website, etc.) then it may go undetected. That's true.

Otherise, admins learn quickly enough. It may be tricky to find at first.
but soon everybody starts talking about that new IPv6 thing and how much 
trouble it causes.

ISP helpdesks will also be quick to ask customers to disable IPv6 to resolve
any problems.

---

From: Tore Anderson <[email-masked]>
Date: Tue, 20 Apr 2010 13:23:11 +0200

* Philip Homburg

> In discussion about 6to4, and other automatic tunneling protocols,
> the big problem seems to be the first hop. I think most of the
> problems would be solved if a 6to4 router would verify the
> performance of the tunnel.

I see, just like I think Microsoft is considering doing.  Yes, I think
that would help quite a bit.  However, due to the asymmetric nature of
anycasted 6to4, it could very well be that even if the connectivity test
to destination A succeeds, connectivity will still be broken for
destination B - because A and B does not necessarily use the same return
relay.

In my opinion, anycasted 6to4 is fine as a last-resort option (along
with e.g. Teredo) for a IPv4-only host to access IPv6-only content.
But it is not as reliable as regular IPv4, and should therefore never be
preferred above it.

>> disable the 6to4 interface entirely if it failed.  They removed it
>> in Vista, but I've heard they're now considering putting back in.
>> But Windows doesn't cause much problems anyway, since it appears
>> their RFC 3484 implementation already implements section 2.7 from 
>> draft-arifumi-6man-rfc3484-revise-02.  (You can argue that the
>> Windows implementation is the reference implementation, as it was
>> Microsoft who authored RFC 3484 to begin with.)
> 
> Does that mean that this policy will prefer a NAT44 IPv4 over a
> configured tunnel?

Yes, that the effect of the change proposed in
draft-arifumi-6man-rfc3484-revise-02, and also how Microsoft's
getaddrinfo() currently behaves.  I strongly support the proposed change.

RFC 3484, however, currently says to prefer 6to4 over IPv4 NAT44 (or
more precisely RFC 1918 IPv4).

> A 6to4 router (like an Airport) could stop advertising the prefix
> when the 6ro4 default router is unreachable, but nobody does that.

Nope.  However, if 6to4 was only used by the end-user operating
system/applications as a last resort connectivity method, this would not
be necessary.  And that is clearly the _intention_ of RFC 3484, which
says to «avoid the use of transitional addresses when native addresses
are available».  I suppose NAT44 wasn't used as widely as now back when
the RFC was written.

> In theory, web browser could configure faster timeout in TCP
> connection setups when destination unreachable ICMPs are received,
> but usually the ICMPs are not there (and the API is probably not
> there either).

Yep, a ICMP dest unreachable (or TCP RST) would cause a rapid fallback.
However, no 6to4 implementation I've seen will transform an outer ICMPv4
dest unreachable to an inner ICMPv6 unreachable that actually will reach
the web browser application.

Best regards,

---

From: Mohacsi Janos <[email-masked]>
Date: Tue, 20 Apr 2010 13:59:02 +0200 (CEST)

> more precisely RFC 1918 IPv4).

However all the existing implemention prefers IPv4 with RFC1918 addresses.

> says to ?avoid the use of transitional addresses when native addresses
> are available?.  I suppose NAT44 wasn't used as widely as now back when
> the RFC was written.

Yes it was the case.

---

From: Tore Anderson <[email-masked]>
Date: Tue, 20 Apr 2010 14:07:03 +0200

* Mohacsi Janos

>> RFC 3484, however, currently says to prefer 6to4 over IPv4 NAT44
>> (or more precisely RFC 1918 IPv4).
> 

That is not the case.  Mac OS X and GNU libc+derivates are two prominent
examples of getaddrinfo()/RFC 3484 implementations that follow the RFC
to the letter.  They prefer 6to4 over (RFC 1918) IPv4.

That's what causing most of the client loss in my dual-stack measurements.

Best regards,

---

From: Mohacsi Janos <[email-masked]>
Date: Tue, 20 Apr 2010 14:24:40 +0200 (CEST)

> That's what causing most of the client loss in my dual-stack measurements.

Mac OS X does not implement RFC 3484.

Linux implementation is not tested by me a while ago, but one of my 
college did and confirmed no problem. Maybe he made a mistake in the test. 
It seems to me
http://people.redhat.com/drepper/linux-rfc3484.html
does not refer to this problem.

I try to inform linux RFC3484 implementers about the problem.

Best Regards,

 	Janos Mohacsi

---

From: Tore Anderson <[email-masked]>
Date: Tue, 20 Apr 2010 14:47:03 +0200

Hi,

* Mohacsi Janos

> Mac OS X does not implement RFC 3484.

I understand.  In any case, Mac OS X does prefer IPv6/6to4 over IPv4/RFC
1918, so in the end the result is the same.

> college did and confirmed no problem. Maybe he made a mistake in the
> test. It seems to me

I'm already on it:

http://sourceware.org/bugzilla/show_bug.cgi?id=11438
https://bugzilla.redhat.com/show_bug.cgi?id=577626
https://bugs.launchpad.net/glibc/+bug/555210
http://bugs.gentoo.org/show_bug.cgi?id=315977
https://bugzilla.novell.com/show_bug.cgi?id=597616
https://qa.mandriva.com/show_bug.cgi?id=58834
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=468801

Best regards,

---

From: Jason Livingood <[email-masked]>
Date: Tue, 20 Apr 2010 11:15:31 -0400

> sent to the same list by me:
<SNIP> 
> data about this problem has been shared with the community.

I didn't write that email - which may explain your puzzlement.  ;-)

Jason Livingood

---

From: Tore Anderson <[email-masked]>
Date: Tue, 20 Apr 2010 17:55:09 +0200

Hi Jason,

* Jason Livingood

> I didn't write that email - which may explain your puzzlement.  ;-)

«From: Jason Livingood <jason_livingood at cable.comcast.com>»
-- http://www.ietf.org/mail-archive/web/dnsop/current/msg08383.html

If you didn't send that message, who did?

Best regards,

---

From: Jason Livingood <[email-masked]>
Date: Tue, 20 Apr 2010 13:15:15 -0400

> If you didn't send that message, who did?

Um - you did.  I copied and pasted in a message you sent to the list, hoping
to hear input from others.

Jason

---

From: Tore Anderson <[email-masked]>
Date: Wed, 21 Apr 2010 08:31:13 +0200

* Jason Livingood

> I copied and pasted in a message you sent to the list, hoping to hear
> input from others.

Hi again, Jason.  Perhaps I was being unclear in my initial response to
you yesterday.  Allow me try again:

- In [1], you post a message quoting data elaborating on the causes of
  IPv6 client loss as observed from a content provider's point of view.
- In [2], a few weeks later, you claim that no public data on IPv6
  client loss exists.

It appears to me that you needed to look no further than in your own
sent-mail folder to find examples of such data, so it is your claim of
this data being non-existent that puzzles me.

[1] http://www.ietf.org/mail-archive/web/dnsop/current/msg08383.html
[2] http://www.comcast6.net/IPv6_DNS_Whitelisting_Concerns_20100416.pdf

Best regards,

---

From: Jason Livingood <[email-masked]>
Date: Wed, 21 Apr 2010 13:24:34 -0400

This is getting a little pointless to debate.  While I am happy that you
have provided data, across the entirety of the public Internet I think we
could use a little more.  The point I was making was that other than that
one post with data solely from you, I haven't seen a wave of data coming
out.  I do not mean to discount the value of your data or that you have
shared, but I still feel we need data from more than just you if the matter
is to move ahead.  I'm sure others, including us, are doing data gathering.
My only point was that none of that has come to fruition as of yet.

Jason