Subject: [rrg] Updating hosts

From: RJ Atkinson <[email-masked]>
Date: Mon, 12 Nov 2012 18:25:23 -0500

On Sunday, 11 Nov 2012, Christian Huitema wrote, in part:
> I think that we are giving up too early on the feasibility 
> of updating all hosts. [...] I also think that a research group 
> should not necessarily shy away from investigations that 
> affect all hosts.

+1

Later on Sunday, 11 Nov 2012, Joel M Halpern wrote, in part:
> I can well believe we can, for a reasonable value of all, 
> update all host TCP and UDP engines with a small, migratable, 
> change.
> 
> We can't change all host applications.
> 
> And we can not make a change that does not interoperate
> with existing hosts.

+1

Indeed, ILNP does update hosts, but does so with a 
clearly documented "backwards compatibility" mechanism
and also a clearly documented "incremental deployment"
scheme.

At least one ILNP prototype already has shown an *un-modified* 
(i.e. no source code changes, no recompilation) FreeBSD ping6 
binary using ILNPv6 between a pair of ILNPv6-enabled hosts.
(NB:  I'm aware of 2 independent implementations of ILNP 
that are underway, one for Linux and one for FreeBSD.)

This is a good initial step towards showing that ILNP
will enable existing "host applications" to be used with
ILNP -- without requiring application changes.

On Monday, 12 November 2012, Tony Li wrote, in part:
> The cost of software changes to the end user are now pretty much 
> lost in the churn of maintenance releases and new devices.
> 
> Thus, the real hurdle that has to be accomplished is to woo 
> the OS providers.  Convince them to implement and distribute 
> and that solves the deployment problem.  And to convince the 
> OS providers, you have to show them the killer app.  Without 
> a value add, it's not in their interest.

+1

The market makes its own determinations about what might
be sufficient "value add" to be worthwhile to distribute.
My crystal ball ('palantir') is cloudy, but here are 
some candidates, as a thought exercise:

1) Handset Mobility

  The ability to migrate TCP (or UDP or ...) sessions of 
  a mobile handset (or Surface or iPad or similar) between
  the (usually lower cost per bit) WiFi interface and the
  (usually higher cost per bit) GSM/LTE/CDMA2000 interface,
  as desired, without losing the application-layer session,
  and without requiring deployment of special Mobile-IP
  routers, might be such a killer app.  [See MILCOM 2008
  paper and the "mobile networks" paper from MILCOM 2009
  for how this works]

2) VM migration across routed IP subnetworks

  The ability to move an entire VM across routed IP subnets
  might be another such application.  Certainly some large
  multi-site data centre operators (e.g. folks who measure
  computing in CPU-Acres or CPU-Hectares) very much would
  like that capability.  [See our MILCOM 2012 paper for how]

3) Site-Driven Multi-homing (w/o adding prefixes to DFZ)

   The ability of sites to multi-home, without having to 
   coordinate with all of their upstream providers, and
   without having to add site-specific routing prefixes
   into the world-wide DFZ RIB appeals to many.
   [See our MILCOM 2009 paper on site-Controlled multi-homing
   for how]

Yours,

Ran

PS:  All of these MILCOM papers are available from U. St Andrews,
     along with nearly all of the other ILNP papers.  :-)
     <http://ilnp.cs.st-andrews.ac.uk>

---

From: Robin Whittle <[email-masked]>
Date: Tue, 13 Nov 2012 12:04:36 +1100

Hi Ran,

> ILNP -- without requiring application changes.

In I tried to figure out how ILNP could work with existing applications,
as it was claimed to do.  The thread is:

  ILNP: stack behavior to work with IPv6 apps? 2010-07-13
  http://www.ietf.org/mail-archive/web/rrg/current/thrd2.html#07108

Neither I nor anyone else was able to show how it could be done.
Sections 10.4 to 10.6 of RFC6741 seem to be relevant:

  http://tools.ietf.org/html/rfc6741#section-10.5

    Those applications that directly use IP Address values in
    application state or configuration will need to be modified
    for operation over ILNP.

My guess is that this would involve most applications.

This thread:

  Fwd: I-D Action:draft-irtf-rrg-recommendation-09.txt - ILNP problems
  http://www.ietf.org/mail-archive/web/rrg/current/thrd2.html#07185
  2010-08-01

contains further critiques of ILNP including the need for an extra
lookup (compared to the existing host protocols, DNS, IP addressing
etc.) when the responding host needs to ensure its return packet will
either go to the purported sending host (as represented by the initial
packet, which might be from another host) or be dropped by the routing
system if no such host exists.

There was no rebuttal of these critiques, but one RRG participant agreed
they were important.  Also, I can't see how ILNP can adapt securely or
rapidly enough to a host getting a new IP address to support
3G/4G/WiFi-style mobility of IP address, especially for voice applications.

As far as I know, ILNP is only practical for IPv6.  Despite a decade and
a half of faith on the part of many IETF people and repeated "real soon
now" statements about IPv6's imminent widespread adoption, I see no sign
of it.  If a 2012-11-07 statement by Nick Hilliard is to be believed:

https://www.ripe.net/ripe/mail/archives/address-policy-wg/2012-November/007374.html

then excluding "the top 10% of the v6 talkers", IPv6 traffic volumes are
an order of 1/100,000 of IPv4 traffic volumes.

I think ILNP represents a logical approach to Locator Identifier
Separation.  (LISP, my Ivip and Fred Templin's IRON are not Locator
Identifier Separation architectures - they are Core Edge Separation
architectures.)  If you accept the impossibility of rapid response to
host mobility, the need for an extra lookup in the common circumstances
I mentioned above, and reliance on the IPv6 Internet, I think it is a
good approach.  The existing BGP system could be retained and there
would be no need to add prefixes to the DFZ for the purposes of
multihoming or mobility.

However I don't believe it can provide the host mobility we need - and
it can't help at all with the IPv4 Internet which everyone uses now and
for the foreseeable future.

I think the IP-address-based host communications and DNS arrangements we
have now are a good thing, in part because it means the recipient host
doesn't have to muck around with costly and time-consuming lookups every
time it replies to a packet in the many cases where that reply must go
only to the host which is identified by the source IP address of the
original packet.

I believe that host mobility (for both IPv4 and IPv6) can be provided
robustly with my TTR Mobility proposal, which is apparently used, in
principle, by IRON for all communications and which could work well with
LISP.  http://www.firstpr.com.au/ip/ivip/#mobile

  - Robin

---

From: Shane Amante <[email-masked]>
Date: Mon, 12 Nov 2012 19:15:58 -0700

Robin,

> an order of 1/100,000 of IPv4 traffic volumes.

Perhaps you missed the following?
http://www.ietf.org/proceedings/83/slides/slides-83-iab-5-technical-plenary.pdf

http://www.extremetech.com/mobile/127213-ipv6-now-deployed-across-entire-t-mobile-us-network

http://conference.apnic.net/__data/assets/pdf_file/0017/50813/vzw_apnic_13462152832-2.pdf
In fact, I would point you to slide 13 + 14 of that PDF as to the success of IPv6 traffic growth.

http://comcast6.net/index.php/8-ipv6-trial-news-and-information/92-deployment-update

Those are just a sampling of recent announcements/activity I can recall recently.  I'm sure if you looked around, you'd see similar activity by other wireless networks and residential broadband providers.

Admittedly, we're still quite a ways from when IPv6 traffic volume will exceed IPv4 traffic growth.  Regardless, it's illogical to (continue to) assume that IPv6 is not being widely deployed in the real-world, right now.

-shane

---

From: Robin Whittle <[email-masked]>
Date: Tue, 13 Nov 2012 15:22:41 +1100

Hi Shane,

Thanks for these links.  The Verizon Wireless slide 14 had no
information on how the IPv6 traffic volume compared with that of IPv4.
Maybe such information was implied in slide 13 but I couldn't clearly
understand it.

Comcast stated that "approximately 6% of the 2012 Olympics served over
YouTube to Comcast customers was over IPv6."  This is a substantial
share of real ordinary user traffic.  I wonder what the impetus for this
was - why didn't the hosts use IPv4?

There will always be IPv4 space available for servers - unless the world
needs a billion or more servers with unique IP addresses, which is not
the case with many websites sharing a single IP address.  That being the
case, as long as mobile devices can be made to work acceptably well
behind IPv4 NAT (maybe this is not the case, but for the foreseeable
future, a unicast IPv6 address is no substitute for a unicast or NATed
IPv4 address), it is hard for me to imagine a commercially compelling
reason for anyone making services available on IPv6.  If there had been
such reasons, there wouldn't be a need for campaigns to improve IPv6
network availability and end-user adoption.

A still more remote development would be anyone making services
available only on IPv6, or in some way better on IPv6 than on IPv4.
That, or the impossibility of mobile and home/office customers being
happy with NATed service, seems to be a pre-requisite for widespread
adoption of IPv6 by end-users who don't run servers - except for one
scenario:  The development of some popular applications which require
true host-to-host communications, especially for mobile devices, which
can't be done with one or the other host behind NAT.  Then, this would
be an impetus for giving such customers (too many customers to
accommodate on their own IPv4 addresses) native IPv6, so each device, or
each of the user's multiple devices, has its own public unicast IP
address.  Such applications are most likely to be intensive real-time
end-user-host to end-user-host communications such as voice, video or
gaming.

With most new developments happening in physically mobile devices, such
applications would need to work smoothly with one or both devices
suddenly getting a whole new /64 and therefore a new IP address.  This
could happen when the device transitions from one base-station to
another (if the base-stations and their controllers have different IP
gateways) - which can happen without the device physically moving at
all.  It could also happen as the device comes into range of one or more
3G/4G/WiFi services come into range - and the ranges can fluctuate
without the device moving.

As far as I know, only TTR mobility can provide this - and it doesn't
require any changes to host stacks or applications (other than the
addition of a TTR tunnelling module alongside or as part of the stack).
 LISP or ILNP would not be able to maintain connectivity in a
sufficiently responsive manner to support any real-time application.
Unless each application handled these mobility problems on its own, I
can't imagine this scenario developing with the current addressing and
routing arrangements, with or without ILNP or LISP.

That's the limit of my current imagination and understanding.

  - Robin

---

From: Shane Amante <[email-masked]>
Date: Mon, 12 Nov 2012 21:44:14 -0700

Robin,

> Maybe such information was implied in slide 13 but I couldn't clearly

No operators share traffic volume statistics publicly, for competitive reasons.  I work as an operator, I should know.  :-)

> was - why didn't the hosts use IPv4?

Ask Comcast?  But, if I were to guess, I would suspect that implementation of "Happy Eyeballs" <https://en.wikipedia.org/wiki/Happy_Eyeballs> in client Operating Systems (Mac OS X) and browsers (i.e.: Chrome, Firefox, etc.) *slightly* favors IPv6 transport compared to IPv4 transport during DNS resolution and connection set-up.  Viola: more v6 traffic than v4.

-shane

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 13 Nov 2012 06:29:31 +0100 (CET)

> an order of 1/100,000 of IPv4 traffic volumes.

Yes, that is the current state of affairs.

However, IPv6 is happening. There is no other plan, people are looking 
into NAT44, NAT444 or equivalent, to bridge the gap until IPv6 is 
supported on enough edge devices, so we then can go dual stacked 
(basically add IPv6 to the beforementioned IPv4 infrastructure), and then 
phase out IPv4 over time. People have stopped saying "if" and started 
asking "how" and "when". This is a good sign.

It tooks us more than 10 years to get here. Anything being discussed right 
now should look 10 years into the future and see itself influensing the 
future, but rely on IPv6 in the current incarnation being the basic 
infrastructure.

We already have multiple IPv4 mobility protocols. 3GPP style tunneling is 
one (I've seen it being re-invented to handle large enterprise Wifi 
networks, at least in the presentations it looks almost identical).

It's my strong opinion that anything that wants to get deployed in the 
real world needs to be "compatible" with the basic IPv6 infrastructure as 
seen today, and overlay on it somehow. It can be host-style like SHIM6, it 
can be tunneling, it can be anything, but I firmly believe the core 
routers need to stay the way they are right now when it comes to 
destination based forwarding with the help of BGP.

The possibility to do rapid adoption of new tech on L3 is over. Anything 
from now on needs to be able to get incrementally implemented using IPv6.

IPv4 is carrying the bulk of real life traffic but IPv6 is where everybody 
is going, that's at least what I see from the operational world. So for 
designing for the 3+ year future, IPv6 is what it needs to handle. Stop 
beating the dying IPv4 horse, it's done a great job but it's being 
replaced and from a design point of view, designing with IPv6 as a 
requirement is a sane assumption.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Robin Whittle <[email-masked]>
Date: Tue, 13 Nov 2012 16:48:47 +1100

Hi Shane,

Thanks very much for your Happy Eyeballs answer in the "Re: [rrg] ILNP:
existing applications & other critiques" thread to my question about why
people watching the Olympics on a Comcast HFC network would sometimes
have their browser, player or whatever access the videos via IPv6 servers:

  http://en.wikipedia.org/wiki/Happy_Eyeballs
  http://tools.ietf.org/html/rfc6555    April 2012

If the IPv4 connection is behind NAT, including with the NAT box in the
provider network, and if the individual hosts have their own public
unicast IPv6 addresses, then this can reduce the traffic through the
IPv4 NAT box.  The algorithm is designed to generally favor IPv6, with
the long-term intention of reducing the use of IPv4.

   "... the client sends two TCP SYNs at the same time over IPv6 and
    IPv4."

If an ACK is received for both, the client selects whether to use IPv4
or IPv6, and caches this for 10 minutes.

   "The simplest venue for the implementation of Happy Eyeballs is
    within the application itself."

It seems that applications need to be altered, by the addition of a
Happy Eyeballs algorithm, to provide reliable performance when the
dual-stack host is connected to both IPv4 and IPv6.  This is
unfortunate, but not surprising - since these are two separate Internets.

  - Robin

---

From: Shane Amante <[email-masked]>
Date: Mon, 12 Nov 2012 23:11:23 -0700

> unfortunate, but not surprising - since these are two separate Internets.

No.  Note that Mac OS X _operating_system_ implements a form of Happy Eyeballs. Thus, there is no need for apps on top of OS X to implement it.  IMO, the only reason apps such as Firefox & Chrome have implemented Happy Eyeballs is because Windows does not implement Happy Eyeballs, so the apps vendors were forced to work around it.

This illustrates the fundamental conundrum we're facing. Apps vendors are making up for the deficiencies of the underlying operating system network stacks.  If we give the O/S vendors something to latch on to, then it makes everyone's lives improved.

-shane

---

From: Robin Whittle <[email-masked]>
Date: Tue, 13 Nov 2012 18:33:35 +1100

Short version: Discussing the implementation of Happy Eyeballs or
               something similar to it in the operating system, rather
               than in the application.

Hi Shane,

> No. Note that Mac OS X _operating_system_ implements a form of Happy
> Eyeballs. Thus, there is no need for apps on top of OS X to implement
> it. IMO, the only reason apps such as Firefox & Chrome have
> implemented Happy Eyeballs is because Windows does not implement
> Happy Eyeballs, so the apps vendors were forced to work around it.
>
> This illustrates the fundamental conundrum we're facing. Apps
> vendors are making up for the deficiencies of the underlying
> operating system network stacks. If we give the O/S vendors something
> to latch on to, then it makes everyone's lives improved.

Maybe it is difficult or perhaps impossible to implement Happy Eyeballs
or the like in the OS in a manner which never causes problems, no matter
the nature of the application.

The example algorithm in section 6 of RFC 6555 involves sending a SYN to
the IPv6 address (assuming the address policy for the host prefers IPv6,
which I understand is typically the case for a dual-stack OS), waiting
300ms and then sending a SYN to the IPv4 address if there was no
response in that time.  From Australia, and especially on a 3G/4G mobile
link, 300ms is a rather short time in which to expect a response from
servers in most of the world.  Assuming the IPv6 address does produce an
ACK which arrives later than 300ms, then the host is sending out two
SYNs instead of one, and likely receiving two SYNs too.  This is
something of an extra burden on a 3G/4G link.

Though Happy Eyeballs is documented for TCP, the Wikipedia page mentions
it being used to choose between TCP and SCTP.  I don't think it could
work with UDP, since there is no SYN/ACK arrangement.  A Happy
Eyeballs-like algorithm could be implemented for UDP with the
involvement of the server, but this would have to be at the application
level.

Perhaps this means that dual-stack problems it is intended to solve
can't be solved for UDP-based protocols without suitable application
changes for the client and server.

This article from 2011-11-12 may not reflect the current behavior of the
"Happy Eyeballs" algorithm (it is actually a different algorithm) in OS-X:

  https://labs.ripe.net/Members/emileaben/hampered-eyeballs

however it does mention how it works, or worked, and some problems which
arose, presumably with applications which use the traditional
getaddrinfo() call, rather than another methods which are apparently
unique to OS-X.

   "It looks like OS X Lion makes debugging connectivity issues to
    dual-stacked websites a royal pain, with the associated cost for
    support staff. While the Mac OS X Lion + Safari combination looks
    to be a true "Happy Eyeballs" implementation, the combination of
    OS X Lion with anything that just uses getaddrinfo() does not make
    the eyeballs much happier it seems."

  "If the way Mac OS X implemented 'Happy Eyeballs'-like functionality
   becomes more widespread (please don't do this Microsoft!) it will
   cause perfectly fine dual-stack hosts with roughly equal performance
   over both protocols, to select the legacy IPv4 protocol in roughly
   half of the cases. This makes the IPv6 capacity in networks
   under-used and that is bad for a  few reasons:

This page reports on problems with OS-X 10.7 (Lion) in April this year.

  http://shibboleth.net/pipermail/dev/2012-April/000651.html

>From 2012-08-10 more reports of trouble with OS-X and particular
browsers - Safari and Firefox:

http://www.jumpertz.net/1/post/2012/11/going-cross-eyed-with-happy-eyeballs.html

A 2012-09-11 Cisco presentation doesn't mention any difficulties:

http://www.data.proidea.org.pl/euronog/2edycja/materials/Krzysztof_Mazepa-IPv4_vs_IPv6-Happy_Eyeballs.pdf

Geoff Huston and George Michaelson (2012-04-16) mention various items of
interest to these questions and mention problems with the OS-X
implementation.

  https://ripe64.ripe.net/presentations/78-2012-04-16-ripe64.pdf

This is not a proper survey of the field.  I am just suggesting that it
may not be straightforward to implement the Happy Eyeballs algorithm in
the OS, especially considering that various apps may access the OS in
different ways, may be doing their own version of Happy Eyeballs etc.
and that the OS approach might upset some assumptions made in higher
level protocols.

  - Robin