Subject: [rrg] hIPv4 critique (draft)

From: Robin Whittle <[email-masked]>
Date: Wed, 10 Feb 2010 11:15:01 +1100

Hi patte,

Below is a draft critique of your hIPv4 proposal.

  http://tools.ietf.org/html/draft-irtf-rrg-recommendation-04#section-5.1
  http://tools.ietf.org/html/draft-frejborg-hipv4-04

Please let me know your comments and I will post a V2 version to the
list.

The bad news is that unfortunately, as far as I know, it is not
practical to use IPv4 header options in traffic packets traversing
the DFZ.  According to what I have been told in the past by people
who I think are knowledgeable in this field, and according to this
research:

  End-to-end measurements on performance penalties of IPv4 options
  Fransson, P.; Jonsson, A.
  Global Telecommunications Conference, 2004. GLOBECOM apos;04. IEEE
  Volume 3, Issue , 29 Nov.-3 Dec. 2004 Page(s): 1441 - 1447 Vol.3
  10.1109/GLOCOM.2004.1378221

  http://www.sm.luth.se/csee/csn/publications/end_to_end_measurements.pdf

most routers process such packets on the "slow path" with software.
The last sentence in the abstract is:

  From the analysis it can be concluded that there is a slight
  increase in delay and jitter and a severe increase in loss rate.

Since your proposal relies entirely on a new IPv4 header option, as
far as I know, it can't be practical as a solution to the routing
scaling problem.

I didn't read all your proposal, because I want to read and critique
others as soon as I can.  I think it is a worthwhile project to try to
find a way that suitably upgraded hosts can break out of the barriers
imposed by IPv4's 32 bit address space.  I don't think anyone has a
practical way of doing this.  It seems that header options can't be
part of any such arrangement.

  - Robin

hIPv4 Critique (1st draft)
--------------------------

hIPv4 is a scalable routing proposal which is neither a Core-Edge
Elimination nor a Core-Edge Separation architecture.  It uses two
additional 32 bit constructs: Area Locator (ALOC) and "Endpoint
Locator" (ELOC) to enable the re-use or global unicast addresses in
separate regions.  Upgraded host stacks and applications use an
enhanced DNS to communicate with hosts in other regions, with
inter-region packet transfer being achieved via special global
unicast addresses using the current DFZ.

hIPv4 packets contain the ALOC, ELOC and some flags in a new 12 octet
header option.  New, but relatively simple, devices at the borders of
regions can modify the packets in transit, swapping the destination
address for the ALOC or ELOC.

Unfortunately, routers in general - including those in the DFZ -
process IPv4 packets with any kind of header option in the "slow
path", which is via a much slower process than the usual rapid
handling provided by their FIB. [1]  Furthermore, they frequently
drop such packets.  Since hIPv4 relies on traffic packets using
header options, it cannot be considered a practical solution to the
routing scaling problem.

hIPv4 is a creative attempt at making a new set of IP protocols which
are based on, and are backwards compatible with, IPv4.  Multihoming
would be achieved in a manner similar to a Core-Edge Elimination
architecture by a network having two upstream providers and so having
each of its hosts accessible via the host's 32 bit IP address, which
is essentially an Identifier within the scope of one or more regions,
and either of the ISP's ALOCs.  Header options are in principle
capable of being used to extend the protocol and addressing
arrangements of IPv4, but unfortunately are not properly supported by
current routers.

[1] End-to-end measurements on performance penalties of IPv4 options
    Fransson, P.; Jonsson, A.
    Global Telecommunications Conference, 2004. GLOBECOM apos;04. IEEE
    Volume 3, Issue , 29 Nov.-3 Dec. 2004 Page(s): 1441 - 1447 Vol.3
    10.1109/GLOCOM.2004.1378221

    http://www.sm.luth.se/csee/csn/publications/end_to_end_measurements.pdf

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 10 Feb 2010 19:15:59 +0200

Hi Robin,

thanks for taking the time to review, some comments inline

> far as I know, it can't be practical as a solution to the routing

Yes, this is indeed very bad if the hIPv4 packets goes on the slow
path - a serious flaw

But what I had in mind is very specific option, that is option 17 -
Extended Internet Protocol, RFC1385 and the value is 0x8A.
So it is actually not a new option, it is something that has been
defined back in 1992.

It is for me unclear if option 17 has been tested in the report you
mention above, from the report:
"Both O-packets and N-packets are UDP packets with payloads
consisting of a unique sequence number (32 bits) and the
send time, ts, (64 bits). O-packets have the IP header length
set to 6 words and bit 160 to 191 (i.e., the options) in the
IP-header set to 0. Setting the first octet among the options to
0, should be interpreted as “end of option list” and should not
require any processing by routers"

Does 0x8A goes into that or not?

The study seems to test different patterns randomly and not
distinguish between option patterns, are there any studies regarding
only IP option 17?

Since 0x8A is defined and as long the router doesn't support EIP it
shouldn't take those packets from the fastpath and punt them to CPU of
the router - it is something the legacy router should ingnore and just
forward to the next-hop. That is the theory, what about real life
implementations - have the ASIC designers taken into account IP option
17 or not?

If somebody has insight on this please educate me, thanks!

Of course, I should have mentioned option 17 in my draft....sorry for
my bad writing!

> part of any such arrangement.
Well, if option 17 is also sent to the router's CPU you could always
create another header to get around the problem - use a UDP-based
"shim" header :-)
But it would be first nice to know how the FIBs are implemented
regarding this specific option...

-- patte

---

From: Robin Whittle <[email-masked]>
Date: Thu, 11 Feb 2010 11:20:31 +1100

Hi patte,

Thanks for your reply.

As far as I know, there's nothing to change about my critique, so
unless you or anyone else can suggest something, the version at
(msg05990) can be used for the RRG Report.

I didn't look in detail at the Fransson and Jonsson paper, but as far
as I know it is reasonable to assume that any header option will
cause the packet to be processed by the slow path.  Other people on
this list - such as the LISP folks, who I think helped design the
routers most widely used in the DFZ - would be able to provide a more
authoritative answer.

Assuming you can't use a header option, I don't think your proposal
can work, since you need a way of adding extra information into all
packets in a way which will not upset either routers or non-hIPv4
hosts.  The only alternative to a header option seems to be, as you
suggest, a UDP header followed by some information you want to
include - the flags, ALOC and ELOC.  But then the packet would not be
recognised by non-upgraded hosts.

So as far as I know, there's no way of doing what you want - or any
other such approach to introducing a new packet format for extended
IPv4-based addressing which would not upset non-upgraded hosts and
routers.

If IPv4 had been designed with a theoretical 48 or 64 bit address
range, and with the header only containing the lower or upper 32 bits
of this, to be zero-extended to the full length by recipients, then
stacks. APIs and applications could have been ready to handle an
extended header format, or a new header format (IPv5?) when the need
arose to go beyond 32 bits.  Likewise, routers could have been ready
to cope with the new arrangements.

But that was nearly 30 years ago, 8 bit and the newer 16 bit
microprocessors - and the excellent new 32 bit 68000 (with 16 bit
data bus) - were running at 8MHz.  There were slightly more than 2^32
people, and the original coax-cable 10Mbps Ethernet standard had just
been published, with 48 bit MAC addresses.

If we are still boxed in by IPv4's lack of upgrade path in ten or
twenty years, I wouldn't be surprised.

That said, we should recognise what a good design it was, since with
DNS and the BGP-based core routing system, the system works very well
(although in some ways, only just works) decades later, with billions
of users and (I guess) a billion or so IP addresses in active use.
Imagine if there had been no such system.  We might have had to adapt
Microsoft, Apple or Novell networking to a global scale - or
implement a globally agreed subset of OSI.

  - Robin

---

From: Patrick Frejborg <[email-masked]>
Date: Sat, 13 Feb 2010 09:29:32 +0200

Hi Robin,

your critique is valid, I did found some more papers around the usage
of options.
Architecturally it would have been a clean approach to use options,
but in the real world that is not much of an option...

So thanks for pointing this out for me!

It took me back to the drawing board and I created a new header so
that the IP header will be 20 bytes and stay in the fastpath though
there are some extensions added by the endpoints
http://tools.ietf.org/html/draft-frejborg-hipv4-05
The extensions are no longer in the option field, instead a real shim
header is created that the hIPv4 endpoints/nodes will identify by a
new protocol number in the IP header - the upper layer protocol (e.g.
transport) number can be found in the shim header. This shouldn't
upset the intermediate legacy routers, or does it?

About the 32-bit address space, I think it is good enough. In the
pre-RRG world it isn't but here at RRG two issues has been discussed
quite a lot that changed my world quite radically, that is, the
core-edge separation/elimination and the decoupling of
identifier-locators.

-- patte

---

From: Robin Whittle <[email-masked]>
Date: Sun, 14 Feb 2010 01:03:25 +1100

Short version:   The 05 version of hIPv4 involves a new kind of IP
                 header, rather than an IPv4 header option.  This
                 would not be practical either, since no router
                 or host will recognise it.

Hi patte,

I don't want to discourage further development of your proposal, but I
think your version 05 is not at all practical since it involves a new
kind of packet format.  This would require all DFZ and all other routers
to be upgraded, and upgrades to all the stacks of all hosts which use
hIPv4.  I am sure these requirements would rule your proposal out of
consideration as a scalable routing proposal, since we can only plan to
develop something which will be voluntarily adopted on a very wide basis.

> your critique is valid, I did found some more papers around the usage

Can you post the references to to the RRG list?

> So thanks for pointing this out for me!

OK - I wish all IPv4 routers could handle IPv4 header options.

> upset the intermediate legacy routers, or does it?  

They would be "upset" - routers wouldn't know what to do with a packet
with such a header.  They would drop the packet.  There's no reason they
would interpret it like some other header they do know about: IPv4 - and
if they did, it wouldn't work, since your new header is longer than the
IPv4 header.

> core-edge separation/elimination and the decoupling of

I think Locator/Identifier Separation is a bad idea:

   Today's "IP addr. = ID = Loc" naming model should be retained
   http://www.ietf.org/mail-archive/web/rrg/current/msg05864.html

I think the routing scaling problem can be solved for IPv4 (and for
IPv6), at least with my CES architecture:

  http://www.firstpr.com.au/ip/ivip/

I believe that in doing so, we will be able to use IPv4 space more
efficiently than at present, since millions of end-user networks will be
able to scalably get just the amount of space they need, rather than the
minimum 256 addresses they can get today with conventional BGP
techniques.  However, even though more than half of the BGP prefixes are
/24, they total a fraction of a percent of the global unicast address
space.  This means that millions of end-user networks will be
accommodated, scalably, without too much address usage.

Still, the 3.7 billion limit on IPv4 global unicast space remains a
serious long-term problem.  The main thing it can't handle is billions
of cellphones on IPv4.  I am not sure it can handle all the homes and
offices which ultimately want a DSL or fibre or whatever service like
people get today with DSL etc.: a single global unicast IPv4 address for
their NAT box, and with which they can port-forward whichever UDP and
TCP ports they like, since they are not sharing this IPv4 address with
anyone else.

I think 64 bits would be fine for a new addressing system.  I think
IPv6's 128 bits is overkill and makes packets too long, and host
software too complex, since CPUs only deal with 32 and 64 bits.

Anyway, that is academic for now - the best routing scalability
solutions do not require anything as drastic as new addressing systems,
and the consequent need to alter all host-stacks and substantially
rewrite all applications.

  - Robin

---

From: Patrick Frejborg <[email-masked]>
Date: Sun, 14 Feb 2010 18:26:08 +0200

Hi Robin

seems to be some misunderstandings in our dialogue, will try to
explain in more details

> develop something which will be voluntarily adopted on a very wide basis.

It is a new format but the legacy routers will not see the new format
since to them this is a legacy IPv4 header - the only change is that
there is new protocol number in the IPv4 header. And routers shouldn't
care about that, the packets should stay in the fast path as long as
the header length is 20 bytes.
The new protocol number is used by the endpoints and the LSR to
distinguish between IPv4 and hIPv4 packets, when the new protocol
number is used the endpoint and LSR knows that after the IPv4 header
there is a shim header and not a transport protocol header.

> if they did, it wouldn't work, since your new header is longer than the

Nope, the routers shouldn't see it all - the locator header is not
part of the IPv4 header. It is a shim header sitting between the IP
and transport header. Thus the hIPv4 packet stays in the fast lane and
gets routed through the legacy network, the LSR and destination
endpoint will know by the new protocol number that there is an
additional header before the transport protocol

>   http://www.ietf.org/mail-archive/web/rrg/current/msg05864.html

Here I disagree, think the HTTP cookie mechanism is great and a
generic model should be created to enhance endpoint mobility. If the
identifier is in the address space you have to use an overlay solution
such as MIP. It matters where you place the identifier in the TCP/IP
model

> TCP ports they like, since they are not sharing this IPv4 address with
Have you ever wondered why PSTN doesn't suffer from this kind of
problem - how can they have billions and billions of landline and
cellphones hooked into the PSTN?
Over here my cellphone number is mine, I can choose any cellco and my
numbers stays the same when I change cellco
http://www.numpac.fi/index.php?site=127

> rewrite all applications.

Exactly.

If you create a two level hierarchy you can preserve the current
addressing scheme and still create a new routing architecture. If you
reserve a 256k/512k/1M address space for ALOC then each ISP can have
their own IPv4 address space for their customers. With one IPv4
address space 1,7 billion users have been able to enjoy the Internet
experience, with e.g. 512k times more IPv4 address spaces we should be
able provide Internet experience for everyone.
It will have implications though - you can't share routing information
of every prefix with every one - then you would run into overlapping
prefixes. Thus one ISP can have only its directly attached customers
in their RIB, the ISP will not accept any routing information from
other ISPs except their ALOC information - the PI and PA prefixes from
other ISP stays within each ISP's RIB. Also, if I'm running an ISP -
why should I be interested if a certain prefix of another ISP is
installed in my RIB or not (using power, space, cooling and FIB
resources if installed)? For me it is enough that I can find the other
ISP (by the ALOC), send the packets over to that ISP and they will
deal with the local forwarding  - it is their problem to route the
packets to the final destination, it is not my problem and actually I
don't care how they do it. The outcome is that the RIB of  core-DFZ is
reduced quite a lot, also the amount of paths are suppressed in the
DFZ because the edge prefixes (PA and PI addresses) are aggregated
behind one ALOC prefix.
This will have implications for multi-homing solutions, but here MPTCP
comes to rescue

-- patte

---

From: Robin Whittle <[email-masked]>
Date: Mon, 15 Feb 2010 12:12:25 +1100

Hi patte,

> seems to be some misunderstandings in our dialogue, will try to

Sorry - I misunderstood your proposal for a new 40 byte header format:

  http://tools.ietf.org/html/draft-frejborg-hipv4-05#section-6

I had thought that this header began with a different 4 bit Version
code, but I understand it uses exactly the same Version bits as the IPv4
header.

I now understand you are defining a new 20 bit header to follow the IPv4
header:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |A|P|S|VLB|L| R |    Protocol   |         LH Checksum           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Area Locator                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Endpoint Locator                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  Private Locator Referral                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

which would have an 8 bit protocol number assigned from:

  http://www.iana.org/assignments/protocol-numbers/

so the protocol field in the IPv4 header would contain the number for
this hIPv5 header.

I think this could be done.  I guess that DFZ and other routers could
handle the packet OK - can any router experts confirm this?  But please
see (msg05993) for mention of the idea that traffic packets which don't
use UDP or TCP headers would run into difficulties with (or at least,
not make best use of?) sets of routers using ECMP/LAG.

I am referring to the [Diff2] of your 05 draft to see how your proposal
has changed with the new header arrangement.

As long as you can be sure that the new packet format is only sent to
hIPv4 hosts, I think you should be able to do whatever you like with it.
 As I understand your proposal, you are relying on some kind of DNS
mechanism for this: "How a hIPv4 session is established follows:".

I am trying to work on GLI-Split at present, so I won't try to think
about your proposal more detail right now.

> there is a shim header and not a transport protocol header.

OK.  At present I can't see any objection to the new header arrangement,
provided it is only sent to hIPv4 hosts.

> additional header before the transport protocol

OK.

> generic model should be created to enhance endpoint mobility. 

OK - This is an upper layer protocol (ULP) approach to
"Locator/Identifier Separation" in which the cookie is the Identifier
used in a persistent session.  The actual end-point IP addresses then
become unimportant.  I support this form of "Locator/Identifier
Separation" - but not converting the basic IP protocols to such an
arrangement.

> If the identifier is in the address space you have to use an overlay
> solution such as MIP. It matters where you place the identifier in the
> TCP/IP model

I think with mobility, it is important to minimise the tasks which must
be preformed by the MN, and to make it work for IPv4 and IPv6 with
ordinary (no-upgrades to stack or applications) correspondent hosts,
while generally having optimal or close to optimal paths.  The only
approach I know which can do this is:

  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

> http://www.numpac.fi/index.php?site=127

When I last looked at it, I concluded that number portability in the
PSTN is a technical and probably administrative nightmare.  I also think
it is a necessity, for reasons of enabling competition.  But the PSTN
was never meant to support such arbitrary assignments of services in the
same number range to completely different networks.

The PSTN has a variable length addressing system in which the roles of
Identifier and Locator are both played by the number itself.  The
text-name role is played by printed and web-based telephone directories,
and dial-up directory assistance.

As I understand your proposal, you are plunking an optional 32 bit field
to the left (more significant) of the current IPv4 address.  So only
hIPv4 hosts can work from these new number ranges - but they can also
work like ordinary IPv4 hosts in the base IPv4 address range, which has
no such extension.

There is another approach to what you are suggesting - adding a 32 bit
field to the right - making a new addressing system where each existing
IPv4 address could "contain" 4 billion of the new kind of addresses.

> their own IPv4 address space for their customers. 

OK - but only if all the customers have their hosts with hIPv4 stacks
and all their applications are upgraded to cope with hIPv4 too.  Both of
these seem impossible to bring about.

> With one IPv4

Yes - and all their pets!

> DFZ because the edge prefixes (PA and PI addresses) are aggregated

OK - I think I have a rough picture of this.  In the future I hope to be
able to think about it more, but I would also want to explore options
for giving multiple new addresses behind each existing IPv4 address.

> This will have implications for multi-homing solutions, but here MPTCP

Since you are proposing rewritten stacks and applications for all hosts
which participate in your architecture, you can also require that most
or all existing apps use the hIPv4 version of MPTCP to do their own
multihoming.  However, as I wrote in the above-mentioned (msg05864) I am
opposed to this in general, since I think the routing system should
handle multihoming and mobility, not individual hosts, much less
applications

  - Robin

---

From: Patrick Frejborg <[email-masked]>
Date: Mon, 15 Feb 2010 13:46:49 +0200

Hi Robin

> not make best use of?) sets of routers using ECMP/LAG.
Ok, will have a look on that one

> mechanism for this: "How a hIPv4 session is established follows:".
Yes, if the destination host is in the same realm - use IPv4
(regardless if the host is upgraded or not)
 If in another realm and the destination host is upgraded - use extensions

> about your proposal more detail right now.

Ok, fair enough

> and dial-up directory assistance.
If you look a little closer there is a hierarchy: country code, area
code and subscribers

I also agree about the admin nightmare, I had the opportunity to do
SIP-I testing with a MVNO and got a look under the hood. It is
definitely not that straightforward when you setup things, including
IN servers.....now when reflecting over the IN-server solution it
starts to have a lot in common with a mapping database in a CES
solution....

With hIPv4 the admin burden changes, the RIRs do not to take care of
the PA-addresses anymore - as long as the PA-address spaces are
defined then it is up to the ISP how they use them internally.
PI-address admin work doesn't go away, it could have more issues than
today - depends what policies are defined by the RIRs

> handle multihoming and mobility, not individual hosts, much less

I don't see why you would need to rewrite applications for hIPv4 or
MPTCP - both are actually still IPv4 compatible. But of course some
applications will suffer such as IPsec and SIP - SIP is easier to fix
since it can accept extensions, IPsec is much harder but most people
have (or have plans) migrated to TLS based VPN solutions for remote
access. The LAN2LAN IPsec will suffer, that's for sure. But something
has to be sacrificed, think what ever is done, you have to give up
something.

-- patte

---

From: Robin Whittle <[email-masked]>
Date: Mon, 15 Feb 2010 23:23:36 +1100

Hi patte,

> If you look a little closer there is a hierarchy: country code, area

Here in Australia, certain prefixes such as 0438 xxx xxx are cellphone
numbers for the carrier Telstra while others such as 0499 (a guess) are
for another carrier such as Optus or Vodafone.  So to do number
portability, the call to some 0438 xxx xxx number can't be connected to
"an exchange which handles Telstra's mobile prefixes", or "the Telstra
exchange for 0438" as it was the case before number portability -
because maybe an 0438 number is now handled by Optus.  Some years ago
when I looked at how number portability worked, I concluded it was
really ugly - with cached database lookups occurring all the time.  I
recall it looked like a hack, which was demanded by regulators.  This
puts up the cost of all calls, but I think it is probably worth it to
increase competition by means of people being able to keep their numbers
when they choose another carrier.

> starts to have a lot in common with a mapping database in a CES

Whatever the difficulties with SIP, consider doing it with CCS7
exchanges which were never designed to do anything of the sort.

> today - depends what policies are defined by the RIRs

I am not following this, but it is late and I am in the middle of GLI-Split.

> has to be sacrificed, think what ever is done, you have to give up

I was just indicating that MPTCP can't be of any help to existing
applications unless they are radically (I guess) rewritten to use MPTCP
- and then to use the hIPv4 version of MPTCP which I guess would be
needed to handle destination hosts with hIPv4 addresses.

  - Robin