Subject: [arch-d] Think globally, act locally

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 11 Nov 2006 00:45:50 +0100

In trying to address the scalability problem identified in the IAB RA  
workshop, we're faced with two challenges: come up with a sound  
architecture, and design and implement something that addresses the  
issue at hand, without unduly getting in the way of the continued  
operation of the network. A clean architecture that can't be  
implemented is just as useless as applying hack on hack until we've  
completely painted ourself into an architectural corner.

Taking a step back, it's a bit strange that there was such strong  
rough consensus that the id/loc overloading is a problem, as  
arguably, this overloading isn't all that pronounced. Rather than  
think of the IP address as both the identifier and the locator, it's  
possible to think of the FQDN as the identifier and the IP address as  
the locator. Or of the IP address as the identifier and the link  
address as the locator. In that light, a new effort to separate  
location and identity isn't something completely new, but rather,  
revisiting and (hopefully) improving something that's already present  
in our current architecture.

In my opinion, architectural sanitation dictates that we at least pay  
lip service to the notion that the current socket API is a terrible  
layer violation, and the proper way to set up sessions and  
associations is by having the application provide the FQDN of the  
correspondent, so that the application is isolated from the structure  
of the addresses used. I.e., for all socket API operations that work  
with addresses, we define alternatives that work with DNS names.  
Taking care in the process that it becomes actually possible to have  
a single code base work over both IPv4 and IPv6 (and IPv8 and so on),  
something that's almost impossible today.

After having done that, we have to face facts and realize that huge  
numbers of applications work with IP addresses, and only the tiniest  
networks don't have IP addresses hardcoded all over the place. It  
would be nice if we could just tell people "don't do that" and then  
have them renumber, but first of all, it doesn't work that way in  
practice (see the various BGP table size reports), and even if it  
did, we'd still have the issue of multihoming, which inflates the  
routing tables regardless of how easy or hard it is to renumber.

It would help, though, if we could give current operators better  
traffic engineering tools. In my opinion, this isn't a particularly  
hard problem, so the IETF should just go ahead and charter a working  
group to develop TE mechanisms that can be bolted on to existing BGP4  
with relative ease. If this saves us a few percent in yearly routing  
table growth, that adds up to significant extra time to work on the  
"real" solution and/or savings on hardware upgrades when amortized  
over a number of years.

Next, another issue that in and of itself isn't going to save the  
day, but is something we've overlooked for too many years: the  
average packet size. Although some other aspects will obviously be  
much the same, a router that forwards 10 Gbps worth of 700 bytes avg  
packets only has to do half the FIB lookups as a router forwarding 10  
Gbps worth of 350 bytes avg packets. Obviously there will always be  
small packets (acks, VoIP, DNS lookups), but a very large part of all  
internet traffic is transfer of data that is at least several  
kilobytes in size, so the transfer is split into several packets of a  
maximum size that is 1500 bytes in the vast majority of cases. This  
value comes from the original Ethernet II 1500 byte payload size.  
(Yes, that's from before the IEEE even came involved with Ethernet.)  
Even though today, modestly priced computers have increased in all  
other dimensions except size and cost by at least a factor of 100  
(including communication speed which is now often 1000 Mbps) Ethernet  
packets are still just 1500 bytes. This is a fine size at 10 Mbps,  
somewhat defensible at 100 Mbps, but completely ridiculous at 1000  
Mbps. This is not entirely trivial to fix, but until now nobody has  
even bothered to try. Again, charter a working group, or have an  
existing working group look at this.

But back to the id/loc thing, as this is the part that will have to  
do the heavy lifting.

The central issue here is that whatever it looks like, it must be  
highly parallelizeable. The problem with the current routing system  
is that it basically requires a full view of the network to be  
present in a single router, and as the network and therefore its full  
view increases in size, so has the router, with all the adverse  
effects described by Tony and others. In the new system, it's  
essential that when the network gets four times as big (measured in  
routing information), it is then possible to continue running the  
network with four devices doing the work previously done by one  
device, where each of the new devices doesn't have to do any more  
work than the one original device in the old situation.

Now this suggests that the place where the change from stable  
identifiers to ephemeral locators happens should be as far out in the  
edge as possible: at least on site exit routers, but maybe even on  
individual hosts. However, there are two issues with this:

1. The people running sites or hosts don't feel the pain from large  
routing tables, so they have little or no incentive to spend time and  
money on a solution like this

2. It is not realistic to push out the complete state of the id->loc  
mappings of the entire network to all sites, or even all multihomed  
sites. This means that the id->loc mapping information must be  
requested on-demand (like the DNS) rather than pushed out throughout  
the network (like BGP). This is a problem for several reasons: it  
takes extra time to request the information, introducing user-visible  
delays, and there is an inherent circularity: in order to ask someone  
for her mapping information, you must first be able to talk to her.  
This can be solved by publishing static mapping information augmented  
by discovering dynamic reachability information (as with the shim6  
REAP protocol), but this only serves to furthr increase delays on  
initial communication.

The alternative is doing the mapping inside service provider  
networks. In my opinion, this has many advantages:

1. The users don't see the locators, so they can't do things that get  
in the way of renumberability

2. The costs are aligned with the benefits: the service providers pay  
for the extra processing, but also reap the benefits in the form of  
smaller routing tables in core routers

3. In service provider networks, there link layer capabilities are  
generally much more advanced than what's common in end-user networks.  
Such capabilities, such as Ethernet VLANs, MPLS, light paths, or even  
IP paths with higher MTUs, can be leveraged to tunnel packets through  
the core.

4. Having the solution be local to a single AS allows for easy  
deployment

So the solution that I envision is something like the following:

For each IPv4 and IPv6 prefix, one of the following must be true:

1. The prefix is present in routing tables throughout the internet so  
that the packet can be forwarded through existing mechanisms, or

2. The prefix is not present in regular routing tables, but it is  
present in a to-define new protocol that distributes id->loc mapping  
data. The prefix functions as a set of identifiers, different types  
of locators may be available: link-based locators (such as MPLS  
tunnels) or IP tunneling/address rewriting.

In the case of 2., the packets are forwarded to a local mapping  
device that is capable of encapsulating packets and transmitting them  
to an equivalent device either at the edge of the local AS where the  
packets can be handed over to the destination AS, or the  
encapsulation is kept intact between ASes and the decapsulation  
happens in the destination AS. This requires a new protocol to flood  
mapping information, which must be dynamic enough to facilitate  
multihoming. But because reachability of intermediate links or ASes  
isn't an issue, the protocol can be much simpler than existing BGP.  
In essence, it functions much like iBGP. Because we get to design  
this new protocol from scratch, we can make sure it's possible to  
have the desired parallel operation where required.

By selectively moving prefixes from the regular routing table to the  
encapsulation devices and back, it's possible to optimize operation  
of the new machinery without requiring renumbering or other invasive  
changes.

When the above is operational, it would be possible to optimize by  
having site exit routers participate in the encapsulation. Most sites  
will probably not be interested in having the mapping state for the  
entire internet, but they can receive feeds with limited mapping data  
views. At this stage, it becomes possible for two sites to detect  
their mutual mapping capability and directly negotiate optimizations  
on-demand. One optimization could be route optimization towards  
mobile nodes. Ultimately, individual hosts may want to participate in  
this process, much like MIPv6, SHIM6 or HIP operation, but aided by  
the flooded mapping information that's available in the site.

An interesting side effect of the encapsulation is that it becomes  
quite easy to expose the location where a packet originated (i.e.,  
the outer source address when doing IP-in-IP tunneling) and if the  
new mapping flooding protocol is designed to be secure, it may very  
well become unnecessary to implement heavy security mechanisms in  
BGP, as BGP will subsequently only be used for PA space from ISPs,  
and the number of PA blocks is small enough to make more traditional  
security mechanisms feasible.

_______________________________________________
Architecture-discuss mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/architecture-discuss