Subject: [Int-area] Architectural reasons why tunnelling is an indication of a failure

From: Pekka Nikander <[email-masked]>
Date: Tue, 2 Aug 2005 11:06:29 +0200

As I have been touting my opinion that tunnelling is an indication of  
deficiencies in the architecture, I want to express the reasons why I  
believe so.  Note that I am _not_ saying that tunnelling is _evil_.   
Tunnelling is often fine as a short term solution, and sometimes also  
as a longer term solution.  However, independent of that,  tunnelling  
(almost) always indicates that the architecture does not provide  
functionality that it should provide.

While there are different reasons for tunnelling, I basically believe  
that all of them tend to be indications of faults in the  
architecture, or sometimes people failing to understand or appreciate  
the architecture.  The only potential exception is when you one is  
tunnelling a local-use lower-layer protocol, such as Ethernet, over a  
wide area global protocol, such as IP, but even that case is  
debatable, in my opinion.  Also note that when one "tunnels"  
something over a lower layer protocol, such as "tunnelling" IP over  
MPLS, one is really misusing the terminology.  Running a protocol  
over a lower layer protocol is not really a case of tunnelling but  
exactly the way a layered architecture is supposed to work.

Now, the forms of tunnelling that I consider are the following:

1. Tunnelling over the very same protocol; e.g., IP-over-IP

2. Tunnelling a protocol over one at the same layer;
    e.g., IPv6 over IPv4

3. Tunnelling a global lower-layer protocol over a higher layer
    protocol; e.g., IP over DNS, IP over UDP, or IP over HTTP

4. Tunnelling a local-scope lower-layer protocol over a global
    high layer protocol; e.g. Ethernet over IP

Sure, there may be other cases, ones that I am missing, and my  
argumentation may not apply to them.

Case 1:  Tunnelling a protocol over itself
==========================================

Examples of this include IPsec in tunnel mode, GTP, and other forms  
of IP-over-IP tunnelling.   My basic concern with this kind of  
tunnelling is that it enforces two different _semantics_ to a single  
identifier space, and therefore makes the system more brittle.   
Conversely, the architecture should provide functionality so that  
such overloaded semantics and brittleness is not needed.

In the case of IPsec, tunnelling seems primarily to be used:

   - to separate local, trusted address space from a global,
     hostile address space

   - to make sure that the local addresses are really
     trustworthy, i.e., to prevent others from injecting
     addresses with local addresses to the local network

   - to hide the details of traffic between the tunnelled sites

However, those are not the root causes.  To find the real reasons  
behind tunnelling in this case we have to consider why people want to  
split the address space in the first place?  The reason seems to be  
that we have no mechanism in the current architecture to provide  
assurance that the source address of a packet really indicates its  
real source.  In other words, (source) addresses in the wild Internet  
are not worth much, and therefore it is preferably to divide the  
address space into trustworthy and non-trustworthy ones.

What comes to the desire to hide the details of the traffic, we have  
to go somewhat deeper still.  The IP architecture does not support IP- 
level privacy, and I claim that it should.  That is, it is far too  
easy to use an IP address to figure out who are the parties  
communicating.  Adding IP-level privacy to the architecture requires  
would require large changes, and therefore people might argue it  
being a non-goal.  Maybe so, but IMHO the existence of tunnelling,  
even here, still indicates a deficiency.  [NB: One way to add IP- 
level privacy would be to have a separate cryptographically protected  
identity layer on the top of IP layer, with build-in address-agility  
(mobility) mechanism, and then start changing IP addresses on a  
fairly fast cycle.]

Another reason for IP-over-IP tunnelling is mobility, e.g., in GTP  
and the proposed NETLMM work.  Both of them are, in my opinion,  
indications that we are relying too much on IP addresses as  
identifiers.  Hence, they indicate deficiencies in the architecture  
in the sense that locators should not be used for host  
identification, thereby removing the need for keeping stable IP  
addresses when a host moves to a topologically different location.

There are probably other reasons for tunnelling a protocol over  
itself, too, but my gut feeling is that in each case a close analysis  
will just reveal that the architecture is either deficient or is  
missing some functionality that should be there.  For example, I am  
pretty confident that similar analysis could be applied to similar  
tunnelling on other layers, such as tunnelling TCP with SSH.

So, tunnelling a protocol over itself is typically indication that  
either a layer of indirection is missing (as in the case of mobility- 
motivated tunnelling) or the layer in question does not provide the  
required security functionality.

2. Tunnelling a protocol over a sister protocol at the same layer
=================================================================

In my mind the prime example of same-layer sister-protocol tunnelling  
is IPv6-over-IPv4 tunnelling.  In this case, what needs to be done is  
adding a common layer either below or above the layer in question.   
HIP is a good example of this; if HIP is used, no IP-cross-version- 
tunnelling is needed since HIP runs equally on both of the two  
versions of IP and even hides the difference from upper layer protocols.

3. Tunnelling a *global* protocol over an upper layer protocol
==============================================================

Example cases in this space are IP-over-DNS and the multiple ways of  
tunnelling IP and other protocols over HTTP.  AFAICT, almost always  
the architecture is failing here in two ways simultaneously.   
Firstly, people are using tunnelling to overcome what they feel as  
overly restrictive security measures.  Conversely, the security  
methods that the administrators have placed there are failing due to  
tunnelling.  In my interpretation, these two sides of the coin both  
indicate that the architecture is lacking suitable security  
functionality.

Other common examples in this space include the various methods for  
tunnelling protocols over UDP (and even TCP or not-quite-TCP) due to  
NATs.  Again, these are indications of a failure of the architecture,  
in this case failure to respond to the shortage of addresses and/or  
to accommodate NATs into the architecture.

4. Tunnelling a local (link layer) protocol over an upper layer protocol
========================================================================

The prime example in this space is tunnelling Ethernet over IP.

This is the only space where I think that tunnelling is often  
appropriate.  However, even here I am tempted to consider tunnelling  
as a failure of the architecture.  Basically, it is an indication  
that there is a need to provide functionality that is available over  
the local protocol (e.g. Ethernet) over wide area links.  The failure  
is that the functionality at hand is not available over the wide area  
(global) protocol while it should be.  However, this is a minor  
issue, and more often an implementation issue than really an  
architectural issue.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

So, my claim is that tunnelling is (almost) always an indication that  
the architecture has one or more of the following deficiencies:

   1. A layer (of indirection) is missing

   2. Some security functionality is missing

   3. The architecture has failed to provide sufficient resources

   4. Some functionality is provided at or on-the-top-of a wrong layer

All of the discussion above is really architectural in nature.  The  
starting point is the layering model, and the discussion is mostly  
about how the layering fails and what are the reasons, in each case,  
for the failure.

There are also technical reasons to resist tunnelling and try to  
pursue people to work on alternative solutions.  Pekka Savola  
discussed some of these at the INT area meeting on Monday, including  
problems with fragmentation and wasting resources.  However, I'd  
prefer to keep that discussion separate and focus in this thread only  
on the architectural issues.

--Pekka

_______________________________________________
Int-area mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/int-area

---

From: Tony Li <[email-masked]>
Date: Tue, 02 Aug 2005 14:53:05 +0200

Pekka,

> functionality that it should provide.

I view tunneling as a means of virtualizing topology.  In some cases,
this virtualization is both productive and the best possible mechanism
for providing the desired service.  In other cases, as with any
technology, tunneling can be thoroughly abused.  I think that it's very
important to carefully distinguish these two applications of the general
concept.

I disagree that the application of tunneling necessarily implies a
failure of the architecture.  It certainly CAN indicate an architectural
failure, but in other cases, it is merely the manifestation of
virtualization in our particular technology, and thus is fundamentally
part of the architecture itself.

> Case 1:  Tunnelling a protocol over itself

More generally put:

	- to create a virtual topology

	- to create a virtual address space, for private use

> assurance that the source address of a packet really indicates its  real
> source.  In other words, (source) addresses in the wild Internet  are
> not worth much, and therefore it is preferably to divide the  address
> space into trustworthy and non-trustworthy ones.

In many cases, users wish to have their own private address space that
is wholly independent of the normal Internet address space.  While we
are usually focused on the deployment of IP in the Internet, we are also
responsible for the use of IP technology that is not visible from the
Internet.  You may well argue that there are questionable security
advantages obtained by not having fully accessible addresses while still
be electrically/optically connected to the Internet, but this is a
capability that many people find to be extremely useful especially when
firewalls are in use.  I do not want to reargue the architectural
implications of firewalls with you, I just want to point out that they
do exist, and they are a substantial contributor to this case.

> Another reason for IP-over-IP tunnelling is mobility, e.g., in GTP  and
> the proposed NETLMM work.  Both of them are, in my opinion,  indications
> that we are relying too much on IP addresses as  identifiers.  Hence,
> they indicate deficiencies in the architecture  in the sense that
> locators should not be used for host  identification, thereby removing
> the need for keeping stable IP  addresses when a host moves to a
> topologically different location.

Here I have to agree with you, the lack of a separation between locator
and identifier causes us to tunnel to provide an explicit locator.  I
think that this feeds into the argument that Geoff Huston was making
directly, and I won't continue it here.

> required security functionality.

I have to ask if security functionality is necessarily a requirement of
the network layer.  I can argue this one both ways, but I don't think
that it is at all clear cut.  Given that not every single packet needs
to be encrypted, that would seem to imply that encryption is the
exception, not the rule, and thus an intermediate layer between network
and transport for providing security is the correct approach.  This is
effectively what we have already with IPsec today, it's just not widely
acknowledged as an architectural component.

Another very useful application of IP over IP tunneling has been the
creation of virtual topology for early deployment and testing of
technology.  We have done this to good effect for both BGP and
multicast.  Is the fact that we cannot pragmatically enable a protocol
across the full Internet topology an architectural flaw?  Or is it
simply a pragmatic compromise?

> adding a common layer either below or above the layer in question.   HIP
> is a good example of this; if HIP is used, no IP-cross-version-
> tunnelling is needed since HIP runs equally on both of the two  versions
> of IP and even hides the difference from upper layer protocols.

As we have seen with IPv6 and many other network layer protocols (e.g.,
private tunneling of AppleTalk) creating a virtual topology is an
extremely constructive mechanism for using the Internet as a global
transport mechanism.  You may well argue that that's an architectural
flaw of the payload protocol, but from an IP perspective, it simply
seems to be another service (and value) that our architecture is
prepared to provide.  I'd claim that this case is a clear benefit.

> in this case failure to respond to the shortage of addresses and/or  to
> accommodate NATs into the architecture.

In this case, I will not argue with you at all.  These are all clearly
architectural failures.  However, they exist and cannot be undone at
this point.

> appropriate.  However, even here I am tempted to consider tunnelling  as
> a failure of the architecture.  Basically, it is an indication  that
> there is a need to provide functionality that is available over  the
> local protocol (e.g. Ethernet) over wide area links.  The failure  is
> that the functionality at hand is not available over the wide area 
> (global) protocol while it should be.  

That functionality may be something as simple as a different network
layer protocol.  It is hardly an architectural failure of the Internet
that we have not absorbed all other applications, much tho we would all
like to.

Regards,
Tony

---

From: Joe Touch <[email-masked]>
Date: Tue, 02 Aug 2005 06:23:43 -0700

> functionality that it should provide.

That is like saying that VM is an indication that memory is deficient,
or that an OS is an indication of a deficiency of running on a bare system.

Tunneling is just the networking equivalent of VM or an OS - a way to
abstract architecture, to allow flexibility, concurrent use for
different purposes, etc.

...
> Examples of this include IPsec in tunnel mode, GTP, and other forms  of
> IP-over-IP tunnelling.   My basic concern with this kind of  tunnelling
> is that it enforces two different _semantics_ to a single  identifier
> space, and therefore makes the system more brittle.   Conversely, the
> architecture should provide functionality so that  such overloaded
> semantics and brittleness is not needed.

The address spaces are different, so different semantics are OK. The
address inside the tunnel is (or should be) interpreted in the context
of the tunnel, just as an applications memory address is interpreted in
the context of the page table.

...
> about how the layering fails and what are the reasons, in each case, 

Sure - and the way we deploy new services and architectures is to deploy
them on tunnels on top of the existing architectures. That's a feature,
not a bug. Tunnels are tools that can be misused or not, but their use
is not in of itself an indication of anything other than growth.

Joe

---

From: Pekka Savola <[email-masked]>
Date: Tue, 2 Aug 2005 16:34:59 +0300 (EEST)

> the context of the page table.

I think Pekka N.'s point here was "why do we need different address 
spaces?" or "why do users want different address spaces?"

Now, my response to that is..

You can certainly see that as an architectural issue, but personally 
and pragmatically, the answer probably is that the cost/benefit of any 
other solution (e.g., for simple access controls etc.) is too high.

That is, there could be other solutions which allow the users the 
same benefits as separate address spaces, but getting them used and 
deployed is probably too costly to be useful -- and the simpler 
approach is to just embrace the separate address spaces in the 
architecture (to some extent in any case).

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Tue, 2 Aug 2005 15:39:26 +0200

Pekka,

> 1. Tunnelling over the very same protocol; e.g., IP-over-IP

I think there are operational reasons why might want to do this.  
Building an architecture based on this is probably flawed though.

> 2. Tunnelling a protocol over one at the same layer;

Here I disagree. This is not a problem of architecture but a clear  
example (subcase) of 1). This is not a flaw in the architecture.

>    protocol; e.g., IP over DNS, IP over UDP, or IP over HTTP

This is a flaw.

>    high layer protocol; e.g. Ethernet over IP

I agree that this probably is a flaw in the architecture but it is  
also currently used for cases where there is little or no choice.  
Implementing whole-sale DSL+PPPoE comes to mind (if I have  
misunderstood the use of tunneling here, please feel free to correct  
me).

- kurtis -

---

From: Joe Touch <[email-masked]>
Date: Tue, 02 Aug 2005 06:44:25 -0700

> spaces?" or "why do users want different address spaces?"

See VM ;-)

We also have plenty of cases - P2P nets use other IDs. Multicast adds a
set of addresses that the base doesn't handle. Deploying any new
protocol - most recently IPv6 - using IPv4 as a base.

Joe

---

From: "James Kempf" <[email-masked]>
Date: Tue, 2 Aug 2005 07:58:10 -0700

Pekka,

I agree with Joe and Tony. Tunnels are a tool for virtualizing the address
space. If you are going to propose that they are a flawed tool, then I think
you need to propose an alternative that has "better" (for some sense of the
word)  properties. The only alternative I can think of (swapping IP
addresses in the header, i.e. NAT) is worse,  but maybe there are other
alternatives.

            jak

---

From: Mark Townsley <[email-masked]>
Date: Tue, 02 Aug 2005 17:12:25 +0200

>space. If you are going to propose that they are a flawed tool, then I think
Tunnels are used for more than just this. And, it's not the tool that it 
flawed per se, Pekka is suggesting that perhaps there is something else 
that is flawed hence we need to apply the tool of Tunneling. We should 
look for that "something else."

- Mark

>alternative

---

From: "James Kempf" <[email-masked]>
Date: Tue, 2 Aug 2005 08:41:44 -0700

> Tunnels are used for more than just this. And, it's not the tool that it

It might help to enumerate these, or, perhaps, to define which of the uses
in Pekka's original note correspond to virtualization and which to some
other use.

> look for that "something else."

Well, to explore Joe's analogy with virtual memory a bit deeper, virtual
memory is needed because the architecture of von Neuman machines consists of
a central processor and a memory store organized as a linear address space
with fixed addresses. This architecture only allows a single process to run
at one time. Virtual memory allows multiple process images to be mapped
sequentially and in parallel into the fixed linear address space, and thus
to have multiple tasks running.

One could argue that the nature of the basic IP network architecture, fixed
addresses located within a static subnet topology, only supports static
hosts and networks at fixed locations. Virtualization via tunnels allows
mobility of networks and hosts, among other things.

Now, I suppose one could argue that something is broken in the basic IP
nework architecture, but, from an engineering standpoint, I'd observe that
changing that might be as hard as changing the von Neuman machine
architecture. Possible, but likely to take a long time.

BTW, in my previous email, I mentioned only address translation as an
alternative tool for virtualization. I think one could argue (and perhaps
Pekka would agree?) that HIP is also such a tool. As it currently exists,
HITs are identified to hosts, but one could expand the concept to better
support address space virtualization so that they identifiy larger
topological aggregates. But, at that point, one has to ask what is the
difference between a 128 bit HIT and a 128 bit IPv6 address.

                      jak

---

From: Scott W Brim <[email-masked]>
Date: Wed, 03 Aug 2005 09:09:21 +0200

> addresses in the header, i.e. NAT) is worse,  but maybe there are other

This is where I come down.  I was revolted when tunnels became
architecture (I believe I argued with Steve Deering about it in Santa
Fe), and I used to think of MPLS as a sign of shortcomings in IP
routing and addressing, but since then requirements have changed.  We
now have to handle "network of networks" scenarios, where you need
complete isolation of client and server "networks".  We could invent
other ways to do this, but in the end they would have the features we
currently call "tunnels".

As an exercise it would be good to explore thinking about these
mechanisms as other than "tunnels".  Perhaps a different conceptual
view would be a better base for going forward architecturally.

swb

---

From: Joe Touch <[email-masked]>
Date: Wed, 03 Aug 2005 05:54:47 -0700

> Building an architecture based on this is probably flawed though.

IP-over-IP may be a confusing way to say what you get, which is more like:
	"IP++ over IP"

IP++ gives you more than the base IP:

	- topology abstraction
	- isolating a set of addresses or services
	- emulation

I.e., VMware will be complete only when we can run VMware inside a
VMware partition. The same is true for networks, which is why IP-over-IP
is the most meaningful virtualization of all, IMO.

Joe

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Wed, 3 Aug 2005 18:55:34 +0200

> is the most meaningful virtualization of all, IMO.

Yes, but those are operational reasons. Not architectural. At least  
to me.

- kurtis -

---

From: Pekka Nikander <[email-masked]>
Date: Mon, 8 Aug 2005 13:58:25 +0200

Brief summary of the discussion so far:
---------------------------------------

I made the claim that tunnelling, while being sometimes (but not  
always) a fine tool, is an indication that the architecture has  
shortcomings, and therefore has "failed" in some sense of the word.   
[Apparently the word "failure" was too strong, but at least it  
generated some discussion :-)]

A number of people replied that tunnels are a useful tool for  
virtualising the address space, similar to virtual memory in von  
Neuman architecture.  One could claim it even to be similar to  
virtual machines.

Clarifications on my side:
--------------------------

I am definitely not saying that tunnelling is bad and that we should  
ban it; quite converse.  As several people mentioned it is a fine  
tool, often allowing much faster evolution and often leading to  
easier operations.

What I am trying to do is to understand the pressures that lead to  
the currently very common use of tunnelling (with its apparent more  
technical problems).  One way to express my feeling is the  
following:  I think that we should not need tunnelling; i.e., our  
architecture should be powerful enough so that (most current) needs  
of tunnelling would disappear.

In this message I am still trying to argue in favour of my position  
that tunnelling is an indication of something missing, mainly  
pointing out that there are alternatives to virtualisation, both in  
computer and network architecture.

If we can go further and try to understand the alternatives and there  
potential implications, that would be even better.

Going further:
--------------

I think comparing tunnelling to VM is a good analogy, and certainly  
something that I missed in my original thinking.  [Thanks to  
everybody that pointed it out.]  However, I also think that we are  
not doing a very good job there.

I am framing my argumentation below as follows:

1. There are architectural alternatives to today's virtual memory.
2. Virtual machine is more than virtual memory.
3. None of our current tunnelling practises come even close to the  
level of abstraction provided by either the virtual memory or virtual  
machine abstractions.
4. Even if we may get a clean virtualisation architecture with IPv6,  
we are not there yet, and may not be clear where, exactly, we should  
be going.

Details:

1. Architectural alternatives to virtual memory
===============================================

If we think about today's virtual memory systems it consist of two  
apparent mechanisms that seem to be orthogonal:

  a. Mapping virtual addresses to (different) real addresses
  b. Protection bits, providing lots of functionality

If we consider address mapping first, there are certainly  
architectural alternatives.  For example, if you have a large enough  
address space, you could use position independent code.  With proper  
support at the processor level that can be done as convenient as  
virtual-memory-based mapping.  The amount of hardware needed is  
basically the same; the difference is on the abstraction.

I am not going to the protection part.  It is apparently a very  
useful abstraction, providing lots of useful functionality such as  
copy-on-write, flush-on-dirty, etc.

The point is that even a good virtualisation abstraction seems to  
consist of several components, some of which don't _necessarily_  
imply others.

2. Virtual machine is more than virtual memory
==============================================

The majority of today's hardware does not support full virtual  
machine functionality, i.e., virtualising also the protected mode.   
In the main stream architectures it has been appearing only  
recently.  Consequently, many of today's virtual machine products  
either use software to "cheat", resulting in architecturally ugly  
(but useful!) solutions, or are fully implemented in software, with  
the apparent performance penalty.

My point here is full virtualisation, such as in the case of virtual  
machine, requires much more than just virtualising the address space.

3. Our current tunnelling practises are not very clean
======================================================

Returning now to the current IP-over-IP tunnelling practises (which  
seem to be closest to virtualisation), I claim we could do much better.

The first issue, which I already touched, relates to muddled address  
space semantics.  While one could argue that RFC 1918 addresses are  
like the virtual address space in a virtual memory system, that  
AFAICT doesn't quite hold.  The difference lies in how you  
communicate with components outside of that virtual space.

Having a very large address space with no overlapping addresses seems  
to lead to a cleaner architecture than mapping address spaces.  For  
us, going to IPv6, Unique Local IPv6 addresses (draft-ietf-ipv6- 
unique-local-addr-XX.txt; sometimes called Hinden-Haberman addresses)  
may help.

If you go there and get rid of RFC 1918 addresses, returning to  
addresses that are globally unique, what remains is the question of  
routing.  For practical reasons, it looks like that people may need  
to use tunnelling even for Unique Local IPv6 addresses.  The reason  
for this is that the routing system does not easily allow them to be  
globally routable, which, in turn, leads to another architectural  
question:

   Should the routing system better support local-only routes?

[There are lots of potential answers to both the narrower practical  
question and to the larger architectural one; e.g. use MPLS, use  
source routing, etc., but that is not the point of this message.  The  
point here is that the fact that tunnelling is needed may, again, be  
an indication that some functionality is missing elsewhere, in this  
case in routing.]

A second (perhaps converse) issue here relates to layering.  Some  
people expressed their belief that IP is such a useful service that  
building virtual topologies over an existing one is beneficial.   
Certainly so, but my question is now much of this is caused by  
current architecture, i.e., the fact that IP addresses are  
simultaneously used as identifiers and locators.  If these two roles  
were separated (most probably leading to two having (sub)layers  
instead of the currently one IP layer), what would that virtual  
topology bring?

My claim is that if we did the split, there would be much less need  
for "managing" the IP address space, leading to a situation where  
most (if not all) long term needs for IP virtualisation would disappear.

4. Full virtualisation for IPv6
===============================

A major question now appears to be what we should do about IPv6 and  
tunnelling?  Based on the discussion to far it appears to me that it  
may be a good idea to try to figure out a full virtualisation  
architecture for IPv6, i.e, one resembling virtual machines (instead  
of virtual memory).  That leads to a number of more practical questions:

- Who would need it?
- What would be the requirements that it would fulfil?
   - e.g. what do people need topology abstraction for?
   - what level of emulation are we expecting
- What is missing (if anything) in the current solutions we have?

Maybe all we need is a BCP explaining how to use existing components  
in the best possible way for virtualising IPv6 in a local network,  
with appropriate barriers between the local and outside world?

But before that I'd like to here what are the real benefits that  
people would expect to get from it, as I am personally having hard  
time seeing them.

--Pekka

---

From: Joe Touch <[email-masked]>
Date: Tue, 09 Aug 2005 14:47:44 -0700

Pekka Nikander wrote:
> virtualising the address space, similar to virtual memory in von  Neuman
> architecture.  One could claim it even to be similar to  virtual machines.
> 
> ban it; quite converse.  As several people mentioned it is a fine  tool,
> often allowing much faster evolution and often leading to  easier
> operations.
> 
> What I am trying to do is to understand the pressures that lead to  the
> currently very common use of tunnelling (with its apparent more 
> technical problems).  One way to express my feeling is the  following: 
> I think that we should not need tunnelling; i.e., our  architecture
> should be powerful enough so that (most current) needs  of tunnelling
> would disappear.

That's like, IMO, saying that a sufficiently powerful computer
architecture would not need virtual memory.

The current uses of tunneling may be point solutions, rather than more
comprehensive architectural extensions, but many (not having done an
exhaustive search, I won't say 'all') are based on the need for exactly
the layer of abstraction and address isolation that virtualization provides.

Again, recalling VM, earlier point-solutions like memory overlays, and
loadable modules explored aspects of VM and were based on the need for
what VM provided, but lacked the integrated, coordinated capabilities
that VM provides (i.e., to share pages between processes, coordinate
resource use, etc.)

> that tunnelling is an indication of something missing, mainly  pointing
> out that there are alternatives to virtualisation, both in  computer and
> network architecture.
>
> something that I missed in my original thinking.  [Thanks to  everybody
> that pointed it out.]  However, I also think that we are  not doing a
> very good job there.
> 
> 4. Even if we may get a clean virtualisation architecture with IPv6,  we
> are not there yet, and may not be clear where, exactly, we should  be
> going.
> 
> Details:
> 
> If we consider address mapping first, there are certainly  architectural
> alternatives.  For example, if you have a large enough  address space,
> you could use position independent code.  With proper  support at the
> processor level that can be done as convenient as  virtual-memory-based
> mapping.  The amount of hardware needed is  basically the same; the
> difference is on the abstraction.
> 
> I am not going to the protection part.  It is apparently a very  useful
> abstraction, providing lots of useful functionality such as 
> consist of several components, some of which don't _necessarily_  imply
> others.

What you've shown is that you can achieve aspects of VM with
alternatives, but not that you can replace what VM does.

Position-independent code means "make the compiler/machine aware of the
need to run anywhere in memory", which isn't the same as "provide an
environment where the compiler/machine doesn't need to know or care".
The latter is what VM provides, and PIC hasn't replaced it.

> The majority of today's hardware does not support full virtual  machine
> functionality, i.e., virtualising also the protected mode.   In the main
> stream architectures it has been appearing only  recently. 
> Consequently, many of today's virtual machine products  either use
> software to "cheat", resulting in architecturally ugly  (but useful!)
> solutions, or are fully implemented in software, with  the apparent
> performance penalty.
> 

Yes - see some of our work on the X-Bone for a discussion on what else
is needed, a more recent version of the overview of which is in:

J. Touch, Y. Wang, L. Eggert, G. Finn, "Virtual Internet Architecture,"
ISI Technical Report ISI-TR-2003-570, March 2003.
http://www.isi.edu/touch/pubs/isi-tr-2003-570/

> like the virtual address space in a virtual memory system, that  AFAICT
> doesn't quite hold.  The difference lies in how you  communicate with
> components outside of that virtual space.

1918 addresses are equivalent to a VPN-ID, except for the number of bits
in the per-overlay address space. Second, communicating between virtual
spaces is not something the basic version of virtual nets get for free -
what they need is something - akin to a gateway - that sits in both
address spaces, and translates between the two. This isn't anything like
a router, though.

> to lead to a cleaner architecture than mapping address spaces.  For  us,
> going to IPv6, Unique Local IPv6 addresses (draft-ietf-ipv6-

If you're completely ignoring the address space differences, then, going
back to VM, you're proposing PIC with a single large address space. That
doesn't allow you to support large programs in a small amount of memory
- - akin to 'revisitation' in virutal nets.

If you consider the large address space split into chunks, that's just
moving the VPN-ID outside to the front of the address and calling the
concatination one large address. That's basically what VM does
(large-memory address = page + small memory address).

> routing.  For practical reasons, it looks like that people may need  to
> use tunnelling even for Unique Local IPv6 addresses.  The reason  for
> this is that the routing system does not easily allow them to be 

The question I would place here is: where are the 'links' in your
virtual topology? Tunnels are those links, which is why you still need
them, IMO.

> question and to the larger architectural one; e.g. use MPLS, use  source
> routing, etc., but that is not the point of this message.  The  point
> here is that the fact that tunnelling is needed may, again, be  an
> indication that some functionality is missing elsewhere, in this  case
> in routing.]
> 
> 
> Certainly so, but my question is now much of this is caused by  current
> architecture, i.e., the fact that IP addresses are  simultaneously used
> as identifiers and locators.  If these two roles  were separated (most
> probably leading to two having (sub)layers  instead of the currently one
> IP layer), what would that virtual  topology bring?

I've already listed the things it brings above, and it's more than just
address space management (below).

> My claim is that if we did the split, there would be much less need  for
> "managing" the IP address space, leading to a situation where  most (if
> not all) long term needs for IP virtualisation would disappear.
> 
> architecture for IPv6, i.e, one resembling virtual machines (instead  of
> virtual memory).  That leads to a number of more practical questions:
> 
> - Who would need it?

P2P nets are a good example where forwarding matches a virtual topology.
There are challenges to supporting forwarding based on application-level
information; that info needs to be at the network layer, e.g., in flat
IDs (hashes) or IDs with other structure than what we currently use.
I.e., we need to be able to use other IDs and other forwarding
algorithms to make this work, but that too is being explored (e.g., our
own Datarouter in Proc. IWAN 2003, also in Carzaniga, A., Wolf, A. L.,
?Forwarding in a Content-Based Network,? Proc. Sigcomm 2003, Aug. 2003,
pp. 163-174.).

> Maybe all we need is a BCP explaining how to use existing components  in
> the best possible way for virtualising IPv6 in a local network,  with
> appropriate barriers between the local and outside world?
> 
> But before that I'd like to here what are the real benefits that  people
> would expect to get from it, as I am personally having hard  time seeing
> them.
> 
> --Pekka

---

From: Tony Li <[email-masked]>
Date: Tue, 09 Aug 2005 17:51:18 -0700

Pekka,

> functionality, i.e., virtualising also the protected mode.   In the
> mainstream architectures it has been appearing only  recently.

I'd like to refer you to http://en.wikipedia.org/wiki/VM/CMS.  This is
to point out that we have been delivering true virtual machine
functionality in "mainstream" architectures since 1966.  Only recently
have we chosen to reimplement it for the average user's desktop system.

You are correct, a virtual machine is much more than just virtual
memory.  In includes virtualizing everything within the computer system,
not just main memory.  If we extend the analogy to our domain, what you
are asking for is a first class virtual network, NOT just a virtual
address space.  What would a virtual network look like?  Well, the
current jargon is Virtual Private Network, but if we take the risk of
opening a VPN, you can easily imagine a virtual Internet.  This includes
virtualizing routers, links, control plane, etc.

I submit to you then, that tunneling today is simply the creation of a
virtual link, or as I said originally, tunnels create a virtual
topology.  Virtual routing is being delivered already today in a crude
form known as 2457 and in a more general form by at least two router
vendors.  Therefore, I believe that we largely already have everything
that you seek.  Subsequent refinements and improvements are of course
inevitable, but conceptually and architecturally, I would claim that we
are pretty much already there.

The last, and perhaps most difficult architectural step necessary to
complete this vision is the ability to virtualize the transport
connection away from the network instantiation.  Just as in a virtual
memory system, where the application program does not care which
physical page a particular piece of virtual address space resides in, we
need to provide an API to the network layer that the transport layer can
operate on, even while the network layer is shifting beneath it.  In a
virtual memory system, demand paging can cause an application page to
change physical addresses.  In a virtual network, we should be able to
shift the network layer completely, without interruption to the
transport layer.

Pekka, you touched on the solution and those who have been in the fray
for awhile know where this leads: the separation of the network address
into locator and identifier.  More generally, not only into a host
identifier, but into a transport connection identifier.  This would deal
with many of the issues and complexity that we see today with mobility,
multihoming, and would even provide such interesting possibilities as
migration of transport connections across hosts.

This is a battle that has been fought long and hard and has been settled
and I do not bring it up again to re-open it.  In fact, I implore all
those that have read this far to resist the urge to renew hostilities.
I only mention this topic out of completeness, not to re-ignite the debate.

> not all) long term needs for IP virtualisation would disappear.

Does it?  As Joe Touch pointed out, even with a large amount of physical
memory, there are distinct advantages to having virtual memory.   True,
you may give up demand paging, but you may still with to have an
independent uniform, and predictable address space for programming
simplicity.  These same needs drive us to create VPNs regardless of the
size of the address space.

Regards,
Tony

---

From: Pekka Savola <[email-masked]>
Date: Wed, 10 Aug 2005 08:16:26 +0300 (EEST)

(I'm using Tony's message only as a springboard..)

> size of the address space.

This may or may not be obvious, but I don't think anyone said it 
explicitly...

I think what PekkaN has looking for is the separation of using 
addresses as a "virtualization layer" to using [something else].  If 
we assume that this kind of virtualization is needed by the 
architecture, we'll either need to stick to doing the virtualization 
using addresses, or using something else (which may not exist yet, 
leading to the issue of something missing in the architecture).

What could those other means be?

Well, for example,
  - more flexible lookup mechanisms than single-faced DNS (the results 
of the lookup might depend on which virtualizations each node "belongs 
to")
  - easier security matching (in case you use a specific range of IP 
addresses to imply trustworthiness), e.g., using more advanced lookup 
methods

These are the issues that will come up more strongly with id/loc 
split.

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Aug 2005 06:21:35 -0700

Pekka Savola wrote:
> I think what PekkaN has looking for is the separation of using addresses
> as a "virtualization layer" to using [something else].  If we assume
> that this kind of virtualization is needed by the architecture, we'll
> either need to stick to doing the virtualization using addresses, or
> using something else (which may not exist yet, leading to the issue of
> something missing in the architecture).
> 
>  - more flexible lookup mechanisms than single-faced DNS (the results of
> the lookup might depend on which virtualizations each node "belongs to")

That's a part of virtual networking; as has been noted, you need more
than just virtual addresses. You also need a separate DNS that takes the
overlay as context to the node lookup, i.e., "gethostbyname(nodename,
overlayname)".

> These are the issues that will come up more strongly with id/loc split.

Even an id/loc split still ends up benefitting from virtualization.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Aug 2005 10:55:07 -0700

PS - some other details below; I had neglected to address some of the
significant concerns about separable routing tables...

Pekka Nikander wrote:
....
...
> 2. Virtual machine is more than virtual memory

In the process of developing the X-Bone architecture, we recognized the
need to keep spaces separate, to the point of needing support for
different clusters of interfaces. Each cluster needs to be assigned to a
forwarding table, where addresses could be unique within the virtual net
each cluster is attached to, but reused across clusters.

Marco Zec implemented "clonable stacks," which turned out to have the
functionality we needed for this part of the endsystem virtual network
support. This includes the notion that "inaddr-any" is meaningful only
within each cluster - i.e., within each cluster there is a "weak host
model" and between clusters there's the "strong host model" (both from
RFC1122).

There are other aspects of support needed. One includes file system
virtualization to allow multiple programs to run with different 'root
file systems'; this is key to revisitation, the use of a single physical
node more than one times in a single overlay (akin to VM paging).
Another includes the need for two-layer tunnels, to virtualize the link
layer as well as the network layer. There are others, described as noted
before in some of our groups' papers.

...
>   Should the routing system better support local-only routes?

This is, FWIW, the kind of support that's already available in Zec's
clonable stacks, and it is (IMO) required. It would also be useful to be
able to install local forwarding algorithms and entire local stacks, but
that's not currently available (though precursors have been tried, e.g.,
in Active Nets, our own DataRouter, etc.)

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Aug 2005 14:05:55 -0700

Tony Li wrote:
> vendors. 

There are a few aspects to this:

	- separate forwarding tables (some router vendors, Marco's
		"clonable stacks" for FreeBSD)
	- multiple, separate instances of routing protocol algorithms
		supported by some router vendors, and in some daemons

> Therefore, I believe that we largely already have everything

See our work on the DynaBone (www.isi.edu/dynabone) ;-) We used a single
overlay deployed on top of a 1,000 parallel alternate overlays, where
the traffic was shifted below without interuppting the transport
protocol. Just as the program in VM uses _its_ address, ignorant of the
paging underneath, our system let the transport protocol operate on the
addresses in the upper overlay, ignorant of the way in which it was
encapsulated and mapped into any of the 1,000 underlying overlays.

> into locator and identifier.

I'm not sure that's the case; it separates the ID/loc of the overlay
from the ID/loc of base underlying network. There is still the need (or
not) to separaet name from ID in each of the two layers, independent of
virtualization. The issue of name vs. location is a networking concept,
common to both real and virtual networks.

> More generally, not only into a host

DynaBone dealt with mobility and multihoming using the layering of
virtualization, and didn't need to address the ID/loc split issue to
achieve the benefits.

---

From: Pekka Nikander <[email-masked]>
Date: Thu, 18 Aug 2005 11:25:02 +0200

Joe,

I am slowly trying to go forward with my thinking re/ tunnelling and  
virtualisation, and got a detail question of something that I don't  
understand.

> http://www.isi.edu/touch/pubs/isi-tr-2003-570/

This paper talks quite a lot about revisiting virtual networks, and  
then later on you stated the following:

> [...] That doesn't allow you to support large programs in a small  
> amount of memory - - akin to 'revisitation' in virutal nets.

Could you please explain what you mean with revisitiation?  I don't  
quite understand it, even after reading your paper the second time  
with this in mind.  Especially, I don't understand how revisitation  
is kind of supporting large programs in a small amount of physical  
memory.  The other thing that I don't understand is the benefits of  
revisitation, other than for research and early deployment purposes.

--Pekka

---

From: Lars Eggert <[email-masked]>
Date: Thu, 18 Aug 2005 11:42:05 +0200

Hi,

On Aug 18, 2005, at 11:25, Pekka Nikander wrote:
> I am slowly trying to go forward with my thinking re/ tunnelling  
> and virtualisation, and got a detail question of something that I  
> don't understand. You referred to a pretty interesting tech report:
>

"Revisitation" means being able to support larger virtual topologies  
than you have physical resources for, e.g., deploying a 100-node ring  
on ten machines in a lab. This is done by hosting multiple virtual  
nodes on a single physical node. (You "revisit" a physical node  
multiple times for one overlay - the term may not be perfect.)

Many existing systems that deploy virtual networks can only deploy  
virtual nets that are at most as large as the underlying physical  
resources, i.e., they can arrange a subset of the physical nodes into  
arbitrary topologies by creating tunnels, but they cannot create  
virtual topologies that have more nodes than available physical  
machines.

Revisitation is similar to VM, in the sense that both enable the  
creation of virtual resources of apparently larger capacity than the  
underlying physical resource. This more completely decouples the two,  
increasing the power of the virtual abstraction. AFAIK the X-Bone is  
still the only virtual nets architecture that support this.

(A different aspect of this decoupling between physical and virtual  
resource would be replication of a single virtual node onto multiple  
physical machines for fault tolerance. This isn't in X-Bone yet, AFAIK.)

Lars
--
Lars Eggert                                     NEC Network Laboratories

---

From: Pekka Nikander <[email-masked]>
Date: Thu, 18 Aug 2005 11:51:20 +0200

Lars,

> "Revisitation" means being able to support larger virtual  
> topologies than you have physical resources for, e.g., deploying a  
> 100-node ring on ten machines in a lab. This is done by hosting  
> multiple virtual nodes on a single physical node. (You "revisit" a  
> physical node multiple times for one overlay - the term may not be  
> perfect.)
>
> resources, i.e., they can arrange a subset of the physical nodes  
> into arbitrary topologies by creating tunnels, but they cannot  
> create virtual topologies that have more nodes than available  
> physical machines.

So far I think I understand.

> creation of virtual resources of apparently larger capacity than  
> the underlying physical resource. This more completely decouples  
> the two, increasing the power of the virtual abstraction. AFAIK the  
> X-Bone is still the only virtual nets architecture that support this.

Hmmm.  Are you trying to say that since we have so few real IP  
addresses available, we need this kind of virtualisation to give more  
IP addresses per host?  If so, I don't understand why?  What do you  
need that many IP addresses for each host?

> resource would be replication of a single virtual node onto  
> multiple physical machines for fault tolerance. This isn't in X- 
> Bone yet, AFAIK.)

I still completely fail to see the utility value of this for anything  
else but emulation kind of work.  Or are you trying to split the two  
roles of IP addresses by creating a layer of indirection within IP,  
using virtual addresses as identifiers and physical addresses as  
locators?

--Pekka

---

From: Lars Eggert <[email-masked]>
Date: Thu, 18 Aug 2005 12:13:19 +0200

Hi,

On Aug 18, 2005, at 11:51, Pekka Nikander wrote:
>> the two, increasing the power of the virtual abstraction. AFAIK  
>> the X-Bone is still the only virtual nets architecture that  
>> support this.
>
> addresses available, we need this kind of virtualisation to give  
> more IP addresses per host?  If so, I don't understand why?  What  
> do you need that many IP addresses for each host?

no, this has nothing to do with IP addresses or IP address space. The  
motivation was to make the deployable virtual networks as independent  
from the available physical resources as possible. Without  
revisitation, virtual networks can only have less or equal the number  
of nodes than the underlying deployment system. With revisitation,  
this restriction is lifted.

To use the VM analogy again, a VM without "revisitation" - i.e.,  
paging, would provide protected virtual address spaces to processes,  
but the virtual address spaces would only ever be as large as the  
physical memory present. With paging, this restriction is lifted,  
increasing the usefulness of the VM abstraction immensely.

(Aside: While VM virtualizes address spaces, virtual networks  
virtualize networks, not the address space the network layer uses.  
I'm not sure if this is where our misunderstanding comes from;  
mentioning it just in case.)

>> (A different aspect of this decoupling between physical and  
>> virtual resource would be replication of a single virtual node  
>> onto multiple physical machines for fault tolerance. This isn't in  
>> X-Bone yet, AFAIK.)
>
> I still completely fail to see the utility value of this for  
> anything else but emulation kind of work.  Or are you trying to  
> split the two roles of IP addresses by creating a layer of  
> indirection within IP, using virtual addresses as identifiers and  
> physical addresses as locators?

The usefulness lies in removing restrictions on the kinds of virtual  
networks that are deployable. To deploy a 100-node ring on Planetlab,  
I first need to check if Planetlab has 100 nodes right now. If it  
doesn't, then I can't. With the virtual Internet architecture of the  
X-Bone, I don't need to care how many physical nodes are available to  
host my virtual network. Similar to how a paging VM supports larger  
virtual memory spaces than available physical memory.

We've used this capability for fault tolerance purposes, e.g.,  
transparently remapping virtual hosts between physical hosts during  
the lifetime of a virtual network.

Lars

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Aug 2005 13:27:31 -0700

Pekka,

Sorry for not responding directly, but Lars' responses are doing fine
(so far ;-)), so I'll let him continue unless there's a need to jump in...

Joe

Pekka Nikander wrote:
> quite understand it, even after reading your paper the second time  with
> this in mind.  Especially, I don't understand how revisitation  is kind
> of supporting large programs in a small amount of physical  memory.  The
> other thing that I don't understand is the benefits of  revisitation,
> other than for research and early deployment purposes.
> 
> --Pekka
> 
>