Subject: Re: Session Stability (was multi-homing vs route aggregation)

From: Jessica Yu <[email-masked]>
Date: Mon, 02 Aug 1999 13:00:45 -0400

Hi,

I have been reading this thread. Being an engineer and network architecture 
of large scaled networks (UUNET and ANSNET and way earlier NSFNET), I feel 
strongly that any solution we come up with here should also be operationally 
MANAGEABLE, that is, it scales operationally. Below is a proposal for 
thoughts and consideration.

							--Jessica

-----------------------------------------------------------------------   

Internet Engineering Task Force                 Jieyun (Jessica) Yu
INTERNET DRAFT                                                UUNET
Expires February, 2000                                 August, 1999

                    A Simple Solution for IPv6 Multihoming

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that other
   groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   Distribution of this memo is unlimited.

   Copyright Notice

   Copyright (C) The Internet Society (1999).  All Rights Reserved.

Abstract

   There is a challenge of providing IPv6 multihoming sites with the
   basic benefits such as redundancy and load sharing, and at the same
   time, scaling the global IPv6 routing table. It is equally
   challenging to provide a solution which is simple enough to be
   operational manageable. This document proposes a simple solution
   which meets the requirements and is also operationally manageable.

1. Purpose

   There is a challenge of providing IPv6 multihoming sites with the
   basic benefits such as redundancy and load sharing, and at the same
   time, scaling the global IPv6 routing table. It is equally

Yu              A Simple Solution for IPv6 Multihoming          [Page 1]

Internet Draft   draft-yu-simple-ipv6multihoming-00.txt      August 1999

   challenging to provide a solution which is simple enough to be
   operational manageable. This document proposes a routing and
   addressing scheme that supports IPv6 multihoming which achieves the
   followings:

      a. Providing redundancy and load sharing for the multi-homed sites
      b. Scaling the global IPv6 Internet routing table 
      c. Simple and operationally manageable

2. Proposal

			   ISP-3 ---- ISP-4
                             |      /  |                              
			     |    /    |                           
			     |  /      |
                           ISP-1 ---- ISP-2
   	            link-1   \          /  link-2
                              Customer-A

   A multi-homed customer will designate a primary ISP and receive IP
   address from its primary ISP's IPv6 aggregation block. In this
   example, ISP-1 is the primary ISP of customer-A and assigns Addr-1-A
   to the customer. Customer-A will advertise addr-1-A to ISP-1 and
   ISP-2 respectively. ISP-2 will advertise Addr-1-A to ISP-1 and to
   ISP-1 only. ISP-1 will, of course, advertise its own aggregation
   Addr-1 to the entire Internet.

   For inbound traffic to Customer-A, traffic will first be forwarded to
   ISP-1 since it advertises Addr-1 which includes Addr-1-A. ISP-1 will
   then forward the traffic destined to Customer-A via its connection to
   ISP-2 and/or its direct link to customer-A, according to certain
   polices. Most common policy would be the use of the shortest exit. By
   using both connections to forward traffic to Customer-A, load sharing
   is achieved.

   For outbound traffic from Customer-A, ISP-1 and ISP-2 can announce
   default and/or specific prefixes based on customer-A's requirements
   and traffic originated from Customer-A will be forwarded accordingly.

   When the link between customer-A and ISP-2 (link-2 in the figure)
   goes down, all traffic will go in/out via the connection between
   Customer-A and ISP-1 (link-1 in the figure). Likewise, when link-1 is
   experiencing an outage, link-2 will be used for the traffic.  This is
   because ISP-1 will continue announce its aggregate block (i.e. Addr-
   1) to the entire Internet and ISP-2 will still advertise Addr-1-A to
   ISP-1, all the inbound traffic to the customer will be take the path:
   ISP-1 -> ISP-2 -> Customer-A. Outbound traffic will automatically
   fall to link-2. This way, redundancy is achieved.

Yu              A Simple Solution for IPv6 Multihoming          [Page 2]

Internet Draft   draft-yu-simple-ipv6multihoming-00.txt      August 1999

   Same mechanism can be extended to those sites multihoming to more
   than two ISPs. Only those ISPs that the customer directly connected
   to would carry the more specific prefix assigned to the multi-homed
   customer.

   The more specific prefix announcement could be covered under the
   peering agreement between ISPs as stated in [RFC2546].

3. Advantages of the proposal

      - Achieves the goal of scaling the global routing table.

      - It scales the global routing table. Only ISPs involved in the
      multihoming attachment need to know the more specific route of
      Addr-1. Other ISPs such as ISP-3 and ISP-4 in figure 1 do not need
      to know specifics.

      If the two involved ISPs has no direct connection, the more
      specific route would need to be carried by other ISPs in the path.
      This seems to be lesser a problem since ISP assigned with an
      Internet visible aggregate block usually has direct connection to
      each other.

      - It is really simple and thus more manageable and scalable
      operation wise. It does not require multiple addresses assigned
      all individual hosts of a multi-homed customer site, neither it
      requires O(N^2) tunnels among different ISP routers.

4. Concerns

      - The primary ISP of a multihoming customer (such as ISP-1 in the
      example) would need to do more work in terms of distributing the
      traffic among other connected ISPs and the customer link.  This
      can be done using BGP policy (BGP community) and using shortest
      exit. It will also carry more traffic for the customer. However,
      this is a value added service to the customer and the primary ISP
      can charge more fees for such services.

      - The primary ISP is the sole interface for the multihomed
      customer to the Internet other than the ISPs the customer has
      directly connection with. Outages such as one between ISP-1 and
      ISP-4 would impact the reachability from customers of ISP-4 to
      Customer-A even though ISP-2 still has good connection to ISP-4.
      However, if the primary ISP is a good quality ISP, this sort of
      situation should happen rarely. The reason is that it's common
      practice that an ISP, especially good quality ISP, has multiple
      connections to other big ISPs at different geographical locations.
      Good quality ISPs also have robust network design to prevent any

Yu              A Simple Solution for IPv6 Multihoming          [Page 3]

Internet Draft   draft-yu-simple-ipv6multihoming-00.txt      August 1999

      failure to impact the whole network. Choosing a good quality and
      robust ISP as primary ISP is a good practice.

   It is desirable to have solution which provides perfect redundancy
   and, at the same time, simplicity to scale management and operation.
   In the absence of a perfect solution, the trade-off needs to be made.
   The author believes that it is very important that the solution has
   to be simple and manageable. Manageability should be the top
   consideration.

5. Security Considerations

   Security considerations are out of scope of this document.

6. References

   [RFC2546] A. Durand and B. Buclin, "6Bone Routing Practice."
   RFC2546, March 1999.ftp://ftp.isi.edu/in-notes/rfc2546.txt.

Author's Address

   Jieyun (Jessica) Yu
   UUNET Technology
   880 Technology Dr.
   Ann Arbor, MI 48108

   Phone: (734) 214-7314

   EMail: [email-masked]

Yu              A Simple Solution for IPv6 Multihoming          [Page 4]

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: "Matt Crawford" <[email-masked]>
Date: Mon, 02 Aug 1999 13:44:59 -0500

> thoughts and consideration.

Jessica,

Your draft is so succinct and clear that this demi-routing-clued
reader can absorb it in minutes.

Consider the situation after your model has been "symmetrized" --
that is, when both ISP-1 and ISP-2 play the role of primary ISP for
Customer-A and assign it address space, and Customer-A announces two
specific routes to each ISP, and each ISP then announces one of them
to the other.  I claim that this is only epsilon more complex than
your model, since the same entities are communicating with each other
at the same points and just announcing on additional prefix.  Yet it
solves one of your "concerns" items without damaging anything in the
"advantages" section.

I believe this symmetrized version is what the community is
visualizing.  It adds complexity in the hosts compared to yours, but
most of that complexity has to be there anyway to support features
like mobility, renumbering and site-local addressing.  From the
viewpoint of ISP operations and global routing, do you see any
drawbacks?
				Matt
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 02 Aug 1999 15:06:04 -0500

I think I agree with Matt.

Another point: a *major* requirement is that IPv6 must allow for
smooth prefix updates (a.k.a. renumbering). Matt's version seems
to make that easy, i.e, at some point in the process of updating from
Prefix 1 to Prefix 2, Prefix 1 becomes deprecated and slowly
fades away.

Another point is that none of this gets rid of the need for
source and destination address selection mechanisms. Multiple
prefixes are part of life with IPv6.

  Brian

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Brian E Carpenter (IAB Chair)
Program Director, Internet Standards & Technology, IBM Internet Div
On assignment for IBM at http://www.iCAIR.org 
Non-IBM email: [email-masked]
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Jessica Yu <[email-masked]>
Date: Tue, 03 Aug 1999 11:10:38 -0400

> reader can absorb it in minutes.

Thanks!
> viewpoint of ISP operations and global routing, do you see any

I think routing wise, the proposal you outlined above would work. But
I am really concerned about the requirement of multi-address per host,
especially N (N>2) addresses per host is required if the site multihoming
to N ISPs.  

I think it'd lead confusion in routing in the site and host management in 
general.  The software of hosts and routers has to be more sophiscated.  
I personally tried multi-address (two addresses) on interfaces of site 
border routers to get around a problem in one of the network we built.  That 
was not a good experience.  The vendor just could not get the software work 
right. One major issue was it only allow the primary interface address not 
the secondary interface to pass dynamic routing information.  It took the 
vendor a long time to fix this problem (by the way, the vendor is 
consider the lead in routing software).  This was done just on a couple of 
routers, image to have hundreads and thousands of hosts with multiple 
addresses on each.

Anyway, I suggest that before we jump into the multi-addressing scheme, 
let's throughly evaluate the complication and operation complexity 
associated with that.  

Thanks!

							--Jessica

---

From: Francis Dupont <[email-masked]>
Date: Tue, 03 Aug 1999 21:21:09 +0200

In your previous mail you wrote:

   visualizing.

=> this symmetrized version is very close to the "mutual backup"
(draft-ietf-ngtrans-6bone-multi-01.txt section 5).

If I take again Jessica's diagram:

                   ISP-1 ---- ISP-2
            link-1   \          /  link-2
                      Customer-A

The idea is simple, both ISP-1 and ISP-2 announce both ISP-1 and ISP-2
prefixes. With the standard BGP way to prefer shortest AS paths ISP-1
will be prefered for ISP-1 prefix routing and ISP-2 for ISP-2 prefix.
This has wanted properties (good aggregation, insensitive to any single
link/router/ISP failure, compatible with BGP default rule for route
selection, ...). I know only two issues: this suppose a reciprocal
agreement between ISP-1 and ISP-2, and this doesn't scale well with
many ISPs (this is not transitive).

One can easily make use of this, the only needed new thing is a link
between ISP-1 and ISP-2 (not through the customer). In the G6 we have
organized the G6 network (G6-bone) with both geographical and
organization divisions, for instance INRIA Rocquencourt is near Paris
and part of the INRIA organization, in order to make use of this we
only need a link between Paris main point and INRIA main point:

                          G6                   /24
                    /           \
                   Paris ---- INRIA            /32
            link-1   \          /  link-2
                   INRIA near Paris            /48

[email-masked]

---

From: Francis Dupont <[email-masked]>
Date: Tue, 03 Aug 1999 21:39:02 +0200

But I am really concerned about the requirement of multi-address per host,
   especially N (N>2) addresses per host is required if the site multihoming
   to N ISPs.  
   
=> RFC 2260 (again) explains in its section 6 why a multi-homed site
should have multiple prefixes (then multi-address per host).
The text is about IPv4 with CIDR and the aggregation constraints of
IPv6 are far stronger then this requirement will remain...
 In fact with IPv6 any interfaces should get two or more addresses,
even on a mono-homed site, because of scoped addresses.
Multi-address per host is an IPv6 fact and there are many available
or proposed tools (like autoconfigurations, A6 and router renumbering)
which should help us.

=> I agree, the common multi-address support shows its limits (or its bugs)
but it can (should!) be fixed (even by leading vendors in routing software :-).

=> multihoming just makes it worse but this issue is there from the
beginning of IPv6 (then it should be no more a real issue?)

Thanks

[email-masked]

PS: Web "virtual-hosting" is another reason to support multi-addressing
and one can put hundreds of addresses (=> performance issue too).
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Thomas Narten <[email-masked]>
Date: Wed, 04 Aug 1999 07:28:57 -0400

> and one can put hundreds of addresses (=> performance issue too).

In the web world, this is apparently no longer a compelling argument
for multiple addresses. This was an issue in older versions of HTTP
(1.0). HTTP 1.1 fixed this problem, and indeed, extensions to 1.0 that
have been widely deployed fixed this as well. My understanding is that
this is no longer an issue today.

I.e., today, HTTP queries sent to the server include the DNS name in
the request (older versions included only the path to the right of the
DNS name). Thus, one can have multiple DNS names map to the same
address and everything works as intended.

Note also that for virtual hosting, the prefix stays the same, only
the interface ID part of an address needs to change. This is not
multi-homing per se and doesn't impact route aggregation.

Thomas

---

From: "Andrew Macpherson" <[email-masked]>
Date: Wed, 04 Aug 1999 13:20:12 +0100

Mostly correct.

One still needs "traditional multi-homing" for Secure Servers

---

From: Francis Dupont <[email-masked]>
Date: Wed, 04 Aug 1999 14:29:23 +0200

In the web world, this is apparently no longer a compelling argument
   for multiple addresses...

=> yes, new HTTP has been fixed (then change "is" in "has been another ...")

=> it has been only another reason... I don't think to have the same
prefix makes things simpler too.

I have added the post scriptum because in BSD kernels interface address
code was improved only with an architectural bug fix and the replacement
in some flavors of the linked list of addresses by a hash table.
Motives were for the first point to get rid of an old bad choice and
for the second point performance problem with "virtual-hosting".

I think we can say that modern Unixes support multi-address without
problems (ie without bugs), can't we?

Regards

[email-masked]

PS: about multi-addresses I don't know how this works with DHCPv6
(I asked at the last IETF meeting then I expect to get an answer).
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Jim Bound <[email-masked]>
Date: Thu, 05 Aug 1999 09:31:01 -0400

Thomas,

>this is no longer an issue today.

Your missing the point I think.  The market does not have to give us in
the standards arena compelling arguments.  IPv6 supports multiple
addresses for Interfaces.  How they want to use that is their choice and
I can think of two reasons why a server would want to support 3
addresses on one interface on a link.  

If you are hinting you would like to change what the Ipv6 architecture
permits then just say so and get it on the table!

>address and everything works as intended.

Fine but we cannot force all to do this in this manner, its a choice
HTTP 1.1 provides not a mandate.  There is also an issue with Firewalls
and Ingress filtering which want real IP addresses used.

/jim

---

From: Jim Bound <[email-masked]>
Date: Thu, 05 Aug 1999 09:35:19 -0400

Jessica,

>thoughts and consideration.

Nice to see proposals occasionaly of late from folks whose job code is
"engineer".

I think your proposal is excellent, elegant, and makes good engineering
design tradeoffs for the network.  Consider me a proponent of this and I
have seen no mail that counters your idea  or proposal.  This will work
and maintain aggregation.

Good Job,
/jim

---

From: Jim Bound <[email-masked]>
Date: Thu, 05 Aug 1999 09:40:13 -0400

Hi Brian,

>prefixes are part of life with IPv6.

Your pushing this now very hard and in 6to4 and in mail.  I am going to
start its own mail thread so we can discuss what it means and
ramifications.  I want this discussed in depth and by the community it
needs rigorous debate and discussion as it affects Hosts and Routers
greatly. 

/jim

---

From: Peter Tattam <[email-masked]>
Date: Fri, 6 Aug 1999 00:48:52 +1000 (EST)

> needs rigorous debate and discussion as it affects Hosts and Routers

Ok.  I'll make a few assertions about a "typical" environment, and tell a story
or two.  In Aus & UK, there is a TV program called "Hypothetical"...  imagine
this...  cameras zooming in on the presenter...

While most implementations are conceding that multiple addresses should be
implementable, and indeed they are, *how* do I choose a source address that
will give me the best performance.  

In a typical working environment, I have one interface.

It has many addresses.  (my site uses provider A, B & C)

There are no hints from the network as to which address I source my packets
with because I'm not allowed to participate in any routing protocols.

The only time I get hints are when part of my network is down in that it tells
me that one particular address should not be used, but under normal conditions
the only way I can tell which network is the best at any particular time is to
try pinging with a source address from either of the providers.

I've only got one router visible to me, hence I only ever configure one default
route.

The router advertisement always gives me the the list of addresses in roughly
the same order but sometimes the implementation alters the order somewhat,
placing the slowest link at the start of the list.

One of the addresses sometimes gives me much poorer response time because it's
really only a secondary link that my provider uses as a backup, and that
provider has staffing problems...  occasionally their frame relay network goes
a bit hazy and starts routing packets via Hong Kong.  They often don't get to
it for several hours, usually when one of my staff finally figures that it's
not regular congestion and there really is a problem & rings them.

One day the network is so unstable that any one or more of my addresses doesn't
respond or goes up & down like a yoyo.  (In real life, someone put a back hoe
through the fibre optic cable, and it's the main line out of town.  Two of my
providers use this cable physically and the other doesn't).

BTW, providers A & B are in a legal battle and have as a result no peering
arrangements any more.  Provider C doesn't care and has peerings with both, but
then provider A threatens to block C's traffic if it doesn't drop peering with
B.  C obliges, but then B gets annoyed.  Peerings are opened and shut at the
rate of 1 per week until C eventually gives up after a month and doesn't bother
peering any more.  Provider A also has a relationship with the optic cable
company and could make things nasty for the other two if it really wanted to.

Meanwhile, the fibre has been fixed, but is still a litte flakey.  As a result,
C has picked up a lot of traffic from A & B's customers because they also have
a satellite link and this saves the day for them.  A couple of month's later,
the fibre is now stable and traffic is back to normal.  A & B have settled
their dispute, but C is now locked out of the peering arrangement because they
stole many of A & B's customers.   At around the same time, there is a solar
storm which knocks out quite a number of channels of the main satellite service
that C uses. Because of all the extra customers, he is now *really* stretched
to the limit & his service is now *really* bad.

My boss now shows up...  the machine that was configured and working last week
is not working very well, but his secretary's machine is going just dandy. They
are trying to send a large file through to India which is worth a very
large contract.  It keeps failing because the network is still unstable with
provider A & B keep dropping in & out alternatively.  The attachment would
normally take 3 hours to send to India because the client at the other end is
operating in a less than optimal location in the phone system.  This is the
third time they have tried to send it.  The first time it got through till 90%,
the second time 40% and the last time 50%.  Unfortunately as a test of concetp,
we are an "IPv6 only" shop, we were so sure that those larger addresses would
solve everything.

He plonks on my table the detailed recommendation that I gave him 6 months ago
that we go "IPv6" because it is a superior system.  I leave the room wondering
how I'm going to keep up the house payments now that I have been demoted to
tech support officer grade 1.

Under this scenario, which system will survive the best, IPv4 or IPv6.

Peter

--
Peter R. Tattam                            [email-masked]
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

---

From: Thomas Narten <[email-masked]>
Date: Thu, 05 Aug 1999 10:52:47 -0400

Jim,

I believe multiple addresses per interface are a fact of life (i.e.,
scoping). In addition, I think they offer potential to help with route
scaling (i.e., multiple address per ISP connection). I was simply
pointing out that in the specific case of web servers (where virtual
hosting is very popular today), one doesn't need multiple addresses to
do this. This is already established practice through HTTP
improvements.

> and Ingress filtering which want real IP addresses used.

The market has apparently already spoken on this issue. The mods are
deployed today.

Thomas

---

From: Jim Bound <[email-masked]>
Date: Thu, 05 Aug 1999 12:24:59 -0400

OK..Tom... I know this has come up in other forums and just checking..

thanks
/jim