Subject: Re: Session Stability (was multi-homing vs route aggregation)

From: "Steven M. Bellovin" <[email-masked]>
Date: Mon, 26 Jul 1999 11:19:23 -0400

See http://www.chem.ucla.edu/~beichuan/etcp/ for details, including
Huitema's draft.

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Lixia Zhang <[email-masked]>
Date: Mon, 26 Jul 1999 08:38:35 -0700

I'm sorry but that is an outdated draft.  We have revised the spec and
implementation since then.
We'll try to put out an updated spec soon.

Lixia

---

From: Jim Bound <[email-masked]>
Date: Mon, 26 Jul 1999 11:56:15 -0400

Lixia,

>We'll try to put out an updated spec soon.

Does it still only support UDP?

thanks
/jim

---

From: Allison Mankin <[email-masked]>
Date: Mon, 26 Jul 1999 12:07:12 EDT

> Does it still only support UDP?

Jim,

Do you mean does it still only support TCP?  UDP reattachment
is not covered.  That was one of the reasons the original draft
was not pursued in the IETF Transport area, that it had limited
applicability.

---

From: Jim Bound <[email-masked]>
Date: Mon, 26 Jul 1999 12:09:42 -0400

Hi Lixia,

>Does it still only support UDP?

Whoops my UDP side of brain is dominant.  mean't.

Does it still only support TCP?

p.s. thanks Steve B. for catching that..........

thanks
/jim

---

From: Jim Bound <[email-masked]>
Date: Mon, 26 Jul 1999 12:24:57 -0400

Hi Allison,

Yep.  OK.  I did not know that about the Transport Area either.
I sent a correction to the list.

Do you think its worth putting a draft together to attempt to intrigue
developers at this time?  I could ressurect my old draft as we have
learned much more about this problem since then and why it may be
important now.

I don't think it has anything to do with the multihome issue though?

Only renumbering?

thanks
/jim

---

From: Lixia Zhang <[email-masked]>
Date: Mon, 26 Jul 1999 09:31:48 -0700

>>Does it still only support UDP?

I wondered that too when seeing your first msg :-)

>Does it still only support TCP?

At this time, Yes.

A little expansion on that yes:
- We had wanted to get this (UDP) part done but it's just one of the
  (many) term projects that had no support for continuition.
- in principle there is no reason why the same code cannot be used 
  for UDP
- in detail, however, doing the same for UDP may require more changes
  to existing implementation.  (One needs to maintain a list of possible
  addresses and info about their usability.  TCP has this notion of
  "connection" which is the entity to attach this addr info to, while
  UDP has no equivalent entity ...)

Lixia

---

From: Shane Kerr <[email-masked]>
Date: Mon, 26 Jul 1999 12:44:35 -0400 (EDT)

> I don't think it has anything to do with the multihome issue though? 

My understanding is that the current proposal for multihoming involves
each host/router/thingy on a multihomed network assuming an IP address for
each published network.  A problem with this is that if the upstream
provider dies, then any sessions will have a different IP address -
effectively renumbering any connections active at that time. 

--
Shane Kerr <[email-masked]>
Software Engineer
American Registry for Internet Numbers

---

From: Lixia Zhang <[email-masked]>
Date: Mon, 26 Jul 1999 09:48:07 -0700

>I don't think it has anything to do with the multihome issue though?

Well, other than some simple changes to TCP, the core of the implementation
is on how to obtain and maintain the list of the addresses for the remote
end.  I believe this piece of functionality would be same/similar to what
needed in multihoming cases (but I must say that I have not followed the
multi-homing thread discussion---meant to but havent had time.  I'm sitting
in this DARPA/NSF "smart environment workshop" for the day which provides
great net connectivities so I managed to read the email of the day).

Lixia

---

From: Jim Bound <[email-masked]>
Date: Mon, 26 Jul 1999 12:55:24 -0400

Hi Lixia,

>  UDP has no equivalent entity ...)

It depends on the approach taken which is why as you may recall I
proposed it be altered at the network layer with an Ext header (the
Mobile Binding would work for example), before the code executes
connection responses or initiators.  I suggested actually updating the
data structures from where the connection code gets their information.
One implementation problem with that is flushing out any cache an
implementation may be using for performance reasons.  But with the new
DST options we can also now poke the network layer to look for the Ext
hdr.  The other problem my proposal had was TCP RST, but I was trying to
propose using anycast addresses for Hosts at the same time which I would
not suggest attempting again. That problem I now realize now is
a separate problem and only indirectly releated to dynamically changing
Transport endpoints.  ICMPv4 or ICMPv6 issues are fixed using the
approach of updating the state of a connection.  Notifying the APIs is
pretty trivial too we just need to add new notification codes, but a
problem for existing applications, so they may not know whats up. 
I think IPsec will work for the security too and would have to be a MUST
for this to be used.

But if we can move this work to support UDP too and get a new spec out
thats great.  But we may want to check out different architectural
issues with solving this problem.  Besides Christian is one of our
brightest folks and I am sure the idea will work, I just think we need
to make sure UDP is not forgotten.`

Do you think this is ready for IETF work or should be done via IRTF
work?  I sent some mail to the list on this the other day.

thanks
/jim

---

From: Jim Bound <[email-masked]>
Date: Mon, 26 Jul 1999 13:23:10 -0400

Hi Shane,

>effectively renumbering any connections active at that time. 

First I would like to see the proposal.  Anyone know when it will be
sent out?  Seems like this is a hot one.  Will get lots of WG review
too.  Who are the authors?  Are the slides available from Oslo?

I don't think we can assume because an upstream providers die that
causes a renumber operation.  That is simply overkill.  If the upstream
provider dies and goes bankrupt and the doors get closed then yes that
means they are really gone.  But I believe that fail safe clauses with
most ISPs will prevent that from happening, I suggest we not address
such an extreme case till we solve the easy ones.  

The problem is if a blackout happens at an upstream provider 
will ones routing policy be agile enough to re-route the packets to
another provider.  Steve Deering gave a good list of ways to solve this
problem and I can't add to it.  I do think RFC 2260 tunneling feature is
a nice approach for the short term, but that requires potentially two
extra lines for the tunnels, one for each provider.

But I would not support for Legal, Moral, Performance, and Transport
implementation behavior reasons (and I have lots more) an upstream
provider altering the endpoints of an IPv6 header to do multihoming.
Nor do I believe this is necessary.

If Node Y is talking to Node X on the internet across provider A with a
prefix that is aggregatable by Provider A within their routing realm.
And Provide A goes down but then you want to use Provider B to get to
Node X from Node Y the answer should not be to change the src address of
Node Y.  For me that is pressing the button that says WRONG ANSWER.

But lets see what the new Multihome spec says and use that as a rallying
point to discuss this issue and get all our ideas on the table,
including the one I dislike and we can debate and defend our positions
as we work to consensus.

Renumbering the other hand has an old story on this mail list.  We
support essentially the kick off of renumbering within IPv6 parts. The
protocol parts can be done gracefully but if implementations hide
addresses in static places then there is nothing we can do with protocol
architecture to avoid such ill behaved artifacts.  Also there are mgmt
interfaces needed to manage any renumbering operation.  But the story
was even if IPv6 gets that right.  Users have the right to use Telnet in
IPv6 as a screen saver so to speak.  And Telnet should not break because
a site has been renumbered.  Personally I disagree and know of no
business application that runs this way that cannot be brought down and
restarted for something as serious as renumbering a site.  Its fantasy
to believe renumbering can ever be made in the next 10 years with any
protocol to just be done on a whim.  That is just silly.

But there are very valid reasons to want to dynamically renumber a
transport connection from a networking perspective like Host Clusters
and Failover supporting IPv6 anycast addresses, Embeded Systems (like
Appliances and Celluar phones and mobile nodes) wanting to change their
address on the fly for dynamic variant reasons across a network or
spectrum of bandwidth.  These IMO are real computer networking science
problems we should be working on like I want my connection now to go
here because my VM is bigger at this node.  So my support of the good
work Christian and Lixia and UCLA is doing is not for Internet
renumbering but for the reasons I just stated.

So I think Multihoming and Renumbering are two different discussions and
Renumbering a site is not a sole reason to go off and solve the problem
of altering transport endpoints on the fly.

Hope this helps clear up where I am coming from anyway,
/jim

thanks
/jim

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 27 Jul 1999 11:54:27 -0500

> effectively renumbering any connections active at that time.

The proposal is to use the RFC 2260 technique to recover
from outages; i.e. active connections get rerouted, not renumbered.

Please give the proponents a little more time to write it up. It's
vacation season after all.

   Brian

  Brian

---

From: Joe Abley <[email-masked]>
Date: Wed, 28 Jul 1999 12:02:16 +1200

Hi Brian,

> vacation season after all.

When you say "the" proposal, what exactly do you mean?

I understood that there were several proposals on the table,
including:

 +  pure aggregation model, no TCP session stability in the face of
    multi-homed upstream failure, algorithm required for TCP endpoint
    address selection

 +  RFC2260 model, algorithm required for TCP endpoint address selection,
    possible work required on individual platforms to allow the triggered
    "fill-in" subnet advertisements

 +  IPv4 model, sacrifice complexity in the core for the sake of
    simplicity at the edge, and trust other techniques to manage the
    routing avalanche (like MPLS)

 +  NAT/other transit address modification to provide some degree of
    address transparency to upstream networks when in the shadow of
    a transit provider failure

... and so on, and so on. Has a decision been taken to persue the
RFC2260 model in preference to others?

Incidentally, are the Oslo minutes published yet? I can't find anything
more recent than Minneapolis (maybe I'm looking in the wrong place).
This is not a loaded question :) I'm just interested in reading the
minutes when they're available.

Regards,

Joe

---

From: [email-masked]
Date: Wed, 28 Jul 1999 09:37:28 +0900

>    "fill-in" subnet advertisements

	RFC2260 does not say so.  RFC2260 (section 5.2) tries to let two
	ISP links back up each other, by having extra connection
	(usually tunnel) between ISP-A router and site router for ISP-B
	(and vice versa).
	End node need not to do anything special.
	This does not require extra (non-aggregatable) routing announcement.

	Renumbering (when one of your address goes away) is separate from
	this story.

itojun

---

From: Jim Bound <[email-masked]>
Date: Tue, 27 Jul 1999 20:45:58 -0400

Hi Brian,

>from outages; i.e. active connections get rerouted, not renumbered.

Glad to hear that.

>vacation season after all.

Sure.  

My issue here is one of the concerns Steve Deering presented is when the
mailing list members in their eagerness to work on IPv6 and get stuff
done quickly take a thread: 1) that is not defined well, 2) solutions
are rumors and no drafts, and 3) people think we have not done something
we should have for IPv6 - folks start proposing solutions to a problem
that is not defined well and my hot button may say we (the designers and
workers initially) screwed up with the Addr Arch or Addr Conf or
whatever and IPv6 is in trouble now .---)...  I have to chime in as I
can't just let that go on the list.  I have been guilty of #1 and #2 too 
but I try hard to not be of late.  In fact I am to the point if the
problem is not defined I am going start acting like Tom Hanks in "Big"
(movie about boy who gets mans body but still is boy) and says to the
marketing and engineering persons building products in a meeting "I
don't get it", no kid will ever buy that stuff!!!!!!

/jim

   Brian

  Brian

---

From: Peter Tattam <[email-masked]>
Date: Wed, 28 Jul 1999 10:46:36 +1000 (EST)

> 	Renumbering (when one of your address goes away) is separate from

There is another minor but important related issue and that is of suboptimal
routing for particular prefixes.

Will this be addressed?

> itojun

Peter

--
Peter R. Tattam                            [email-masked]
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

---

From: [email-masked]
Date: Wed, 28 Jul 1999 09:50:27 +0900

>routing for particular prefixes.

	No I do not think so.  Source address selection is a separate question
	from RFC2260.

	Here comes a draft that describes how RFC2260 would work with IPv6
	(I'll submit this very soon to the correct path).
	Please be sure to read this with RFC2260.

itojun

Internet Engineering Task Force                        Jun-ichiro Hagino
INTERNET-DRAFT                                         Research Lab, IIJ
Expires: January 23, 2000                                  July 23, 1999

             IPv6 multihoming support at site exit routers
                     draft-itojun-ipv6-2260-00.txt

Status of this Memo

This document is an Internet-Draft.

Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups.  Note that other groups
may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time.  It is inappropriate to use Internet-Drafts as reference material
or to cite them other than as ``work in progress.''

To view the list Internet-Draft Shadow Directories, see
http://www.ietf.org/shadow.html.

Distribution of this memo is unlimited.

The internet-draft will expire in 6 months.  The date of expiration will
be January 23, 2000.

Abstract

The document describes operational requirements and mechanisms for basic
IPv6 multihoming support.  The mechanism in this document can be
combined with more sophisticated (or complex) multihoming support
mechanisms, and it will be able to serve as foundation for others.  The
document is largely based on RFC2260 [Bates, 1998] by Tony Bates.

This document is an Internet-Draft and is in full conformance with all
provisions of Section 10 of RFC2026.

1.  Problem

IPv6 specifications try to decrease the number of backbone routes, to
cope with possible memory overflow problem in the backbone routers.  To
achieve this, the IPv6 addressing architecture [Hinden, 1998] only
allows the use of aggregatable addresses.  Also, IPv6 network
administration rules [Durand, 1999] do not allow non-aggregatable
routing announcements to the backbone.

Hagino                  Expires: January 23, 2000               [Page 1]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

In IPv4, a multihomed site uses either of the following technique to
achieve better reachability:

o  Obtain a portable IPv4 address prefix, and announce it from multiple
   upstream providers.

o  Obtain single IPv4 address prefix from ISP A, and announce it from
   multiple upstream providers the site is connected to.

The above two methodologies are not available in IPv6, but on the other
hand IPv6 sites and hosts may obtain multiple simultaneous address
prefixes to achieve the same result.

The document provides a way to configure site exit routers and ISP
routers, so that the site can achieve better reachability from
multihoming connectivity, without violating IPv6 rules.  The technique
uses already-defined routing protocol (BGP or RIPng), and tunnelling of
IPv6 packets, and introduces no new protocol standard.

The document is largely based on RFC2260 [Bates, 1998] by Tony Bates.

2.  Goals and non-goals

The goal of this document is to achieve better packet delivery from a
site to the outside, or from the outside to the site, even when some of
site exit link is down.

Non goals are:

o  Choose the "best" exit link as possible (How do you measure "best"
   exit link anyway?).

o  Achieve load-balancing between multiple exit links.

3.  Basic mechanisms

We use technique described in RFC2260 section 5.2 onto our
configuration.  To summarize, for IPv4-only networks, RFC2260 says that:

o  We assume that our site is connected to 2 ISPs, ISP-A and ISP-B.

o  We are assigned IP address prefix, Pref-A and Pref-B, from ISP-A and
   ISP-B respectively.  Hosts near ISP-A will get an address from Pref-
   A, and vice versa.

o  In the site, we locally exchanage routes for Pref-A and Pref-B, so
   that hosts in the site can communicate with each other without using
   external link.

Hagino                  Expires: January 23, 2000               [Page 2]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

o  ISP-A and our site is connected by ``primary link'' between ISP
   router ISP-BR-A and our router E-BR-A.  ISP B and our site is
   connected by primary link between ISP router ISP-BR-B and our router
   E-BR-B.

        (ISP A)                         (ISP B)

        ISP-BR-A                       ISP-BR-B
            |                             |
            |Primary link                 |
            |                             |
            |                             |
        +---|-----------------------------|--+
        | E-BR-A                      E-BR-B |
        |                                    |
        | Pref-A     <---------->     Pref-B |
        +------------------------------------+

o  Establish a secondary link, between ISP-BR-A and E-BR-B, and ISP-BR-B
   and E-BR-A, respectively.  Secondary link usually is IP-over-IP
   tunnel.  It is important to have secondary link on top of different
   medium than primary link, so that one of them survives link failure.
   For example, secondary link between ISP-BR-A and E-BR-B should go
   through different medium than primary link between ISP-BR-A and E-BR-
   A.  If secondary link is an IPv4-over-IPv4 tunnel, tunnel endpoint at
   E-BR-A needs to be an address in Pref-A, not in Pref-B (tunnelled
   packet needs to travel from ISP-BR-B to E-BR-A, over the primary link
   between ISP-BR-A and E-BR-A).

        (ISP A)                         (ISP B)

        ISP-BR-A                       ISP-BR-B
            | |                         | |
            | \-----------------------+ | |
            |     Secondary link      | | |
            |  +----------------------|-/ |
            |  |                      |   |
            |  |                      |   |
            |  |                      |   |
            |  |                      |   |
        +---|--|----------------------|---|--+
        | E-BR-A                      E-BR-B |
        |                                    |
        |                                    |
        +------------------------------------+

o  For inbound packets, E-BR-A will advertise (1) Pref-A toward ISP-BR-A
   with strong preference over primary link, and (2) Pref-B toward ISP-
   BR-B with weak preference over secondary link.  Similarly, E-BR-B
   will advertise (1) Pref-B toward ISP-BR-B with strong preference over

Hagino                  Expires: January 23, 2000               [Page 3]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

   primary link, and (2) Pref-A toward ISP-BR-A with weak preference
   over secondary link.
   Note that we always announce Pref-A to ISP-BR-A, and Pref-B to ISP-
   BR-B.

o  For outbound packets, ISP-BR-A will advertise (1) default route (or
   specific routes) toward E-BR-A with strong preference over primary
   link, and (2) default route (or specific routes) toward E-BR-B with
   weak preference over secondary link.  Similarly, ISP-BR-B will
   advertise (1) default route (or specific routes) toward E-BR-B with
   strong preference over primary link, and (2) default route (or
   specific routes) toward E-BR-A with weak preference over secondary
   link.

Under this configuration, both inbound and outbound packet can survive
link failure on either side.  Routing information with weak preference
will be available as backup, for both inbound and outbound cases.

4.  Extensions for IPv6

RFC2260 is written for IPv4 and BGP.  With IPv6 and BGP4+, or IPv6 and
RIPng, similar result can be achieved, without violating IPv6
addressing/routing rules.

4.1.  IPv6 rule conformance

In RFC2260, we announce Pref-A toward ISP-BR-A only, and Pref-B toward
ISP-BR-B only.  Therefore, there will be no extra routing announcement
to the outside of the site.  This conforms to the aggregation
requirement in IPv6 documents.  Also, RFC2260 does not require portable
addresses.

4.2.  Address assignment to the nodes

In IPv4, it is usually assumed that a node will be assigned single IPv4
address.  Therefore, RFC2260 assumed that addresses from Pref-A will be
assigned to nodes near E-BR-A, and vice versa (second bullet in the
previous section).

With IPv6, a node can be assigned multiple IPv6 addresses.  So we can
assign (1) one address from Pref-A, (2) one address from Pref-B, or (3)
two addresses from both address prefixes, to single node in the site.

This will allow more flexibility to nodes in the site.  However, this
may make source address selection on a node more complex.  Source
address selection itself is out of scope of the document.

Hagino                  Expires: January 23, 2000               [Page 4]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

4.3.  Configuration of links

With IPv6, primary link can be IPv6 native connectivity, RFC1933
[Gilligan, 1996] IPv6-over-IPv4 configured tunnel, 6to4 [Carpenter,
1999] IPv6-over-IPv4 encapsulation, or some others.

If tunnelling-based connectivity is used in some of primary links,
administrators may want to avoid IPv6-over-IPv6 tunnels for secondary
links.  For example, if:

o  primary links to ISP-A and ISP-B are RFC1933 IPv6-over-IPv4 tunnels,
   and

o  ISP-A, ISP-B and the site have IPv4 connectivity with each other,

it makes no sense to configure a secondary link by IPv6-over-IPv6
tunnel, since it will actually be IPv6-over-IPv6-over-IPv4 tunnel.  In
this case, IPv6-over-IPv4 tunnel should be used for secondary link.
This configuration has a big win as secondary link will be able to have
the same path MTU than the primary link.

4.4.  Using RFC2260 with IPv6 and BGP4+

RFC2260 approach on top of IPv6 will work fine as documented in RFC2260.
There will be no extra twists necessary.

4.5.  Using RFC2260 with IPv6 and RIPng

It is possible to run RFC2260-like configuration with RIPng [Malkin,
1997] , with careful control of metric.  Routers in the figure needs to
increase RIPng metric on secondary link, to make primary link a
preferred path.

If we denote the RIPng metric for route announcement, from router R1
toward router R2, as metric(R1, R2), the invariants that must hold are:

o  metric(E-BR-A, ISP-BR-A) < metric(E-BR-B, ISP-BR-A)

o  metric(E-BR-B, ISP-BR-B) < metric(E-BR-A, ISP-BR-B)

o  metric(ISP-BR-A, E-BR-A) < metric(ISP-BR-A, E-BR-B)

o  metric(ISP-BR-B, E-BR-B) < metric(ISP-BR-B, E-BR-A)

Note that smaller metric means stronger route in RIPng.

Hagino                  Expires: January 23, 2000               [Page 5]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

5.  Issues with ingress filters in ISP

If the upstream ISP imposes ingress filters [Ferguson, 1998] to outbound
traffic, story becomes much more complex.  A packet with source address
taken from Pref-A must go out from ISP-BR-A.  Similarly, a packet with
source address taken from Pref-B must go out from ISP-BR-B.  Since none
of the routers in the site network will route packets based on source
address, packets can easily be routed to incorrect border router.

One possible way is to negotiate with both ISPs, to allow both Pref-B
and Pref-A to be used as source address.  This approach does not work if
upstream ISP of ISP-A imposes ingress filtering.  Since there will be
multiple levels of ISP on top of ISP-A, it will be hard to understand
which upstream ISP imposes the filter.  In reality, this problem will be
very rare, as ingress filter is not suitable for use in large ISPs where
smaller ISPs are connected beneath.

Another possibility is to use source-based routing at E-BR-A and E-BR-B.
In this case, secondary link needs to be IPv6-over-IPv6 tunnel.  When an
outbound packet arrives to E-BR-A with source address in Pref-B, E-BR-A
will forward it to secondary link (tunnel to ISP-BR-B) based on source-
based routing decision.  The packet will look like this:

o  Outer IPv6 header: source = address of E-BR-A in Pref-A, dest = ISP-
   BR-B

o  Inner IPv6 header: source = address in Pref-B, dest = final dest

Tunneled packet will travel across ISP-BR-A toward ISP-BR-B.  The packet
can go through ingress filter at ISP-BR-A, since it has outer IPv6
source address in Pref-A.  Packet will reach ISP-BR-B and decapsulated
before ingress filter is applied.  Decapsulated packet can go through
ingress filter at ISP-BR-B, since it now has source address in Pref-B
(from inner IPv6 header).  Notice the following facts when configuring
this:

o  Not every router implements source-based routing.

o  Interaction of normal routing and source-based routing at E-BR-A
   (and/or E-BR-B) can be vary by router implementations.

o  Interaction of tunnel egress and filter rules at ISP-BR-B (and/or
   ISP-BR-A) can be vary by router implementations and filter
   configurations.

6.  Observations

We limited the number of ISPs to 2 in the document, but it can easily be
extended to the cases where we have 3 or more upstream ISPs.

Hagino                  Expires: January 23, 2000               [Page 6]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

If you have many upstream providers, you would not make all ISPs backup
each other, as it requires O(N^2) tunnels for N ISPs.  Rather, it is
better to make N/2 pairs of ISPs, and let each pair of ISP backup each
other.  It is important to pick pairs which are unlikely to be down
simultaneously.  In this way, number of tunnels will be O(N).

Suppose that the site is very large and it has ISP links in very distant
locations, such as in US and in Japan.  In such case, it is wiser to use
this technique only among ISP links in US, and only among ISP links in
Japan.  If you use this technique between ISP A in US and ISP B in
Japan, the secondary link make packets travel very long path, for
example, from host in the site in US, to E-BR-B in Japan, to ISP-BR-B
(again in Japan), and then to the final destination in US.  This may not
make sense for actual use, due to excessive delay.

Similarly, in a large site, addresses must be assigned to end nodes with
great care, to minimize delays due to extra path packets may travel.  It
may be wiser to avoid assigning an address in a prefix assigned from
Japanese ISP, to an end node in US.

If one of primary link is down for a long time, administrators may want
to control source address selection on end hosts so that secondary link
is less likely to be used.  This can be achieved by marking unwanted
prefix as deprecated.  Suppose the primary link toward ISP-A has been
down.  You will issue router advertisement [Thomson, 1998; Narten, 1998]
packets from routers, with preferred lifetime set to 0 in prefix
information option for Pref-A.  End hosts will consider addresses in
Pref-A as deprecated, and will not use any of them as source address for
future connections.  If an end host in the site makes new connection to
outside, the host will use an address in Pref-B as source address, and
reply packet to the end host will travel primary link from ISP-BR-B
toward E-BR-B.

Some of non-goals (such as "best" exit link selection) can be achieved
by combining technique described in this document, with some other
techniques.  One example of the technique would be the
source/destination address selection heuristics on the end nodes.

7.  Security considerations

The configuration described in the document introduces no new security
problem.

If primary links toward ISP-A and ISP-B have different security
characteristics (like encrypted link and non-encrypted link),
administrators needs to be careful setting up secondary links tunneled
on them.  Packets may travel unwanted path, if secondary links are
configured without care.

Hagino                  Expires: January 23, 2000               [Page 7]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

References

Bates, 1998.
T. Bates and Y. Rekhter, "Scalable Support for Multi-homed Multi-
provider Connectivity" in RFC2260 (January 1998). ftp://ftp.isi.edu/in-
notes/rfc2260.txt.

Hinden, 1998.
R. Hinden and S. Deering, "IP Version 6 Addressing Architecture" in
RFC2373 (July 1998). ftp://ftp.isi.edu/in-notes/rfc2373.txt.

Durand, 1999.
A. Durand and B. Buclin, "6Bone Routing Practice" in RFC2546 (March
1999). ftp://ftp.isi.edu/in-notes/rfc2546.txt.

Gilligan, 1996.
R. Gilligan and E. Nordmark, "Transition Mechanisms for IPv6 Hosts and
Routers" in RFC1933 (April 1996). ftp://ftp.isi.edu/in-
notes/rfc1933.txt.

Carpenter, 1999.
B. Carpenter and K. Moore, "Connection of IPv6 Domains via IPv4 Clouds
without Explicit Tunnels" in draft-ietf-ngtrans-6to4-02.txt (June 1999).
work in progress material.

Malkin, 1997.
G. Malkin and R. Minnear, "RIPng for IPv6" in RFC2080 (January 1997).
ftp://ftp.isi.edu/in-notes/rfc2080.txt.

Ferguson, 1998.
P. Ferguson and D. Senie, "Network Ingress Filtering: Defeating Denial
of Service Attacks which employ IP Source Address Spoofing" in RFC2267
(January 1998). ftp://ftp.isi.edu/in-notes/rfc2267.txt.

Thomson, 1998.
S. Thomson and T. Narten, "IPv6 Stateless Address Autoconfiguration" in
RFC2462 (December 1998). ftp://ftp.isi.edu/in-notes/rfc2462.txt.

Narten, 1998.
T. Narten, E. Nordmark, and W. Simpson, "Neighbor Discovery for IP
Version 6 (IPv6)" in RFC2461 (December 1998). ftp://ftp.isi.edu/in-
notes/rfc2461.txt.

Acknowledgements

The document was made possible by cooperation from people in ipngwg
multihoming design team, people in KAME project and George Tsirtsis.

Hagino                  Expires: January 23, 2000               [Page 8]


DRAFT         IPv6 multihoming support at site exit routers    July 1999

Author's address

     Jun-ichiro Hagino
     Research Laboratory, Internet Initiative Japan Inc.
     Takebashi Yasuda Bldg.,
     3-13 Kanda Nishiki-cho,
     Chiyoda-ku,Tokyo 101-0054, JAPAN
     Tel: +81-3-5259-6350
     Fax: +81-3-5259-6351
     email: [email-masked]

Hagino                  Expires: January 23, 2000               [Page 9]

---

From: [email-masked]
Date: Wed, 28 Jul 1999 10:02:26 +0900

>	No I do not think so.  Source address selection is a separate question

	Oops sorry, I just meant that this must be tackled separately
	from RFC2260.  Some proposals are there, for example.
	- draft-yamamoto-wide-comm-model (expired)
	- draft-draves-ipngwg-simple-srcaddr-01.txt

itojun

---

From: Joe Abley <[email-masked]>
Date: Wed, 28 Jul 1999 13:50:48 +1200

> 	This does not require extra (non-aggregatable) routing announcement.

My apologies -- I mis-read the RFC (or at least skimmed it over-
superficially). This is the section that I remembered:

5.1. Advertising reachability information by enterprise border routers

   When an enterprise border router connected to a particular ISP
   determines that the connectivity between the enterprise and the
   Internet is up through all of its ISPs, the router advertises (to the
   border router of that ISP) reachability to only the address prefix
   that the ISP allocated to the enterprise. This way in a steady state
   routes injected by the enterprise into its ISPs are aggregated by
   these ISPs, and are not propagated into the "default-free" zone of
   the Internet.

I think the tunnels you refer to are those included in the following
section, under the heading of "Further Improvements", which I had
either not noticed or had forgotten about:

   An enterprise border router would maintain EBGP peering not just with
   the directly connected border router of an ISP, but with the border
   router(s) in one or more ISPs that have their border routers directly
   connected to the other border routers within the enterprise.  We
   refer to such peering as "non-direct" EBGP.

[...]

   Forwarding along a route received over non-direct peering should be
   accomplished via encapsulation [RFC1773].

I see you have forwarded a draft document for how this might apply to
IPv6, which is a good thing, I think -- as I believe I have illustrated,
there is potentially more than one interpretation of what "RFC2260"
can mean with respect to the multi-homing issue.

Regards,

Joe

---

From: Steve Deering <[email-masked]>
Date: Tue, 27 Jul 1999 19:07:03 -0700

At 12:02 PM +1200 7/28/99, Joe Abley wrote:
>Incidentally, are the Oslo minutes published yet? 

Not yet.  Stand by...  (I just made an editing pass over them, to convert
them from Word to ASCII, and have sent them back to Allison for
proofreading.  They will be posted as soon as she gives me the OK.)

Steve

---

From: Jim Bound <[email-masked]>
Date: Tue, 27 Jul 1999 22:14:11 -0400

>   accomplished via encapsulation [RFC1773].

In my mail today earlier I stated I thought 2260 could help too but at
added cost.  I feel the first one above will not fly here because it
breakes aggregation of IPv6 model.  The second one will work but you
have to have another connection to the other ISP and they diagram that
in 2260.  But if want to be multihomed and use both it costs more and
that seems natural to a capitalist like me.

/jim

---

From: Joe Abley <[email-masked]>
Date: Wed, 28 Jul 1999 14:24:04 +1200

Hi,

>                      draft-itojun-ipv6-2260-00.txt

This draft clarifies the aspects of RFC2260 which you mentioned in
an IPv6 context considerably. However, I am not convinced that this
approach fully addresses the issue at hand.

Under your proposal, host renumbering is avoided for the specific
failure mode where a multi-homed site suffers a single circuit failure
to one of its ISPs.

In my experience, multi-homing with IPv4 is rarely a simple affair
(especially when the customer multi-homing does not meet the criteria
to warrant a nice large address prefix, and are using something only
marginally acceptable in the core like a /24).

For this reason, customers who are concerned about circuit failure
between their network and their ISP generally obtain multiple circuits
to the same ISP. This avoids the route aggregation problems, and gives
them all the protection they require. It's also usually cheaper than
buying separate service from a second ISP. We call this "multi-
attaching" rather than multi-homing, but that's just us.

Customers who go to the trouble of multi-homing typically do so because
they are worried about the stability of _transit_ through a single ISP.

For example, we are multi-homed (currently) to Concentric Networks and
Telstra -- both organisations provide us with global transit. This has
saved our bacon in the past when one of those providers has suffered
an outage which left us connected, but broke the transit (in the
example that springs to mind, we were left with excellent connectivity
to one Concentric POP and nothing to the rest of their network).

I'm not making a point about Concentric in particular, by the way -- they
were bitten by a nasty fault, and they have since eliminated the single
point of failure in their usual efficient manner. However, the point is
that transit failures _do_ happen, regardless of how robust a single
provider engineers their network.

Under your draft, a failure of this kind would still require
renumbering at the edge of the network, and special treatment for
established sessions (if they were to persist, which I would suggest
probably ought to be documented as a requirement).

As I mentioned earlier, when I read rfc2260 my attention was drawn to
section 5.1, which I think addresses this exact issue: it attempts to
ensure that the purity is maintained in the default-free zone in the
absense of failures, but can be sacrificed in the interests of
connectivity during failures only.

I'm not naive enough to believe that 2260, as it stands, is without
operational concerns. However, I believe the framework in 5.1 addresses
a more relevant operational concern than that in 5.2.

Regards,

Joe

---

From: [email-masked]
Date: Wed, 28 Jul 1999 12:01:03 +0900

>marginally acceptable in the core like a /24).

	I agree, I believe the word "multi-homing" has several aspects in it.

>to one Concentric POP and nothing to the rest of their network).

	Just curious, how did you solve this case for IPv4?  You had AS number
	and Concentric (or ISP-A) announced your route properly so you were
	saved?

>probably ought to be documented as a requirement).

	For newly established sessions a sentence in Observation section
	may help (setting preferred lifetime of unreachable prefix to 0).

	The draft gives few help about session stability (make connected tcp
	session survive failures).  As Abstract section says, this is very
	basic support and is waiting for more sophisticated (complex)
	mechanisms on top of it.  
	I believe we need a session layer for it if we try to design it
	properly, this is not the job for layer 3 :-)

>a more relevant operational concern than that in 5.2.

	Section 5.1 is, I believe, not acceptable for IPv6 for aggregation
	concerns.

itojun

---

From: Joe Abley <[email-masked]>
Date: Wed, 28 Jul 1999 16:14:51 +1200

> 	and Concentric (or ISP-A) announced your route properly so you were

Correct. Telstra's advertisement of our route (which is normally "far
away" from most of the internet) became the most available prefix, and
hence traffic flowed in that direction.

> 	Section 5.1 is, I believe, not acceptable for IPv6 for aggregation

I don't think any of the approaches I've seen so far address all
concerns. I presume we are looking for a compromise.

Joe

---

From: Jim Bound <[email-masked]>
Date: Wed, 28 Jul 1999 23:30:49 -0400

>they are worried about the stability of _transit_ through a single ISP.

Or for geographical reasons which is why we are multihomed.
There are many reaons folks want to be multihomed is my point.  Should
we document all those reasons?

>probably ought to be documented as a requirement).

I don't see that?  Nor the need for renumbering?

I also dont' support such a solution with renumbering for the e2e
reasons I sent out which is not just technical but mostly that reason.

>connectivity during failures only.

The auto-route-injection in 5.1 breaks aggregation too unless one uses
the optimized IBGP discussed, which I think may work and dovetail well
with Itojun's proposal.  But I still believe Itojun's proposal will work
as a 2260 mechanism and if you want to state cases where it will not
work please do the following:

- Highlight the solution Itojun proposes in the draft and pictures.

- Then describe a scenario where it will not work and why.

I know your not hand waving here but an engineer that puts for work in
this community deserves some order in responses when a WG member says
the proposal won't work.  We grill each others specs but sometimes we
need to grill the input.

>a more relevant operational concern than that in 5.2.

I think we should verify that 5.1 does not break the IPv6 AGGR model for
that address space.

/jim

---

From: [email-masked]
Date: Thu, 29 Jul 1999 12:37:08 +0900

>I think we should verify that 5.1 does not break the IPv6 AGGR model for

	Sorry I'm sure that 5.1 does break the aggregation.  It announnces
	same address prefix (portable, or a prefix from ISP-A) toward two or
	more ISPs.

itojun