Subject: Re: a response to "A Simple Solution for IPv6 Multihoming"

From: Jessica Yu <[email-masked]>
Date: Thu, 16 Sep 1999 16:34:57 -0400

I think with this rounds discussion, many things has been clarified and 
the remain issue is "Does SMPL provide reasonable redundancy?" 

For this issue, I have already stated my reason of why it'd provide
reasonable redundancy and I do not seem to be able to convince you.
I do not wish to continue beating a dead horse but let me ask you
a question:

In the last 2-3 years, how many times a good quality ISP went complete 
isolated from the Internet?  I do not recall one.  Many good quality ISPs 
are building very redundant physical network to the extend every device
is mirrored.  Now some ISPs (at least one of them) offer 100% uptime service.  
If it's not met, a fee will be repaid to the customer.  It's more
incentive for ISP to have a robust network.

No one claims it is a perfect solution. As I am sure you know how hard
to have a perfect solution under the contrains of mandatory aggregation
and also meet the management scalability. 

Multihoming to the same ISP with multiple (usually 2) connections as
you suggested is certainly a method many multihomed customer use today
for IPv4. It helps avoid leaking longer prefix to the Internet and is
very much encouraged by ISPs.  Some still choose still use different
ISPs to multihome with to avoid completely locked in with one ISP and
we can understand why they do that.
 
>> 	With the RFC2260/itojun proposal as is, I am not total convinced
>> 	that it provides managment scalability for ISPs who potentially
>> 	have to manage hundreds or more of them.  I like to be convinced
>> 	otherwise. 
>> 
>
>Given that RFC2260/itojun is the only proposal presented that actually meets 
>the goals of multihoming, I don't understand the need for convincing.

You missed my point. I am saying I am not convinced for it's management
scalability.  Tunnel in general is a network management nightmare. It's
easy to make config mistake and hard to debug.  Tunnel between ISPs
routers is even scareier.

>>- -- 
>>--b

						--Jessica

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Ben Black <[email-masked]>
Date: Thu, 16 Sep 1999 14:04:03 -0700

> isolated from the Internet?  I do not recall one.  Many good quality ISPs 

I am not going to name names because I don't think it is appropriate.  
Given where you work, I would suggest investigating events associated with
AS0.  Even the very best ISPs, and I definitely include UUNet in that group
have catastrophic failures.  It happens.  One way to enormously reduce the
risk of such a single event eliminating all global reachability for a customer
is to multihome to multiple ISPs in such a way that total failure of one does
not imply total reachability failure for the customer.  In the case of SMPL,
that single point of failure is inherent and unacceptable.

> incentive for ISP to have a robust network.

So a fee is repaid and the company is out all the revenue they would have
generated by being up, not to mention the longer term market effects of
failures.  Justifying the single point of failure in SMPL is not going to
make it go away.

> routers is even scareier.

Tunnels are inferior to simply advertising a single prefix for your network
into multiple ISPs.  However, the tunnels required in RFC2260/itojun are
a necessary evil to accomodate constraints in the IPv6 routing architecture.
SMPL may lack tunnels, but it similarly lacks removal of a single point of
failure.

-- 
 --b

---

From: Lloyd Wood <[email-masked]>
Date: Thu, 16 Sep 1999 22:01:45 +0100 (BST)

On Thu, 16 Sep 1999, Jessica Yu wrote:

> isolated from the Internet?  I do not recall one. 

That is a meaningless statement, since good-quality ISPs by definition
don't have isolation outages. I'm pretty sure an entire redneck US
state accidentally got cut off from the rest of the planet for sixteen
hours by a backhoe at one point.

A more concrete counterexample?
http://www.ntk.net/index.cgi?back=archive99/now0226.txt&line=18#l
or search www.ntk.net for 'Telehouse'.

L.

there are still too many chokepoints, and not enough redundant meshing.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: Jessica Yu <[email-masked]>
Date: Fri, 17 Sep 1999 14:02:40 -0400

>SMPL may lack tunnels, but it similarly lacks removal of a single point of

	I just read draft-itojun-ipv6-2260-00.txt again and turns out
	that same redundancy issue you worried about SMPL would happend
	to that scheme as well.  Let me explain.

	The prosaol have a subset of hosts in a multihoming site assigned
	with address space from ISP-A and a subset with that from ISP-B.
	And under the assumption of aggregation, only ISP-A advertise
	it's aggregate (say address-A) to the entire Internet and only 
	ISP-B advertise address-B to the entire Internet e.g. ISP-C, ISP-D
	ect.  If ISP-A has massive failure, half of the hosts assigned to 
	address space from A at the multihoming site is not going to be 
	reachable. If ISP-A lost all its connection to ISP-D, then ISP-D
	and anything behind it can not reach the hosts in address-A space
	of the site.  Same thing applies if ISP-B had massive failure.  So 
	the reachability of a multihoming site realies on a single ISP as 
	much as that in SMPL's situation.  If anyone claim SMPL does not 
	provide reasonable redundancy, then RFC2260/itojun does not do it 
	either. 

	The second issue with RFC2260/itojun proposal that I hope to
	get clarification is that how exactly the tunnel or the
	secondary link is exactly built.  Any tunnel has to have
	physical path underneath it. 

	Just use the picture in the I-D

	(ISP A)                         (ISP B)

        ISP-BR-A                       ISP-BR-B
            | |                         | |
            | \-----------------------+ | |
            |     Secondary link      | | |
            |  +----------------------|-/ |
            |  |                      |   |
            |  |                      |   |
            |  |                      |   |
            |  |                      |   |
        +---|--|----------------------|---|--+
        | E-BR-A                      E-BR-B |
        |                                    |
        |                                    |
        +------------------------------------+_ 

	The Secondary link (tunnel) between ISP-BR-A and E-BR-B has to
	be built either on physical link ISP-BR-A to E-BR-A or some links 
	between ISP-A and ISP-B. Obviously, one can not use the former
	since the tunnel is servered as the backup when the link between
	ISP-BR-A/E-BR-A goes down. So the tunnel has to be using links
	between ISP-A and ISP-B which will entirely depend on how they
	route each other.  If someone worries about link congestion
	between ISPs, you can not avoid it here as well.  I think Rob
	from Sprint pointed out the same issue in his earlier message
	although I have not read it carefully yet.

	One issue with tunnels is that it will make packets which would
	normally fit in max MTU size exceed the max MTU size due to the
	adding of IP header for encapsulation. This will cause routers
	to waste resource for packet fragmentation which led to very 
	poor performance.  Many Tunnel based VPN users suffer from the
	problem.

	Another issue with tunnel is the performance. It certainly 
	cause router resource to encapsulate and decapsulate every single
	packet using tunnels. A large ISP's border router (or customer
	aggregation router) usually connects hundreds of end customers. 
	It has scaling impact to have that router to do all these
	extra resource comsuming tasks.  Also, the performance would be
	bad for the multihoming customer as well with the tunnel.  When
	the tunnel is in use, that's means one connection is down, all
	the traffic will concentrate on the other sole connection and
	the sole poor router.  Adding more burden on it (it has to
	decapsulate every packet) it just not wise.

	Also, with this scheme, the operator of the multihoming site has
	to be very clueful and ip routing savor since it's a complexed
	routing setup (just read the RFC2260/itojun draft and try to
	config your site).  Most of the site do not have such expertises.
	As a result, misconfiguration or error happen will very likely to
	happen. For example, if the priority is not set right, traffic
	may well use the tunnel even thought the primary link is health.
	That will very much impace the performance.  Connectivity will
	be impact if tunnel is not set right, tunnel associated problem
	is usually very hard to detect. Anyway, I can go on and on but
	I think the point is made and I will stop.

	In summary, RFC2260/itojun does not seem to provide better
	redundancy than SMPL. The problem associated with tunnel mechanism
	should not be overlooked.

							--Jessica

---

From: [email-masked]
Date: Sat, 18 Sep 1999 03:39:31 +0900

First of all, let me clearify this: As draft-itojun-ipv6-2260 states
	in very early part of the document, the draft covers only part of
	multihoming problems.  There are so many goals in multihoming.
	To name a few:
	- load balancing
	- low delay
	- various failure modes
		ISP links
		ISP routers, site border routers
		the whole-smash shutdown in ISP
	draft-itojun-ipv6-2260 covers ISP link failure, no others.  For other
	issues too many policies are involved, and most of those need global
	knowledge about the network (like current bandwidth usage in both
	ISP links) so I don't feel like we have any generic solution.
	I think my draft is clear enough about the scope of the problem.

	There's no single solution against this kind of problem.
	Also, we do not really need to pick only single technology against the
	problem.  We can pick techniques depending on site configuration.
	(we may need to suggest "default combo" of technologies, though)

>	Just use the picture in the I-D

	Secondary link to ISP-A must be configured to go through different
	physical medium than primary link to ISP-A.  There's no other
	requirement imposed.  It is up to you to choose appropriate medium
	for secondary link.

	Secondary link can be configured in various ways depending on
	configuration in your site (explored in section 6).  For example,
	- if primary link is RFC1933 tunnel, you can configure another
	  RFC1933 tunnel.  There'll be no MTU issue raised.
	- If primary link is IPv6 leased line, you can of course configure
	  secondary link as IPv6-over-IPv6 tunnel.  In this case you have
	  MTU issue.
	- you may be able to purchase extra leased line for secondary link,
	  of course.

>	poor performance.  Many Tunnel based VPN users suffer from the

	I think the above statement is not correct.
	If you properly send icmp6 need fragment message, the end nodes
	will do path MTU discovery and edge routers do not need to fragment
	the packets.
	I agree lower path MTU may degrade performance, but I believe edge
	routers do not need to perform extra fragmentation on tunnel ingress.

itojun

---

From: Jessica Yu <[email-masked]>
Date: Fri, 17 Sep 1999 10:45:23 -0400

Yes, there has been fiber cuts which affect part of the service area
of some ISPs.  But to put this into the context of our discussion, this 
does not mean that the ISP is completely isolated so that it can not
announce the aggregate including the multihoming site (customer-A in our
example) out to the Internet. So the customer can still utilize the 2nd
ISP for its connectivity. 

Also, there is highly a chance that a massive fiber cut in an area 
effects mutliple ISPs in the area, so a multihoming site is very likely
to loose all its connections even with today's arrangement.  

By the way, your URL does not seems to pointing to something relevant.

						--jessica
 
Date:    Thu, 16 Sep 1999 22:01:45 BST
To:      Jessica Yu <[email-masked]>
cc:      [email-masked], [email-masked]

From:    Lloyd Wood <[email-masked]>
Subject: Re: a response to "A Simple Solution for IPv6 Multihoming" 

Reply-To: Lloyd Wood <[email-masked]>
Organization: speaking for none
Content-Length: 697

L.

---

From: Magnus Ahltorp <[email-masked]>
Date: 18 Sep 1999 04:00:38 +0200

> 	poor performance.  Many Tunnel based VPN users suffer from the

This is not true. In IPv6, routers do not fragment packets.

> 	decapsulate every packet) it just not wise.

I fail to see the performance problem. When encapsulating a packet in
a IPv6 packet, there is *one* field (the size) that needs to be filled
in. The rest of the packet is static for a specific tunnel. It will
cost one more routing table lookup, but not more. If the current
routers you're using cannot do this simple operation in full speed, go
buy one that will. If the router manufacturer is really clever, even
the extra lookup is not needed.

Decapsulating packets is even simpler. It's just a matter of stripping
the header off the packet an reinserting it into the routing table
lookup. Cost: one more routing table lookup.

> 	should not be overlooked.

Yes it does, since you can assign multiple addresses to each host.

Magnus Ahltorp
Department of Teleinformatics
Royal Institute of Technology
Stockholm, Sweden

---

From: Lloyd Wood <[email-masked]>
Date: Sun, 19 Sep 1999 00:21:56 +0100 (BST)

> ISP for its connectivity. 

The examples I gave to answer your question are where this did not
occur.

L.

interesting definition of 'isolated'.

> Return-Path: [email-masked] 
> X-Sender: [email-masked]
> In-Reply-To: <[email-masked]>
> X-url:   http://www.ee.surrey.ac.uk/Personal/L.Wood/
> X-no-archive: yes
> MIME-Version: 1.0
> Content-Type: TEXT/PLAIN; charset=US-ASCII
> Content-Length: 697
>

---

From: Jessica Yu <[email-masked]>
Date: Mon, 20 Sep 1999 13:35:46 -0400

>Yes it does, since you can assign multiple addresses to each host.

	If multiple addresses per host is used then there is not 
	necessary of using tunnle to gain redundancy.  If multiple 
	addresses per host is not used, then as I mentioned above, it 
	does not provide better redundancy than SMPL.

	Please also be aware that using multiple addresses per host 
	will introducing another array of difficult technical issues 
	in addition to complex management issues. 

					--Jessica

---

From: Bill Sommerfeld <[email-masked]>
Date: Mon, 20 Sep 1999 13:52:46 -0400

> 	in addition to complex management issues. 

well, given that v6 introduces link-local and particularly site-local
addresses, you have to tackle these issues anyway...

					- Bill

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 20 Sep 1999 13:58:07 -0500

> addresses, you have to tackle these issues anyway...

Indeed. It's a baseline assumption that IPv6 hosts may have multiple
addresses, including multiple global addresses. That's exactly
why we can have more complex multihoming solutions....
it's a feature, not a bug.

  Brian

---

From: [email-masked]
Date: Tue, 21 Sep 1999 09:40:09 +0900

>	does not provide better redundancy than SMPL.

	You need to provide redundancy (cope with link failure) when
	you have multiple addresses assigned to a host.

	Consider you have multiple address on a host (say A:x and B:x).
	You register A:x and B:x onto the DNS database.
	For inbound connection (outside to your site), it is up to outsider
	to choose which address to use.  Outsider may send a packet to A:x
	even if ISP link A is down.  Therefore, boundary routers need
	to provide reachability regardless of link failure.  So you need
	2260/itojun.
	For outbound (your site to outside) it is up to host (not the
	boundary routers) to choose source address for the connection.
	If ISP link A is down and the host happen to use A:x as the source
	address, reply packet will fail to come back.  Therefore, again you
	need to provide reachability regardless of link failure.

	From my understanding (correct me if I'm wrong) SMPL assumes the
	adjacent ISPs to have specific connectivity.   I think the assumption
	needs to be decreased.

itojun

---

From: Magnus Ahltorp <[email-masked]>
Date: 21 Sep 1999 03:08:59 +0200

> 	does not provide better redundancy than SMPL.

There is a reason to use tunnels even if multiple addresses are used.
The reason is the simple fact that people want their connections to
live through a failure. In Francis' draft, the same thing is achieved
by using binding updates.

> 	in addition to complex management issues. 

Since all IPv6 hosts must have a link-local address, each IPv6 host is
in exactly one of these sets:

1. Those that have only link-local addresses.
2. Those that have multiple addresses.

The members of the first set is not affected by any routing.

The members of the second set obviously already have multiple
addresses.

Magnus Ahltorp

---

From: Quality Quorum <[email-masked]>
Date: Mon, 20 Sep 1999 23:03:55 -0400 (EDT)

> 	you have multiple addresses assigned to a host.
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I am sorry but I do not see that it is so. It seems to me that correct 
expression is 'multiple prefixes assigned to a site'.

               ISP-A                            ISP-B

               Access-router-A            Access-router-B
            Customer-subnet A            Customer-subnet-B

                 Host       Customer-internal-router

Say, your primary ISP is -A, your host has only one prefix (given
by ISP-A), one address and one DNS entry. ISP-B advertizes 
an expensive route to customer subnet A  - and this is it.

> 	to provide reachability regardless of link failure.  So you need

So, it seems to me that (1) we do not need 2260/itojun here, (2)
it will spill as many routes (namely one) into DFZ as 2260/itojun would.

> itojun

Thanks,

Aleksey

---

From: Jessica Yu <[email-masked]>
Date: Tue, 21 Sep 1999 11:37:37 -0400

>addresses, you have to tackle these issues anyway...

	These really are different problem to tackle.  Link-local and
	site-local addresses are not globally visible so there is
	no global routing associated issues. Contrastly, the multiple 
	addresses per host for multihoming sites (relevant to this thread 
	of discussion) DO have these issues (e.g. selection of one of 
	multiaddress could have impact on the global connectivity or
	failure of connectivity).  Therefore, having mechanism to deal 
	with the link-local/site-local problem does not mean we also have
	a solutions for multi-address multihoming issues.

						--Jessica

---

From: Jim Bound <[email-masked]>
Date: Tue, 21 Sep 1999 12:54:13 -0400

Folks,

Catching up on mail on this subject.  Seems we keep discussing the same
thing over and over and I hope the Japan meeting is more productive than
this mail.  It seems to me that we need some guidelines for the meeting
discussion or you will all be in questionable technical ratholes for the
whole time.

I think Steve Deering, Bob Hinden, and Christian Huitema provided the
list with clear axioms about what IPv6 does do and whats its
architecture can do and possibly what problems we should solve first and
then what problems we can solve next.  

I also thought Vijay provided some good light on the axioms about
computation and scaling that were valid too without making hard stats,
which we cannot do until we see what the market does with IPv6 (esp ISPs
and Telcos); hence, making radical changes is most likely not prudent
right now for multihoming.  That is why I am still convinced SMPL is the
right proposal and Francis Dupont's recent draft raises some issues too.

The other thing is that I am getting less tolerant of new folks on this
list that blast the old folks and its clear they have not read all the
IPv6 specs that affect this discussion.  Lucky for you I am not on this
discussion previously cause I am going to start being very direct with
those folks and telling them publicly your input is sketchy cause you
have obviously not read all the releative IPv6 specs and speaking
without knowing what your talking about.  All new folks owe the WG to go
and sit down and read and understand all the specs before you tell folks
that are on this list for a long time they are wrong.  Also not
listening to what folks say is pretty unsocial too.  Folks still arguing
against or only we provide aggregation really are clueless.  

I also thought Georges idea of summarizing the proposals is a good idea
as that helped with the transition specs quite well.  

We also need once we are done to ascertain the implementation affects of
all of this and what can be done at what speed and deployed.

Lastly if you get to the point where one wants to swap IPv6 addresses on
the fly discussion DO NOT ASSUME you have a valid solution if it does
not address UDP and the affect to the APIs that are listening for
connects on sockets.   But I doubt you will have time to go here.

I will be working offline with a few folks to get my algorithm and
src/dst address suggestion if that is to be to the group in Japan thru
Richard.

thanks
/jim