Subject: Re: a response to "A Simple Solution for IPv6 Multihoming"

From: Jessica Yu <[email-masked]>
Date: Thu, 16 Sep 1999 11:23:19 -0400

>> 	Do you have any suggestion of maitaining the scaling of the global 
>> 	routing table but not by imposing the restrict aggregation?  If not,
>> 	we still have to work under the assumption.
>>  
>
>Yes, realize that the routing table has sustained linear growth, while
>computing power has increased exponentially.  This argues for using
>existing aggregation and multihoming mechanisms.

	Six years ago when we developing CIDR for IPv4, many folks
	did not think route aggregation was needed at all. Their argument 
	was that memory was cheap and computing power was increasing 
	fast. I am not sure if there are still people think that
	CIDR was not needed for IPv4 now.

	The need for mandatory aggregation for IPv6 could well be a 
	different story.  I think it's a good topic to debate on the 
	list and as a separate topic from SMPL.  Again SMPL is working
	under the assumption. So if we want to continue this thread,
	let's discuss under the mandatory aggregation assumption.

	If the result is that mandatory aggregation is not needed for
	IPv6, then SMPL is not needed at all and I'd be happy to
	put it to rest.  

>> >Unfortunately, the fundamental assumptions regarding the purposes of
>> >multihoming lead the author of the document astray.  The main goals
>> >of multihoming are: redundancy in the face of link and/or network
>> >failure, load balancing, and performance improvement.  Of these, only
>> >load balancing is directly addressed, and even then with limited
>> >success.  Similarly, although improving performance by connecting
>> >directly to networks which are common destinations is effective, the
>> >document does not provide options for accessing content providers
>> >accessible through other networks.
>> 
>> 	I'd argue that the realistic goal of multihoming are 
>> 	a) Redundancy and b) load sharing. The simple multihoming
>> 	proposal (SMPL) addressed both. 
>> 
>> 	'performance improvement' you used here is a vague term, 
>> 	redundancy and load sharing are performance improvement.
>> 	If you mean the improve throughput or goodput of the traffic
>> 	from/to a multihomed site, this is not a realistic goal to
>> 	achieve via multihoming. 
>> 
>
>Nonsense.  If I am sending a large amount of traffic to ISP-A, but I only
>have connectivity to ISP-B and ISP-B has poor connectivity to ISP-A, then
>adding connectivity directly to ISP-A will improve the performance, as
>measured by such factors as latency, packet loss, etc.

 	If this is what you mean, then SMPL allows that.  With SMPL, 
	customer will using its direct link to send traffic to destinations 
	directly connected to its ISPs.  Look at the picture below, 
	Customer-A will send traffic to customers of ISP-2 using link-2 
	and that of ISP-1 using link-1.  Again, please read the proposal
	carefully. 

	If you expect an ISP which Customer-A has no direct connection
	with to send traffic to customer-A according to the path dictate
	by customer-A, then it's unrealistic.

>> >A selection of quotes from the document will help in clarifying its
>> >shortcomings as a solution to multihoming within the existing,
>> >documented IPv6 routing framework.
>> 
>> >From section 2 (Proposal):
>> 
>> >                           ISP-3 ---- ISP-4
>> >                             |      /  |
>> >                             |    /    |
>> >                             |  /      |
>> >                           ISP-1 ---- ISP-2
>> >                    link-1   \          /  link-2
>> >                              Customer-A
>> 
>> >   A multi-homed customer will designate a primary ISP and receive IP
>> >   address from its primary ISP's IPv6 aggregation block. In this
>> >   example, ISP-1 is the primary ISP of customer-A and assigns Addr-1-A
>> >   to the customer. Customer-A will advertise addr-1-A to ISP-1 and
>> >   ISP-2 respectively. ISP-2 will advertise Addr-1-A to ISP-1 and to
>> >   ISP-1 only. ISP-1 will, of course, advertise its own aggregation
>> >   Addr-1 to the entire Internet.
>> 
>> >It is clear from this description that the only redundancy provided
>> >with this solution is for the customer links (link-1 and link-2) and
>> >not for the ISPs themselves.  Specifically, if ISP-1 suffers from a
>> >failure within the POP to which link-1 terminates, whether as an
>> >isolated failure or a more extensive failure within ISP-1, then
>> >Customer-A is no longer able to maintain connectivity despite their
>> >possession of a perfectly good alternate path through ISP-2.  From an
>> >engineering perspective, no robustness has been gained through this
>> >multihoming which could not be gained more simply with multiple links
>> >to ISP-1.
>> 
>> 	This is not correct. If the POP to which link-1 connects goes 
>> 	down, the connectivity to Customer-A will still remain and
>> 	link-2 will be used.
>> 
>> 	This is because even the POP is down, ISP-1 will still
>> 	advertising Addr-1 which including the address of Customer-A
>> 	to the Internet (ISP-1 will still advertise its block). So
>> 	traffic destinted to Customer-A will still be sent to ISP-1.
>> 	Since ISP-2 advertises the specific address of Customer-A to
>> 	ISP-1, so ISP-1 will just forward the traffic to ISP-2 which
>> 	will further forward traffic to Customer-A vis link-2.
>>  
>
>Your eggs are still all in the ISP-1 basket.  Even the best ISPs have
>wide-scale failures.  Your later assertion that this is reasonable
>and acceptable is contrary to the redundancy goals you agreed with
>for multihoming.

	Ok, so looks like we agree that a pop failure should not
	affect the connectivity of the multihoming site.

	Now, you are talking about the connectivity of the multihoming
	site will be affected if ISP-1 has a massive failure.  In order
	for customer-A complete lost connection, ISP-1 will have to lost
	all its mutliple internet connections at the same time, since
	as long as ISP-1 has external connection, the aggregated routes 
	including Customer-A's block will be advertised to the Internet 
	and the connectivity for customer-A will still be there, which
	is very very rare.  Even for a good quality ISP to lost half
	of its external connections at the same time is very rare. The
	risk won't be higher than the current multihomed site with 
	failure to both links to its directly connected ISPs at the same
	time.  So this certainly not introduce less redundancy that a
	multihoming site has today. 

	This is also documented in my draft, chosing a good quality of
	primary ISP is strongly recommended.
 
>> 
>> 		
>> >From section 2 (Proposal):
>> 
>> >   For inbound traffic to Customer-A, traffic will first be forwarded to
>> >   ISP-1 since it advertises Addr-1 which includes Addr-1-A. ISP-1 will
>> >   then forward the traffic destined to Customer-A via its connection to
>> >   ISP-2 and/or its direct link to customer-A, according to certain
>> >   polices. Most common policy would be the use of the shortest exit. By
>> >   using both connections to forward traffic to Customer-A, load sharing
>> >   is achieved.
>> 
>> >From this statement it is unclear exactly how inbound load balancing
>> >is to be achieved.  Is the author suggesting use of the (rather
>> >troublesome) concept of eBGP multipath?  In the topology described in
>> >the diagram above, implementation of a load sharing mechanism across
>> >link-1 and link-2 would be a decidedly non-trivial matter.  Again,
>> >given the lack of redundancy outlined above, it would be far more
>> >efficient for Customer-A to simply purchase multiple links to ISP-1.
>> 
>> 	No, I am not suggesting the use of eBGP multipath. The method
>> 	is the same as the common mechanism used by ISPs today to deal
>> 	with forwarding traffic to routes learned from multiple network
>> 	points i.e. to use IGP metric to find the closest exit to forward 
>> 	the traffic.  It could use to ISP-2 or could use its direct link 
>> 	to Customer-A. 
>> 
>
>Why would there be more than route to the destination in ISP-1s IGP?  ISP-1
>would pick the *single* best path in BGP and use that *single* path when
>sending traffic to the customer.  Are you suggesting that the IGP would
>see multiple paths to the same destination even though BGP only saw one?
>Or are you suggesting that BGP would carry multiple routes to the same
>destination?

	BGP routes of Customer-A's block will be learned by ISP-1 from both 
	connections (ISP-2 and Customer-A), BGP routes are with same
	local_pref,etc. so the next tier of BGP route selection comes down
	to IGP metric within the ISP-1 network.  The lower IGP metric won 
	and the traffic will be send to that exit point. That's called 
	shortest out routing.

>>> 	I assumed that the reader would be familiar with the mechanism
>>> 	since it's so commonly used.  But maybe I can not assume this
>>> 	and in next revision, I should put more details to explain it. 
>>>  
>> >Additionally, Customer-A must now depend heavily on the connectivity
>> >between ISP-1 and ISP-2.  Often, links to multiple providers are
>> >purchased specifically to avoid traffic crossing these (often
>> >congested) peerings.
>> 
>> 	Unless the traffic are sourced and destined within the multihoming 
>> 	site and its directly connected ISPs (e.g. from customer-A to src/dst
>> 	behind ISP-1 or ISP-2), the traffic has to go through links between 
>> 	ISPs.  This is the case today.  
>> 

>Except that today if there is poor connectivity between a pair of given ISPs
>the result is that you have poor reachability to one of those ISPs.  In the
>configuration you describe, the result is poor *global* reachability in
>the case of failure of a *single* ISP.

	No, you still did not get it. As a multihome site, you have
	no way of dictating what path the incomming traffic will
	take.  So your traffic may well be sent to the 'bad path'
	regardless. It's all depends on the policy of the ISP and
	those ISP could be far far away from you and have no interests
	of taking care of your concern since you do not pay them
	directly. 
	 
>> 	Even without the SMPL, in today's network, a multihomed site
>> 	just can not dictate how the traffic goes back to it.  It's
>> 	all depends a other ISPs' routing policy.  Looks at the 
>> 	picture, even not using the SMPL, it does not gaurantee
>> 	that ISP4 will use ISP-2 to send traffic to customer-A,
>> 	it may well be sending the traffic via its link to ISP-1. 
>> 
>
>The point is that *both* of the paths to the customer are *usable*.  In
>the current system of multihoming, if ISP-1 failed, all traffic would
>flow through ISP-2.  The issue is now how well the customer can control
>reachability to their own network, but whether there is reachability at
>all in the case of upstream failure.

	Are you talking about the redundancy again?  See the discussion
	to this topic above.	

>> 	I also think it's wrong to assume that the links between
>> 	ISPs are by default congested. It was more so the case when
>> 	all ISPs exchange traffic at NAPs (i.e. Public exchange point).
>> 	There are a lot of private links between ISPs at multiple 
>> 	locations and the situation is different now.	 
>> 
>
>No, the situation is not different now.  Private peerings are quite often
>congested and neither ISP involved in a private peering has any reason
>to change that.

	That's simple not true.  I worked on ANSNET's private peering and
	now we are part of UUNET.  In the case of both ISPs, a great
	effort has been made to create private peering at many locations
	and to add bandwidth to improve performance there. Also, since
	I worked with many other ISPs to create private peering, I can say
	they are equally care for private peering to work well.

	May I ask which ISP do you work for?  I hope you speaking for
	your own experience not just speculation. 

>> >From Section 4 (Concerns):
>> 
>> >      - The primary ISP of a multihoming customer (such as ISP-1 in the
>> >      example) would need to do more work in terms of distributing the
>> >      traffic among other connected ISPs and the customer link.  This
>> >      can be done using BGP policy (BGP community) and using shortest
>> >      exit. It will also carry more traffic for the customer. However,
>> >      this is a value added service to the customer and the primary ISP
>> >      can charge more fees for such services.
>> 
>> >Stating that the primary ISP "would need to do more work" is putting
>> >the problem a bit lightly.  The addition of references to BGP policies 
>> >and BGP communities does little to clarify how exactly the author
>> >believes traffic could effectively be distributed.
>> 
>> 	Again, these are common practice which are already in place.
>> 	I should not have assumed the everyone would familiar with
>> 	it.  I will give more detailed explanation to it. 
>> 
>
>Please do.

>> 
>> >From Section 4 (Concerns):
>> 
>> >      - The primary ISP is the sole interface for the multihomed
>> >      customer to the Internet other than the ISPs the customer has
>> >      directly connection with. Outages such as one between ISP-1 and
>> >      ISP-4 would impact the reachability from customers of ISP-4 to
>> >      Customer-A even though ISP-2 still has good connection to ISP-4.
>> >      However, if the primary ISP is a good quality ISP, this sort of
>> >      situation should happen rarely. The reason is that it's common
>> >      practice that an ISP, especially good quality ISP, has multiple
>> >      connections to other big ISPs at different geographical locations.
>> >      Good quality ISPs also have robust network design to prevent any
>> >      failure to impact the whole network. Choosing a good quality and
>> >      robust ISP as primary ISP is a good practice.
>> 
>> >This is really the crux of the problem with the solution put forth in
>> >the document.  The assumption that a given ISP will never have
>> >failures, or that rare failures are acceptable to customers who
>> >multihomed for redundancy, is simply not valid from an engineering
>> >perspective. If the primary goal of multihoming were load sharing,
>> >this solution would provide one, potentially very complicated,
>> >answer.  However, the issue of multihoming for redundancy is totally
>> >ignored, drawing into question the need for a second ISP in the
>> >equation at all.  Why not simply purchase additional bandwidth from
>> >the same ISP?
>> 
>> 	Since noboday can gaurantee absolutely and 100% no outage,
>> 	the measure of 'rarely happen' is really quite a good measure.
>> 	
>
>But irrelevant since you must rely entirely on a single ISP, despite the
>fact that you have more than 1 connection.

	Again, see the dicussion above.

>> 	Even with today's practice, nobody can ganrantee that the two
>> 	ISPs the multihoming site connecting to, will not go down at the
>> 	same time thus leave the site with no connection.  But one
>> 	can say, it rarely happen so we will have two ISP connections instead
>> 	of 3 or 4 or more.
>> 

>Yes, and people do exactly that.  But using the technique described in
>your draft, adding those additional ISPs would provide no benefit.

	For this type of company, which is in minority, may deserve 
	other treatment.   

>> 	Also, if the alternatives to SMPL is to have multiple address 
>> 	assigned to all my hosts (could hundread thousands) or complcated
>> 	tunnels,etc. My feeling (based on my network operationa and engieering
>> 	experience) the chances of my connectivity will be impacted due 
>> 	operation error, misconfigurations, etc. woulud be much more high 
>> 	than use of SMPL.  
>> 

>My feeling, also based on my network operations and engineering experience,
>is that I would rather rely on a slightly more complicated configuration
>which allows arbitrary increases in connectivity to provide additional
>redundancy than on a system that is "simple" but allows almost no
>redundancy and absolutely no scaling.

	You are welcome to propose something which are just slightly
	more complicated.  This proposal has to address managibility to
	the satisfaction of ISPs who will potentially have to manage 
	hundreds if not thousands of such connections.  If it fails
	on this, then it'd be hard to convince ISPs to deploy it. 
  
>
>> >From Section 4 (Concerns):
>> 
>> >   It is desirable to have solution which provides perfect redundancy
>> >   and, at the same time, simplicity to scale management and operation.
>> >   In the absence of a perfect solution, the trade-off needs to be made.
>> >   The author believes that it is very important that the solution has
>> >   to be simple and manageable. Manageability should be the top
>> >   consideration.
>> 
>> >If manageability of the solution is to be the top consideration, the
>> >response put forward does not measure up.  In the name of avoiding
>> >routing table growth the author has suggested adding significant
>> >complexity to the configuration and management of multihomed customers
>> >without showing any benefit for the effort.
>> 
>> 	As mentioned above, the routing aggregation is a given
>> 	contrain that this proposal works under.  The simplicity
>> 	I am mentioned here is more in compare of other proposals
>> 	in terms of complicated tunnel and mulit addressing for
>> 	every single host in a multihoming site.  Compare to
>> 	that, this proposal is highly scalable and managable.
>>

>You are half right.

	Do you mind to explain?
	 
>> >An area not addressed at all in the document is that of multihoming
>> >for access to networks attached to a given ISP, rather than access to
>> >that ISP.  For example, using the diagram above, if Customer-A
>> >purchased connectivity to ISP-2 because ISP-2 had better connectivity
>> >to ISP-4, then the offered multihoming solution is of no use.  ISP-4
>> >will never see reachability to Customer-A via ISP-2.
>> 
>> 	Again, as mentioned above, even without the SMPL, in today's 
>> 	network, a multihomed site just can not dictate how the traffic 
>> 	goes back to it.  It's all depends a other ISPs' routing policy.  
>> 	ISP-4 may well have the policy of sending traffic to ISP-1
>> 	instead of using its link of ISP-2. There are many reasons which 
>> 	led ISP-4 do that.  In short, ISP-4 will look into the interets
>> 	of itself rather Customer-A's interests, especially Customer-A
>> 	is not even directly pay for ISP-4.  
>> 
>
>Yes, but ISP-4 will *never* send traffic *directly* to ISP-2 even though
>there is connectivity to the customer via ISP-2.  Even if ISP-4 loses 
>connectivity to ISP-1.

>> 	If someone expects the incomming traffic will take the path it
>> 	wished, then he or she needs to be more realistic. Likewise,
>> 	one should not ask a multihoming proposal address this. 
>> 
>
>No, I expect a multihoming proposal to provide a realistic level of
>redundancy.  SMPL does not meet this requirement.

	I think SMPL provide a reasonable and realistic level of redundancy.
	Again, see the redundancy discussion above. 

>> >The primary lesson of the draft proposal is that the proposed, forced
>> >aggregation hierarchy places sever restrictions upon the engineering
>> >of reliable, high poerformance networks.  To multihome effectively, a
>> >network operator must somehow qualify as a "long-haul provider" or
>> >other entity appropriate for allocation of its own address space.  In
>> >this author's  opinion, the current IPv6 addressing architecture
>> >creates an environment which provides scalability of the global
>> >routing table at the expense of any real hope for redundancy through
>> >multihoming.
>> 
> >Solutions which avoid the use of multihoming with a single address
> >space are possible and offer more hope for a robust network than those 
> >currently proposed.
> 
> 	Looks like you have an issue of the mendetory aggregation for
> 	IPv6. You should raise this to IPv6 WG for this. My proposal
> 	just works under contrain of this assumption.  It's not an
> 	endorsement of such restriction. 
> 

>I do indeed have an issue with the aggregation policies.  However, working
>within them, I believe the RFC2260/itojun proposal provides value as
>a mechanism for improving reachability for a customer network and SMPL
>does not.

	With the RFC2260/itojun proposal as is, I am not total convinced
	that it provides managment scalability for ISPs who potentially
	have to manage hundreds or more of them.  I like to be convinced
	otherwise. 

>> 
>> >REFERENCES
>> 
>> >   [AGGR]    Hinden, R., O'Dell, M., and S. Deering, "An
>> >             Aggregatable Global Unicast Address Format", RFC 2374, July
>> >             1998.
>> >
>> >   [ARCH]    Hinden, R., "IP Version 6 Addressing Architecture",
>> >             RFC 2373, July 1998.
>> >
>> >   [SMPL]    Yu, J., "A Simple Solution for IPv6 Multihoming",
>> >             draft-yu-simple-ipv6multihoming-00.txt, August 1999.
>> 
>> 
>> 
>> 						--Jessica
>
>- -- 
> --b

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Ben Black <[email-masked]>
Date: Thu, 16 Sep 1999 12:32:48 -0700

> 	primary ISP is strongly recommended.

And as documented in my response, one should select a quality ISP no
matter what your final connectivity goals.  However, since SMPL relies
entirely on a *single* ISP for its "redundancy", why not simply connect
to the same ISP twice at different POPs?  Then there is no special 
configuration requirement and no need for *any* added complexity.

> 	and the traffic will be send to that exit point. That's called 

You mean the lower IGP metric that won't ever be used for tie-breaking
since the the direct advertisement from the customer will have a shorter
AS Path than the advertisement from the second ISP?  Or did you have
a different metric in mind?

> 	of taking care of your concern since you do not pay them

The issue is not whether you can dictate *which* of the several paths is used 
for inbound traffic, but whether the multiple paths are even visible to the
global network.  In both the current IPv4 multihoming system and the RFC2260/
itojun proposal there are multiple visible paths, such that in the case of
loss of one of those paths, for whatever reason, there is still reachability.
SMPL simply does not provide for redundancy in the case of primary ISP failure.

> 	Are you talking about the redundancy again?  See the discussion

Yes, I am talking about the illusory redundancy in SMPL again.  See the
discussion of this topic above.

> 	your own experience not just speculation. 

I am.

> 	Again, see the dicussion above.

Please quantify "rarely".  Where exactly does it fall in the range between
"not bloody likely" and "shyah right!"?

> 	For this type of company, which is in minority, may deserve 

So your SMPL proposal is not sufficiently general to handle any but the
absolute minimum case with absolutely minimal redundancy, requiring still
another solution if an engineer wants more than 2 links and more than
"rarely happen" levels of reliability.  Why not just have a single solution
for all cases?

> >> 	that, this proposal is highly scalable and managable.

It's manageable.  

> 	Again, see the redundancy discussion above. 

It provides very little redundancy and, again, please define "realistic"
as the RFC2260/itojun proposal provides significantly higher levels of
redundancy and yet still falls short of current practice in IPv4 multi-
homing.

> 	have to manage hundreds or more of them.  I like to be convinced

Given that RFC2260/itojun is the only proposal presented that actually meets 
the goals of multihoming, I don't understand the need for convincing.

-- 
 --b

---

From: Christian Huitema <[email-masked]>
Date: Fri, 17 Sep 1999 08:30:52 -0400

At 11:23 AM 9/16/99 -0400, Jessica Yu wrote:

>	... As a multihome site, you have

May I point out that, in the current architecture, the only way to specify
the path for incoming traffic is by specifying one of several alternative
addresses, i.e. a different address for each incoming path? This is exactly
what you get with the combination of aggregatable addresses and DNS "A6"
records. One may argue however that there are two missing pieces in the puzzle:

1) a way to quickly update the DNS records, or otherwise affect the choice
of addresses, during partial loss of connectivity, or when some of the ISPs
undergo severe congestion.

2) a way to migrate TCP connections from one pair of addresses to another.

In fact, combining AH or ESP and Mobile IPv6 provides a solution to the
second problem. Maybe we should concentrate on the first one, which is a
subset of the general problem of "choosing the nearest server." 
-- Christian Huitema

---

From: Ben Black <[email-masked]>
Date: Fri, 17 Sep 1999 11:10:06 -0700

There may also be a 0th option: use the existing multihoming and aggregation
mechanism for IPv4 in IPv6.

-- 
 --b

---

From: [email-masked]
Date: Sat, 18 Sep 1999 03:24:28 +0900

>> subset of the general problem of "choosing the nearest server." 

	Is it really possible?  From my understanding currently available
	mobile-ipv6 draft does not address key setup (interaction with IKE).
	If you can clarify how you will be configuring keys on TCP connection
	time, and address pair change, it would be really helpful.

itojun

---

From: Ben Black <[email-masked]>
Date: Fri, 17 Sep 1999 11:35:12 -0700

Is it realistic to assume 1) millions of multihomed sites sooner than
the current growth rate would indicate and 2) that the networking
hardware will not be up to the task by then?

On Fri, Sep 17, 1999 at 11:33:00AM -0700, Steve Deering wrote:
> At 11:10 AM -0700 9/17/99, Ben Black wrote:
> Ben,
> The mechanism used in IPv4 does not scale to millions of multihomed sites,
> using existing routing technology.  We would like to be able to support
> many millions of multihomed sites in IPv6 (e.g., many small enterprises
> connected to multiple ISPs for reliability, households connected to
> the Internet via their phone wire and their cable TV wire, etc.).
> 
> So yes, we could (and probably will) use the same mechanism as in IPv4,
> but only for some relatively small number of lucky or rich customers.
> That still leaves a requirement to develop new multihoming mechanisms
> for all the other potential multihomed sites.
> 
> Steve

-- 
 --b

---

From: Steve Deering <[email-masked]>
Date: Fri, 17 Sep 1999 11:33:00 -0700

Ben,

Steve

---

From: Ben Black <[email-masked]>
Date: Fri, 17 Sep 1999 11:57:44 -0700

I was suggesting use of the existing mechanism as an alternative to
using the IPv6 aggregation architecture currently specified.  There
is no relation to issues of dictating inbound traffic.

On Fri, Sep 17, 1999 at 02:55:04PM -0400, Jessica Yu wrote:
> Ben,
> 
> If I understand Christian's message correctly, he is talking about 
> possible mechanisms for an end user site to dictate the path for incomming
> traffic.  Not that I am against using the current multihoming mechanism for
> IPv6 (I am really not) but I completely fail to see the relevance of your 
> comment below to his, unless you still think that a site is able to dictate 
> the path for its incomming traffic which is really a fantacy. 
> 
> Look at the following example:
> 
> 		                  x
> 			          |
> 				ISP-F ----------ISP-G 
> 			         |     \       	  | 
> 				ISP-A - ISP-B    ISP-H 
> 				 |        |       |
> 				ISP-C - ISP-D - ISP-E
> 					  \      /
> 					   site-1
> 					     |
> 					     y	
> 					 
> 
> x sends traffic y of site-1, site-1 may wish the traffic go through path
> ISP-F - ISP-B - ISP-D.  But with current mechanism, ISP-F and ISPs along
> the way will choose whatever next hot it think is appropriate to forward 
> traffic.  At every hop, ISP will just consider it's own policy with its own
> interests to choose the next hop for the traffic.  Site-1 do not get
> any say nor ISP-F, ISP-B ... would care about what site-1 wants since it
> is not even their own direct customer. 
> 
> 							--Jessica  
> 
> Date:    Fri, 17 Sep 1999 11:10:06 PDT
> To:      Christian Huitema <[email-masked]>
> cc:      Jessica Yu <[email-masked]>, [email-masked]
> 
> From:    Ben Black <[email-masked]>
> Subject: Re: a response to "A Simple Solution for IPv6 Multihoming"
> 
> Return-Path: [email-masked] 
> References: <[email-masked]> <4.1.19990917082246.00965
>      [email-masked]>
> Mime-Version: 1.0
> X-Mailer: Mutt 0.95.3i
> In-Reply-To: <[email-masked]>; from Christian 
>      ***Huitema on Fri, Sep 17, 1999 at 08:30:52AM -0400
> Content-Type: text/plain; charset=us-ascii
> Content-Length: 1500
> 
>  --b^C

-- 
 --b

---

From: Jessica Yu <[email-masked]>
Date: Fri, 17 Sep 1999 14:55:04 -0400

Ben,

		                  x
			          |
				ISP-F ----------ISP-G 
			         |     \       	  | 
				ISP-A - ISP-B    ISP-H 
				 |        |       |
				ISP-C - ISP-D - ISP-E
					  \      /
					   site-1
					     |
					     y	

							--Jessica  

Content-Length: 1500

-- 
 --b^C

---

From: Steve Deering <[email-masked]>
Date: Fri, 17 Sep 1999 14:29:12 -0700

At 11:35 AM -0700 9/17/99, Ben Black wrote:
>the current growth rate would indicate

(a) I don't know what the current growth rate is, or how much latent
    demand has been suppressed by the current difficulties in getting
    a new prefix advertised globally.  Do you have any data on that?

(b) Even if the current growth rate in multihoming is low, I think it
     would be foolish to bet on that continuing to be the case.

(c) A number of ISP people have said many times that IPv6 had better
    *reduce* the size of the routing tables in the default-free zone,
    because what you call the "current IPv4 multihoming solution" is
    just barely working at the present scale.

> and 2) that the networking hardware will not be up to the task by then?

Workable flat routing to many millions of sites in an Internet built
by multiple, competing ISPs will require something other than just
Moore's Law improvements in hardware.

Steve

---

From: Ben Black <[email-masked]>
Date: Fri, 17 Sep 1999 15:07:43 -0700

>      would be foolish to bet on that continuing to be the case.

However, it would be equally foolish to expend a large amount of effort
designing and implementing the current IPv6 aggregation mechanism.  What
is needed for realistic design is a body of information addressing (a)
above.  I know this information exists, but is also extremely
proprietary.

>     just barely working at the present scale.

I suspect this information is either old or provided by ISP people who
need to upgrade to current router platforms.  An AGS+ is just not a core
router, but an RSP4 can certainly handle at least 250k routes (the
limiting factor is not memory, but CPU).  Other manufacturers are using
higher performance CPUs and that trend will only increase.  I am
confident that routers capable of handling over 1 million routes will
ship within 12 months.  By the time IPv6 could even be deployed with
the number of routes in the current IPv4 backbone, I would expect at
least 2 million routes could be handled, if not more.

> Moore's Law improvements in hardware.

Perhaps on the management side, but definitely not on the hardware side.
The "something" that would be required to handle millions of routes is
an efficient automated management system.  I think it is naive to think
we can constrain routing table growth just because we want to avoid
figuring out how to manage the network.  This thread alone has shown
that there are numerous problems associated with putting constraints
in place, particularly whent he reasons for those constraints are not
supported by facts.

-- 
 --b

---

From: Steve Deering <[email-masked]>
Date: Fri, 17 Sep 1999 16:03:24 -0700

At 3:07 PM -0700 9/17/99, Ben Black wrote:
>designing and implementing the current IPv6 aggregation mechanism.

The IPv6 aggregation mechanism is just CIDR with bigger addresses.
CIDR was designed and implemented years ago.  No particular additional
effort is required, other than the effort of responding to email
like yours :-).

>need to upgrade to current router platforms.

Your suspicions are wrong, on both counts.

>                                          This thread alone has shown

There's that misunderstanding again.  IPv6 places no constraints on the
amount of route de-aggregation.  We are doing prudent address allocation
that enables the degree of aggregation that works well with the current
state of routing technology (hardware, policy, people, etc.)  However,
any ISP or group of ISPs that are able and willing to do less aggregated
routing are free to do so.  I suspect and hope that will be the direction
of evolution of Internet routing, but I am unwilling to bet the farm
on it.

Steve

---

From: Steve Deering <[email-masked]>
Date: Fri, 17 Sep 1999 16:21:00 -0700

You:  "Is it realistic to assume...millions of multihomed sites sooner
      than the current growth rate would indicate...?"

Me:   "I don't know what the current growth rate is, or how much latent
      demand has been suppressed by the current difficulties in getting
      a new prefix advertised globally.  Do you have any data on that?"

You:  "I know this information exists, but is also extremely proprietary."

In other words, you don't know either.  That makes your initial rhetorical
question specious.

Steve

---

From: Ben Black <[email-masked]>
Date: Fri, 17 Sep 1999 16:32:38 -0700

> In other words, you don't know either.  That makes your initial rhetorical

That is one interpretation.

-- 
 --b

---

From: Vijay Gill <[email-masked]>
Date: Fri, 17 Sep 1999 20:09:50 -0400 (EDT)

> Your suspicions are wrong, on both counts.

The IBGP route table of some promising local isps is substantially larger
than the global routing table. Thus ISP's that can barely handle the
current default free routing table need to buy faster boxes.  The critical
path is not amount of RAM, but rather the amount of CPU needed to crunch
the routing tables and disseminate the information to peers.

Given a linear (albeit steeper) curve, Moore's law still outstrips prefix
growth.

Another issue to consider is speed at which you can do lookups for
forwarding.  Using 256-way mtries allows you to do a (current) longest
prefix match using 4 memory cycles.  OC-192c lookups and forwarding at
wirespeed for small packets is doable today. With v6, this issue becomes
harder. It has been proven in the backbone of promising local isp's that
current traffic patterns are not ameneable to caching.  The argument for
having a few entries in routing tables is that they result in a smaller
forwarding table, allowing implementation tricks and very fast ram to do
lookups and perhaps cache a certain amount of information with less churn
than in the v4 forwarding tables, allowing for wirespeed forwarding at
192c and better. 

However, one could argue that the optics and electronics to drive the
interfaces comprise most of the cost of a high speed forwarding board and
that the saving on forwarding engines is thus noise. 

> of evolution of Internet routing, but I am unwilling to bet the farm

Given that multiple circuits to the enduser is still expensive, it is
perhaps overly pessimistic to assume a massive growth in the number of
routing table entries from multihoming punching through aggregated blocks.
The number of routes resulting from applying the current CIDR/mhoming
techniques to v6 will scale with O(# of multihomed organizations) rather
than O(# of NSP's).  Planning to engineer for users multihoming
cellphone/set top boxes may be overkill. 

Given we have an linear growth today, it would perhaps seem wise to let
the market speak on this matter and see how things shape up.  Waiting to
see what technologies and techniques are available at the time this starts
to be a problem and then applying those techniques to the issue at hand
might seem to be a good approach? 

/vijay

---

From: Steve Deering <[email-masked]>
Date: Fri, 17 Sep 1999 18:20:06 -0700

>the routing tables and disseminate the information to peers.

I seem to recall (but this may be wrong, or no longer true) that another
gating issue was the need for a human in each ISP to make a decision
whenever a new prefix is introduced into the DFZ, to decide whether or
not they are willing to offer transit service to traffic destined to
the new prefix, and to record that decision in a policy database.  This
would presumably not apply to intra-ISP-only prefixes.

>routing table entries from multihoming punching through aggregated blocks.

Or perhaps it is overly optimistic to assume that circuits will remain
expensive enough to discourage major growth in multihoming?

>cellphone/set top boxes may be overkill. 

I think the consequences of over-engineering in this case are much less
severe than the consequences of under-engineering.  And we certainly have
a history of under-predicting, rather than over-predicting, Internet growth.

>the market speak on this matter and see how things shape up.

If the growth is only linear today, that may be largely due to the
current IP address rationing policies, prefix-length filtering policies,
and the costs and complexities for a customer to join the unaggregated-
prefix club.  Therefore, it seems unwise to assume the same growth rate
if those barriers were removed.

Anyway, if the market wishes to support large-scale hole-punching for
IPv6 site prefixes, there's nothing to prevent that from happening,
and I would be happy to see it.  However, I think it would be foolish
of us not to be prepared for the case that such a market does not
materialize, or in which the market price of a "hole" is the dominant
disincentive to multihoming.

>Waiting to see what technologies and techniques are available at the time
>this starts to be a problem and then applying those techniques to the
>issue at hand might seem to be a good approach? 

As Bob Hinden pointed out a few days ago, it is much easier to start with
aggregated addresses and then do less aggregated routing on them, than to
start with de-aggregated addresses and try to impose aggregation later.

Steve

---

From: Vijay Gill <[email-masked]>
Date: Fri, 17 Sep 1999 22:53:37 -0400 (EDT)

> would presumably not apply to intra-ISP-only prefixes.

For most ISP's, the scenario goes something like this:

Peering in this case is defined as a settlement free, non-transit
connection.

ISP A has a peering relationship with ISP B C D and buys transit from ISP
P.  Now ISP P is a promising local ISP with strong _prefix_ filtering
applied to data it accepts from it's "customers", ISP A in this case. 

ISP P also has peering relationships with B and D.  For peering circuits,
ISP P applies no filters, relying upon (presumably)  clued B and D to do
the right thing at the edges.

ISP A applies for, and gets a /16 from a registery.  ISP A immediately
sends mail to a ISP P and tells them to allow the new /16 through the
prefix filter P has applied to the BGP session between A and P.  A updates
the RADB with the /16 as well. 

Customer C comes in and buys a T-1 from A.  A allocates a /24 to C,
statically routed.  All is good. C now has global transit. 

C now wants to multihome for purposes of redundancy and gets a T-1 from B
as well.  But since they've already used numbered their machines from the
/24 allocated by A, they tell B that they'll just use that ip block.  B
says ok, sets up a BGP session with C 

Now, Customer C's /24 is reannounced to A and to B both of them annouce
the /24.  P accepts the /24 since it is already covered by the /16 in the
filter for C.  B accepts it because it has a peering relationship with A.

Repeat till you run out of /16.

There is only one human step in the way and assuming P has setup an auto
filter generator (with sanity checking), even that step is now just a
mouse click away. 

> expensive enough to discourage major growth in multihoming?

I don't see many people multihoming their houses with a DSL link to
provider A and a cable link to provider B.  It is fairly time intensive to
setup and configure on part of all parties involved.  Also, most people
don't have dual phone lines for redundancy, but rather for convenience.
And phonelines have lifeline services. I just don't see multihoming every
maytag and subzero unit.

> >techniques to v6 will scale with O(# of multihomed organizations)
> rather >than O(# of NSP's).  Planning to engineer for users multihoming

Growth will slow down as the market saturates. The upper steady state
bound is the GDP of the nation.  _Bandwidth_ demand is separate from this
issue.

> if those barriers were removed.

Why should some of those policies be changed for v6 when they are in place
for v4? Specially things like prefix-length filtering, prefix filtering
and ASN requirements? These things serve as a small buffer against things
like 7007 and their ilk.

> IPv6 site prefixes, there's nothing to prevent that from happening, and
> I would be happy to see it.  However, I think it would be foolish of us
> not to be prepared for the case that such a market does not materialize,
> or in which the market price of a "hole" is the dominant disincentive to
> multihoming. 

The market is driven by money. ISP's will go out of business if they
cannot support the routing table growth and as such, there is a strong
motivator to keep things under control.  It is completely clear from
discussions here that there is not a clue as to how to make multihoming in
V6 work and be consistent with the desire for super-aggregation.  Maybe
some more operational experience with v6 will result in a method that
works. Maybe the problem is intractable.

> start with de-aggregated addresses and try to impose aggregation later.

This is would be consistent with the post-CIDR allocation policies applied
today.

/vijay

---

From: Daniel Hagerty <[email-masked]>
Date: Sat, 18 Sep 1999 03:43:29 -0400

> And phonelines have lifeline services. I just don't see multihoming every

    You must be living on a different internet than I do, or at least
do different things with it.

    The current, IPv4 based internet has a number of limitations for
what one sees as the ultimate goal of it.  Let's not worry about all
this fancy talk about aggregation and algorithm run times, yada, yada,
yada.  What's the idea here again?

    The variety of applications (with different needs of the
underlying transport) that people hope to haul across this bearer
service is always growing.  The unachievable goal is zero delay and
infinite end-to-end potentially bidirectional bandwidth through the
network.

    The customer is expecting the technology to work with the
reliability of a telephone, if not better.  If a person calls "the
data-services company" and the answer to "it doesn't work" is a
hurricane (current events for US East), they'll often accept that.
They won't take much short of that.

    In short, the end user only cares about the application.  They
don't care about the network connecting them; it's supposed to
be "invisible".

<INSERT "Topics in Large Scale Internet Routing; routing policies as
reflected in BGP4 path selection, route asymmetry, failures, etc,
etc">

The result is that end to end quality of service for a given
application can be quite variable over any time granularity.

    Personally, I fit the description of people you don't see many of.
I spend a lot of time typing on distant computers using this internet
thang.  Trust me.  It can suck, and I'm far from the only one who
thinks so.  This application has very low tolerance to delay, due to
the human at the local keyboard & display, engaged in a tight feedback
loop with the distant machine.

    Life as an end customer of an IPv4 internet provider in 1999,
interested in the this application "typing" can be damned frustrating
at times.  It often consists of suffering from pessimal routing
policies of different ASes in relation to the actual available
connectivity and the service needs of my application's traffic.

    For US$90 a month, one can get connectivity over two physical
plants to two different, high-Kwality ISPs, and getting there isn't
*too* much pain to setup. (as long as you act clueless and temporarily
run win98 for the cable modem guys... ;-)

    Being in this position I can select the provider with a better
path for what I'm doing at the moment.  I know that I'm not the only
one doing this sort of thing.

    Other than knowing how to deal with two interfaces, and how to
pick and choose routing, and grok random failures due to poor source
address selection, it can be setup as a windows box and do
small-values of reasonable with routing.  More reasonable things can
do even more.

    You will note that the problems experienced are all end host
issues; the providers have no knowledge of one another.

    Since that the only way that seems to work to provide high
reliability in the backend tends to involve two of the object, what
makes you think the local-loop and the terminating provider are any
different?

    As you pointed out, dealing with most of the issues with a
multihomed customer can be automated to a few mouse click based
decisions by the human (given development), and auto-provisioned from
there.

    So can we stop beating this dead horse?  There are a number of
much harder issues that go hand in hand with the technology; TCP
session address renumbering, multihomed *host* address selection,
provider network management issues and how it relates to customer
needs, etc.

- Just another guy in the trenches

---

From: Francis Dupont <[email-masked]>
Date: Sun, 19 Sep 1999 19:26:23 +0200

In your previous mail you wrote:

   One may argue however that there are two missing pieces in the puzzle:

=> A6 RRs are supposed to help us there but we have no experience with them
and in the BIND worker list the discussion is about the idea to enforce
minimum RR TTLs in order to save money with dialup connections (with
short TTLs DNS caches are less effecient and DNS traffic increases).
(Perhaps DNS is again not the good tool for a solution to this problem?)
 This is an optimization issue (because with a non-brain damaged
application enough addresses will be tried) which should be discussed
at the interim meeting (long term mechanisms?).

   second problem.

=> I have tested this, it should be usable when mobile IPv6 implementations
will use IPSec and correspondent node code will be in every IPv6 stacks
(this should happen!)

Regards

[email-masked]

---

From: Francis Dupont <[email-masked]>
Date: Sun, 19 Sep 1999 19:44:50 +0200

=> there are two solutions:
 - build the Security Association at the connection establishment
   (ie before a path failure). This has no cost if the connection
   uses a security service, for instance in order to avoid TCP reset
   from the bad guy.
 - build the Security Association when needed, ie after the path failure
   and before sending binding updates. In this case an address in a right
   prefix (the care-of address) should be used as the source.
   Multi-homing is a favourable case because this address is a regular,
   valid and long term address of the node (in real mobility care-of
   addresses will change at each movement and become invalid, ie packets
   to an old care-of address can jump in a dark hole).
We discussed last week about security implications of IPv6 mobility
at the IPv6 mobility test week (where no security was used unfortunately).
The only requirement which should be added in the draft is a SA with
a mobile node as the source SHOULD use wildcard source as a selector.
Of course this is a constraint for security policies...

Regards

[email-masked]

---

From: Francis Dupont <[email-masked]>
Date: Sun, 19 Sep 1999 19:58:49 +0200

My main concern about solutions based on RFC 2260 is this is
a solution only for the link failure case (there are many other
problems like management cost of tunnels, MTU, ... but this one
directly makes the RFC 2260 scheme unusable for me).

In France almost all ISPs use a so-called "subscriber equipment"
which is a router managed by the ISP at your end of the link.
Typically you get in your site:

+--------+              +--------+
|  your  |              |   ISP  |
| router |              | router |<===== link to ISP ====//===>
+--------+              +--------+
    ^                        ^
    |                        |
    V                        V
  ==============================
dedicated LAN (Ethernet, FDDI, ...)

RFC 2260 is applicable only if the ISP would like to setup the tunnel
from its router (the "subscribe equipment") and of course they don't
like this! I can send a list of E-mail addresses of French ISP if
you need to enrich your French...
The issue is not the same for the other end of the tunnel because
the ISP can believe (if you don't speak too much about RFC 2260)
it is the primary ISP.

Regards

[email-masked]

---

From: Francis Dupont <[email-masked]>
Date: Sun, 19 Sep 1999 20:10:08 +0200

In the end2end mailing list Vernon Schryver wrote:

   >From what I've seen, assymetric routes are far more common in LANs
   >and corporate campus networks than on the Internet.  They come from the
   >the prudent provisioning of redundant routers.  Dual-homing a single
   >Ethernet within a campus is cheap and painless, unlike dual-homing
   >a network on the Internet.

=> if multi-homing inside a site is very common this can change many
things because main issues with the multi-ISP multi-homing are not
technical (cost, ISP ill will, ...). For instance routing based strategy
(like Jessica's one) should regain our interest.
 I work in a nearly academic environment (with no backup :-) then
I don't know if this is an important case?

Regards

[email-masked]

---

From: [email-masked] (Niels Möller)
Date: 25 Sep 1999 16:04:38 +0200

There is one thing about the authentication needed to avoid making
things like tcp-hijacking trivial, which I've been thinking about for
a few days. Please correct me if the analysis is somehow flawed.

>    uses a security service, for instance in order to avoid TCP reset

If you only want protection against tcp-hijacking, I suspect that it
is good enough to use something like anonymous diffie-hellman to
create a key that is used to authenticate any packet that requests
subsequent tcp packets to be sent to (or accepted from, if any
filtering based on source address is done) a new address or location.

The reason is quite simple: Any attacker that can perform an active
man in the middle attack in this handshake could do arbitrary evil
things with the tcp session *without* hijacking it.

The important thing here: Assume we want things like tcp to survive
renumbering (using mobile-ip, or gse-like addresses, whatever), and we
think that the (in)security of ordinary tcp is good enough for the
application. Then we can get comparable level of (in)security, and
survivability, *without* introducing any key management
infrastructure, preshared secrets or certificates. 

I also think it would be a good thing to clarify exactly what level of
security is wanted for (in)secure traffic. This does seem like a
strange thing to be talking about, but I suspect it is important. It's
definitely one of the main points of the esd-analysis. Perhaps it is
good to be a little more concrete, so lets take an example of an
reasonable application using insecure tcp:

Say I'm using ssh to connect from a local machine (which might be
mobile, or renumbered from time to time). I don't need any extra
authentication for the _data_ in the tcp session (as sufficient
authentication is provided by the application). Connections can live
for a long time, and I'd like them to survive things like site
renumbering, temporary ISP-failures of a multihomed site, i.e. all the
scenarios that have been discussed here.

What level of security is it reasonable to expect from the network and
transport layers? As far as I can tell, we'd like it to at least not
be trivial for an attacker to cause the tcp session to die. Is that
true for ipv4? And is it correct that simple "anonymous"
authentication as described above provides similar (or better)
protection also in the presens of things like binding-update?

/Niels

---

From: Francis Dupont <[email-masked]>
Date: Tue, 28 Sep 1999 05:09:59 +0200

=> I believe you should simply use IPSec which provides the security
services/levels you'd like to get. Mobility/bindings can add a major
security problem then authentication and replay prevention are
mandatory for some binding options.

=> I think it is very/too easy to do arbitrary evil things with TCP
sessions (then you'd like to protect them and to pay for security
from the beginning).

=> IKE (IPSec SA & key management) gives many possibilities.
Unfortunately it is a new protocol then we have not enough experience
of it (perhaps it should be improved ?). Some of them would like
to fully understand mobility (and now multihoming) implications for IKE...

=> I agree, this should be a good thing...

=> a problem with upper layer security services is they cannot protect
TCP connections against RST attacks for instance. When network layer
security services (aka IPSec for IP) are available they are better.
(This comment does not apply for authenticated E-mail where the
entities are human beings, not hosts or processes, but SSL/TLS or
in most cases SSH should be replaced by IPSec IMHO).

   What level of security is it reasonable to expect from the network and
   transport layers? As far as I can tell, we'd like it to at least not
   be trivial for an attacker to cause the tcp session to die.

=> today this is trivial (just send a RST) for a man in the middle.

   Is that true for ipv4?

=> yes and it is true for IPv6 if you don't use IPSec (which is
mandatory then should be available for IPv6).

   And is it correct that simple "anonymous"
   authentication as described above provides similar (or better)
   protection also in the presens of things like binding-update?
   
=> IPv6 mobility draft mandates some security services at the
network layer (because binding-updates are at this layer).
There is a (draft) standard which provides the service but
it is only referenced (ie. the door is still open).

Regards

[email-masked]

---

From: [email-masked] (Niels Möller)
Date: 29 Sep 1999 19:20:45 +0200

> mandatory for some binding options.

I didn't mention ipsec explicitly, because I haven't read all the
relevant specs. If you can use ipsec in "anonymous" mode, that is
great. Not because anonymous mode provides "real" security, but
because it is trivial to manage, and because it is good enough to stop
the kind of threats (hi-jacking, bogus rst or binding-update messages)
that seem to be show-stoppers for all the session-stability proposals
that I have seen here.

If we happen to have a real, properly authenticated shared key with
the other host, we should of course use that instead. But as usual,
the cryptography and protocols are a lot easier than the management of
keys and policies.

> in most cases SSH should be replaced by IPSec IMHO).

I think it makes sense to use a somewhat weak (MITM-vulnerable, shared
keys shared by several parties, whatever) to get some protection
against things like RST attacks, and use an application level secure
protocol like ssh or tls on top of that. The primary reason is that
with an application like ssh, a user can use his or her own security
and trust model, without a lot of coordination with sysadms. In
particular, the parties that the user is willing to trust may be
somewhat different from the parties the sysadm trusts.

> it is only referenced (ie. the door is still open).

I guess this is analogous to the authentication requirements for ipv6
source routing.

/Niels