Subject: Re: about draft-baker-v6ops-l3-multihoming-analysis-00

From: Fred Baker <[email-masked]>
Date: Tue, 21 Nov 2006 11:59:34 -0800

On Nov 21, 2006, at 10:03 AM, marcelo bagnulo braun wrote:
> First, i think that the draft makes a good job in identifying that  
> not all multihoming sites are the same...

Thanks for your comments. I think you and I very much agree that  
there are different requirements. For example, I think that a Fortune  
100 company probably has the same address-independence requirements  
as an ISP has, while a homeowner changing his IP address when he  
moves from house to house seems very reasonable.

I went through your comments, and concluded that while you and I  
don't fully agree, our disagreements were relatively minor. For  
example, I don't think of renumbering as "costly", as it is something  
my company does from time to time and doesn't find so. But I do think  
of the complexities of remembering and managing all the places where  
one writes down an IP address. Early in the note I found myself  
arguing with you in the same direction as other comments you made  
later in the note.

What this tells me is that we haven't got a very good requirements  
document. The requirements on the table don't describe traffic  
engineering requirements at all, and the requirements in RFC 3582  
have several problems.

What might be helpful (thinking out loud here, not making a  
pronouncement; I wonder what people think) to the community at large  
would be a requirements document or set of documents that discusses  
not "requirements of a multihoming solution" but "addressing and  
routing requirements of an end to end network". It would have to  
discuss traffic engineering, from the perspective of saying what  
requirements were reasonable and solvable and what were not. It would  
have to discuss backbone and access ISPs and their business models,  
the requirements of large companies, the requirements of SOHO  
networks, and would have to at least wonder whether there was a zone  
in between. In the end, it might provide advice on AS number  
assignment among other things.

In the end, there are a small number of solutions that can be looked  
at generally to provide a scalable address management architecture.  
The one we are looking at right now suggests the numbering of ISPs  
and asking smaller edge networks to manage multiple prefixes overlaid  
within them. Looking at the discussion of address selection  
discussion that Arifumi is leading and the parallel discussion of an  
update to BCP 38, I think that has some serious questions. The  
exchange-based addressing model I mention in the draft has some  
issues in the business model and in creating what amounts to another  
kind of overlay network. GSE depends, for scalability, on rewriting  
part of the source address in a datagram, which helps with a number  
of the routing problems but begs the question of how the transport  
associates a request with a response. For example, if I use networks  
1 and 2 and you use networks 3 and 4, and I send a message from 1::me  
to 3::you and get a reply from 4::you to 2::me, how do I identify  
that the datagram I received is a reply to the same session? Doing so  
requires the Endpoint ID to truly be globally unique, which is a tall  
order, or it requires some form of dongle at the transport layer that  
addresses the issue. One could imagine the locator/id separation  
resulting in a mandate for the use of IPSEC integrity mode to inform  
the transport's multiplexor. The one thing that doesn't scale is the  
one thing we are doing now, which is having the RIRs assign prefixes  
to SOHO networks.

What we are trying to do in this is inform that discussion.

---

From: Fred Baker <[email-masked]>
Date: Tue, 21 Nov 2006 12:56:14 -0800

On Nov 21, 2006, at 11:59 AM, Fred Baker wrote:
> It would have to discuss traffic engineering, from the perspective  
> of saying what requirements were reasonable and solvable and what  
> were not.

Let me put hands and feet on that. One of the ISPs involved in the  
IAB workshop mentioned a need to provide for traffic engineering that  
would, among other things, allow one ISP to cause another ISP to load  
balance traffic across multiple intervening ISPs. In other words,  
imagine this scenario:
                           ,-.
                          /   \
                         /ISP B\
              ,-.       /       \       ,-.
             /   +-----+---------+-----+   \
            /     \     \       /     /     \
           /       +-----+-----+-----+       \
   The    ;         :     \   /     ;         :  ISP D
Internet |  ISP A  |      `-'      |  ISP D  |
          |         |               |         |  Customers
          |         |      ,-.      |         |
          :         ;     /   \     :         ;
           \       +-----+-----+-----+       /
            \     /     /       \     \     /
             \   +-----+---------+-----+   /
              `-'       \       /       `-'
                         \ISP C/
                          \   /
                           `-'

The ISP D has access to some number of undersea cables owned by two  
other ISPs (B&C) and needs to get access to "the Internet" (whatever  
that is) beyond the ISP A. The total bandwidth available is larger  
than the total traffic load, but total headroom is perhaps 30%  
(enough to handle the occasional loss of one cable but not two or  
three). For whatever reason, lets presume that a direct customer  
relationship between A and D doesn't work - maybe neither owns the  
cables. To a certain extent, ISP D can accomplish its goals by  
advertising different sets of sub-prefixes on each cable. However,  
the interests of the other ISPs differ from its, which means that  
ISPs B&C might accept its sub-prefixes, convert them to the actually- 
allocated prefixes, and advertise those north, with the effect of  
completely screwing up the attempted traffic engineering.  
Alternatively, ISP A might view ISPs B and C as primary and  
alternate, with the same effect.

I can think of a number of fairly simple solutions for this class of  
problem that can be effected by the ISPs themselves, but it seems to  
me to be borderline intractable as a general case. One ISP-specific  
solution might be for ISP D to write an agreement with ISP A about  
how it uses its links through B and C for traffic destined to D, for  
example, and ISP A might implement it by its internal routing  
regardless of the prefixes that B and C advertise to it. But in the  
general case, I not only have to have two or four ISPs agree, but all  
ISPs agree, and that seems (ahem) dubious. You heard it here first :-)

So we have requirements being posed to us from the ISPs, but I'm not  
sure that all of them are problems that we can solve in the general  
case. We need to tease the ones we can solve apart from those that  
are probably just too hard.

---

From: Fred Baker <[email-masked]>
Date: Wed, 29 Nov 2006 13:59:42 -0800

On Nov 22, 2006, at 7:32 AM, marcelo bagnulo braun wrote:
> El 21/11/2006, a las 20:59, Fred Baker escribió:
>>> First, i think that the draft makes a good job in identifying  
>>> that not all multihoming sites are the same...
>>
>> there are different requirements. For example, I think that a  
>> Fortune 100 company probably has the same address-independence  
>> requirements as an ISP has, while a homeowner changing his IP  
>> address when he moves from house to house seems very reasonable.
>
> and when you change the isp of your home network? would it be  
> acceptable to renumber in this case?

That's a question; it really depends on what changes when I change  
ISPs. I'll observe that I have one ISP, and when @Home went south  
they renumbered my home. It wasn't much of a big deal for me, in part  
because I run a NAT (like everyone else does, I suppose) for IPv4  
traffic and don't have IPv6 service. The one problem that I  
encountered was that the VPN tunnel I use to go to work required that  
my company know what IP address I am coming in from and I had to  
advise them of the new one. The counterpart in an IPv6 world would be  
(see RFC 4192) that they configure my router (RFC 4076) with the new  
prefix, allow an interval for my network to make use of it, advise me  
that they have done so and give me time to adjust DNS and advise my  
company etc, and then withdraw the old prefix. In a PA situation, I  
don't see any reason that this would be inappropriate.

The question is whether PA is appropriate in a multihomed network.  
shim6 says it is, and would have each ISP provide its own prefix. GSE  
says it is, but leaves the hosts effectively ignorant of the prefixes  
assigned (within the enterprise network, the routers only look at the  
subnet number, and the hosts only look at the host identifier, and  
the border routers rewrite the part used by the ISPs). A variation on  
GSE would have the border router ICMP back to the host indicating  
that it should be using a certain "routing goop", and have the host  
resend using that. Metro and PI would simply have a single prefix on  
the edge network in the first place. If the distance and complexity  
questions are those of a typical home (see slides 3 and 4 of ftp:// 
ftpeng.cisco.com/fred/v6ops/Fred_network_paths.pdf for what I think a  
typical home looks like; the slide needs some updating now and in the  
coming years will have various large datarate home appliances added  
to it, but I suspect it's close) I can't imagine any of those having  
real operational problems. The big issue with multiple prefixes that  
I see are related to the hiccups around network changes.

One concern I have with the "/48 is what is assigned to the edge,  
even the PI multihomed edge" is that one-size-fits-all seems an  
uneasy solution. My canonical examples are my home and my company,  
and there is a lot of space in between, but consider my company if  
you will. Cisco has about 50,000 employees including temps and  
contractors, and puts a LAN (if it were IPv6, which it is not, it  
would be a /64) in many of our homes for home offices. Let's take a  
flier and say it put one in each telecommuting home. That would  
consume a /48, and it would have to be global in scope, not ULA. We  
have a lot of labs (a single test rack, used by a single engineer for  
unit or integration testing, might have a dozen LANs in it), a lot of  
buildings, and so on. A company the size of Cisco would find uses for  
another /48 easily just doing that, and for ULAs in test racks. I  
wouldn't be too surprised to find that we actually wanted to run /48s  
by continent or theatre. And while, yes, Cisco is #83 on the Fortune  
100 list, many of the 82 above us on that list dwarf us in size.  
Consider Exxon Mobil, Ford, or General Electric... http:// 
en.wikipedia.org/wiki/List_of_Fortune_500#1-100. It's not obvious to  
me that the assumption that a single 16 bit subnet number used  
throughout the corporate network makes sense; I would think the  
architecture has to allow for several /48s. In other words, I think  
we need to be able to use what GSE calls the "routing goop" within  
the enterprise network, not just in the ISP backbone.

That said, I'm hard pressed to describe anything resembling a home in  
similar terms. In fact, I tend to think that the authorities that  
assign prefixes have rational arguments for considering assignment of  
44, 48, 52, 56, 60, and 64 bit prefixes to be service options that  
might be bundled with other aspects of the service or might be sold  
separately.

> actually i would like to point out that RFC3582 is not a  
> requirements document but it is a _goals_ document (its title is  
> "Goals for IPv6 Site-Multihoming Architectures")

yes. That said, it is the closest thing we have to a requirements  
document, and it is what is thrown in my face as having not been met  
when various solutions are discussed.

> I would also like to point out, that "portability" is not listed as  
> a requirement in RFC3582, so there are absolutelly no  
> considerations about renumbering/porivder lock in/portability in  
> the document

OK, I pulled that in from another sector. Marla reports that many  
edge networks in nanog-etc want address portability, and I have heard  
it from other sources as well. If we can provide a certain level of  
address portability without screwing up the network, there is value  
in it, I think.

>> pronouncement; I wonder what people think) to the community at  
>> large would be a requirements document or set of documents that  
>> discusses not "requirements of a multihoming solution" but  
>> "addressing and routing requirements of an end to end network". It  
>> would have to discuss traffic engineering, from the perspective of  
>> saying what requirements were reasonable and solvable and what  
>> were not. It would have to discuss backbone and access ISPs and  
>> their business models,
>
> i think this is very interesting point that you bring here....
>
> Are we considering a multihoming solution for ISPs or for end sites?

I think the discussion has to take into account both ISPs and their  
customers, which might very well mean that it needs to be divided  
into separate discussions. For example, I have been told by some ISPs  
that they want PA addressing because it gives them a market lock, and  
by some of their customers that they don't want PA addressing because  
it gives ISPs a market lock. That kind of divide is a place where a  
single consensus is simply not going to form. That said, my statement  
above says "lets go beyond multihoming; what actually are the set(s)  
of requirements for routing and addressing end to end and in the  
middle?"

> I mean i think it is clear that the requirements for a multihoming  
> solution for an ISP and for an end site are likely to differ (or at  
> least it is not obvious to me that they are the same or that the  
> same solution could support both ISP and end site multihoming)
>
> imho, different requiremetns sets should be determined including:
> - small end sites
> - soho end sites
> - large sites
> - ISPs
>
> Probably there need to be different types of ISPs but i don't have  
> enough background for this...

I guess I think of ISPs as being broadly divided into transit  
networks and access networks, although defining those terms crisply  
is going to be pretty tough. We used to talk about Tier 1 networks as  
global, Tier 2 networks as regional, and tertiary networks as being  
local; I tend to think of networks now as providing either  
interconnection between businesses or residential/SOHO access, and  
see FTTH as muddying that puddle.

> imho it would be interesting to at least try to produce these  
> documents and see if it is possible to reach concensous on the  
> requirements for each scenario

I agree. It is likely to be input to the RAM discussion that the IESG  
is in the process of chartering, and may be more appropriately moved  
there at some point. Is there a part of that you would like to  
contribute to?

>> In the end, there are a small number of solutions that can be  
>> looked at generally to provide a scalable address management  
>> architecture.
>
> scalable in which sense?

:-)

in the sense I used the term - or more to the point chose to not use  
the term - in the document referred to in the subject line. Since I  
couldn't find a crisp definition, I went in the direction of trying  
to enumerate the number of prefixes that each algorithm resulted in  
under a consistent set of assumptions that seemed to have some hope  
of being in the general neighborhood of reality. Larger and less firm  
numbers (as in "O(10^7)") are less desirable than smaller numbers (as  
in "O(10^5)") that have obvious derivations.

> so multiaddressing does provide scalability for the routing system,  
> but imposes a new way of doing a bunch of stuff

Each of the solutions on the table changes something. Multi- 
addressing, as you term it, is pretty straightforward if the end  
systems and routers directly support it, so while it is different it  
isn't all that scary to me. GSE is similarly straightforward - I  
implemented roughly the same set of algorithms in an XNS/IPX router  
in the late 1980's. Metro requires no changes to the ISP or customer  
premises equipment - it can be deployed today - but requires  
differences in business relationships. PI is something we are  
deploying today, and if the approach is to be applied to businesses  
with ten or more employees (Vince's slides from the IETF meeting)  
drives the amount of memory in a backbone router into the sky. There  
is some form of change in every one of these, and part of the  
discussion needs to be that for the viability of everybody's networks  
and businesses nobody's cows can be considered sacred. All options  
are on the table and the trade-offs have to be made with as little  
emotion as humanly possible.

>> GSE depends, for scalability, on rewriting part of the source  
>> address in a datagram, which helps with a number of the routing  
>> problems but begs the question of how the transport associates a  
>> request with a response. For example, if I use networks 1 and 2  
>> and you use networks 3 and 4, and I send a message from 1::me to  
>> 3::you and get a reply from 4::you to 2::me, how do I identify  
>> that the datagram I received is a reply to the same session? Doing  
>> so requires the Endpoint ID to truly be globally unique, which is  
>> a tall order, or it requires some form of dongle at the transport  
>> layer that addresses the issue. One could imagine the locator/id  
>> separation resulting in a mandate for the use of IPSEC integrity  
>> mode to inform the transport's multiplexor.
>
> i really don't see how this could work... I mean how would you  
> establish any-to-any IPSec SAs that can prove endpoint ID  
> ownership? I can see two ways of doing this, crypto ids or a third  
> trusted party. In any case, you end up with the problem of how to  
> delegate the capability of rewriting to the routers... and so on.  
> My point is this can be made to work, but it won't be as simple and  
> nice as the current proposal

I was pointing out a problem the current proposal doesn't solve...  
How does, for example, an RFC 3041 address gain global uniqueness? If  
a MAC address is used, how do we ensure global uniqueness for  
addresses that have the "local" flag set? If the endpoint identifier  
isn't globally unique, why do I believe that I can treat it as such  
an identifier without the routing goop that tells me which instance  
I'm looking at?

> what it seems to be an insolvable problem (or at least extremely  
> hard) is to build a solution that has the scalability required for  
> a massive adoption (the one required for a small end site  
> multihoming solution) and provides all the features required by big  
> sites....

Personally, I don't think that is necessary. The obvious way to solve  
a fortune-something-company problem is to treat the fortune  
<something> company as if it were an ISP and give it a PI assignment  
comparable to an ISP. If that is a /32 and the guy only needs a /52,  
so be it. We're talking about a few hundred of these globally, and  
once they have a PI prefix they won't be back for a new prefix for a  
while. I'd like to see the policies of the RIRs say as much. That  
said, for SOHO networks and mid-sized companies (do they have  
different requirements?) that's definitely not the right solution,  
and we need an approach that works both for them and for the ISPs  
that serve them.

---

From: Fred Baker <[email-masked]>
Date: Thu, 30 Nov 2006 12:40:23 -0800

Brian has suggested that much of this discussion belongs on the ram  
list. See

	https://www1.ietf.org/mailman/listinfo/ram

and please join it if you feel it is appropriate for you to do so.

Where IPv6 Operations has a useful role here, I think, is to bring  
out internet drafts and perhaps RFCs that inform that discussion.  
That may include reposting existing internet drafts, and may include  
new ones. The key thing, though, is that the question is not "what  
are the multihoming requirements for the entire Internet", but "what  
are the end-to-end addressing and routing requirements for" what I  
will call (for lack of better terminology, not become I like it all  
that well) "Transit ISPs, Access ISPs, large edge networks, mid-sized  
edge networks, and SOHO and residential networks?" Multihoming is  
part of that, but inter-ISP traffic engineering is also part, and  
there may be other parts. If two groups of people find that they have  
differing requirement sets, the solution is not to force them to come  
to some unrecognizable consensus, but to have them describe the part  
of the Internet they are describing requirements for and then  
accurately cull out those requirements. The classic example of such a  
divide that I mentioned in another note is the idea that some ISPs  
want PA addressing as a market lock, and some edge networks detest PA  
addressing because it is one. One useful note along those lines might  
be a well researched set of expectations of the Internet, including  
each of its various ecological zones, in ten, twenty, and fifty years.

Consider a call for submissions to have been placed.

Note, by the way, that I don't automatically assume that all  
individual submissions are intended or need to become working group  
documents, or that all internet drafts are intended or should become  
RFCs. You will note that I quite happily post internet drafts to pose  
a discussion and happily let them die if the discussion completes and  
they are no longer needed. This isn't a race to get written up in the  
record books. But useful contributions to those discussions will be  
welcomed, and if the WG is of the opinion that they should be  
archived I'm all for it.

---

From: David Meyer <[email-masked]>
Date: Thu, 30 Nov 2006 14:08:08 -0800

> and please join it if you feel it is appropriate for you to do so.

	Thanks Fred.

> Consider a call for submissions to have been placed.

	In addition, there is a workshop report that is in the
	works (we have a -00.txt candidate in hand), so I'm
	hoping that will be available for your questions and
	comment in the next few days. 

	Thnx, and look forward to the discussion.

	--dmm