Subject: [rrg] FW: I-D Action:draft-irtf-rrg-recommendation-06.txt

From: Tony Li <[email-masked]>
Date: Sat, 06 Mar 2010 15:56:33 -0800

Hi all,

This is hopefully the last draft prior to the meeting.  If you need further
changes, we have about 49 hours before the deadline.  Please send email.

No, the recommendation is NOT in the text yet.  Yes, we will be able to make
further modifications to the document after the meeting, but this should
largely be confined to editorial and non-substantive corrections.

Thanks,
Tony

------ Forwarded Message
From: <[email-masked]>
Reply-To: <[email-masked]>
Date: Sat,  6 Mar 2010 15:45:02 -0800 (PST)
To: <[email-masked]>
Subject: I-D Action:draft-irtf-rrg-recommendation-06.txt

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

 Title           : Recommendation for a Routing Architecture
 Author(s)       : T. Li
 Filename        : draft-irtf-rrg-recommendation-06.txt
 Pages           : 65
 Date            : 2010-03-06

It is commonly recognized that the Internet routing and addressing
architecture is facing challenges in scalability, multi-homing, and
inter-domain traffic engineering.  This document surveys many of the
proposals that were brought forward for discussion in this activity,
as well as some of the subsequent analysis.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-irtf-rrg-recommendation-06.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

------ End of Forwarded Message

---

From: Robin Whittle <[email-masked]>
Date: Sun, 07 Mar 2010 13:42:25 +1100

Hi Tony,

Maybe I missed something on the mailing list, but I only just
realised from the agenda of the next meeting:

  http://trac.tools.ietf.org/group/irtf/trac/wiki/RRGagendaAnaheim

that the Recommendation is to be discussed all day then.  In "FW:

> largely be confined to editorial and non-substantive corrections.

Can you explain what happens next?

Is the Recommendation to be discussed on the mailing list between
now and the meeting?  Will it be discussed after the meeting, and
if so for how long?

Do you judge consensus regarding the Recommendation at the meeting
or on the list?

Is there an absolute deadline for finalising the RRG Report?

What happens once it is finalised?  I understand the IESG will
consider it.  Will the question of what to do next - which, if
any, architectures are to be developed in the IETF - be discussed
on the IETF list and/or the RRG list?

I assume the RRG itself will continue, at least in the form of an
email list, until the IESG's decision.

  - Robin

---

From: Robin Whittle <[email-masked]>
Date: Sun, 07 Mar 2010 14:02:44 +1100

Hi Tony,

Thanks for removing "agreed" from the mention of the RRG Design Goals.

Is there any reason for the apparently un-ordered sequence of the
proposals?  They are not alphabetically arranged, or grouped in any way.

On 4 March I sent you a revised critique for Compact Routing, cut to
500 words by moving some non-critical parts of it to Hannu's "rebuttal".
Hannu replied to us he was happy with these, but they are not in the
current draft.

I would appreciate it if you removed (in the References section) the
link to "whittle-ivip-db-fast-push", which is now obsolete.  Please
also remove the link to "Ivip Summary".  I haven't been able to
maintain this properly as a separate document and it is now just a
link to Ivip-arch, which is a good introduction to Ivip, and to
my suggested Recommendation text.

Below my signature is some rfc2xml stuff for two new IDs, most
importantly draft-whittle-ivip-drtm.  If you can make the mention of
draft-whittle-ivip-drtm in the "rebuttal" link to the reference, that
would be great.

  - Robin

	<reference anchor='I-D.whittle-ivip-drtm'>
		<front>
			<title>DRTM - Distributed Real Time Mapping for Ivip and LISP</title>
			<author initials='R' surname='Whittle' fullname='Robin Whittle'>
  					<organization />
			</author>
			<date year='2010' month='March' day='06'  />
		</front>
				
		<seriesInfo name='Internet-Draft' value='draft-whittle-ivip-drtm-01' />
		<format type='TXT' target='http://www.ietf.org/internet-drafts/draft-whittle-ivip-drtm-01.txt' />
	</reference>

	<reference anchor='I-D.whittle-ivip-glossary'>
		<front>
			<title>Glossary of some Ivip and scalable routing terms</title>
			<author initials='R' surname='Whittle' fullname='Robin Whittle'>
  					<organization />
			</author>
			<date year='2010' month='March' day='06'  />
		</front>
		
		<seriesInfo name='Internet-Draft' value='draft-whittle-ivip-glossary-01' />
		<format type='TXT' target='http://www.ietf.org/internet-drafts/draft-whittle-ivip-glossary-01.txt' />
	</reference>

---

From: Tony Li <[email-masked]>
Date: Sat, 06 Mar 2010 21:32:02 -0800

Hi Robin,

> that the Recommendation is to be discussed all day then.

As always, the agenda will not be final until the meeting starts.  And
sometimes not even then.  ;-)

> Can you explain what happens next?

No.  ;-) ;-) ;-)

What I would like to have happen next is nothing but fantasy.  What I expect
will happen will look like:

    - Some initial discussion
    - Presentation of the recommendation
    - Feedback afterwards
 
> now and the meeting?  Will it be discussed after the meeting, and

Yes, folks are welcome to discuss the recommendation on the mailing list
between now and the meeting.

Yes, it will undoubtedly be discussed after the meeting, probably for
several years.  ;-)
 
> Do you judge consensus regarding the Recommendation at the meeting

Given the lack of consensus that we have seen so far, there's not going to
even be a call for consensus.

> Is there an absolute deadline for finalising the RRG Report?

Not that I know of.  Of course, if we take too long, the IRTF Chair may step
in.
 
> email list, until the IESG's decision.

The IESG is free to do whatever it likes with the recommendation.  My guess
is that there will be cursing involved.  ;-)   Eventually however, I would
expect that the IESG would open a WG in the routing area to pursue this.
Where the discussions occur, if any, is up to the IESG.

I would expect that the RRG would continue on and address other routing
research issues.  The RRG has been a standing group for some time and has
looked at numerous issues.  This is only the latest work item.  Other
obvious things that may be taken up: prevention of micro-loops, multi-party
routing in mobile networks, etc.  I would expect that the next work item
would be under new leadership, more appropriate to the subject matter.

Tony

---

From: Tony Li <[email-masked]>
Date: Sat, 06 Mar 2010 21:46:03 -0800

Hi Robin,

> proposals?  They are not alphabetically arranged, or grouped in any way.

The are arranged in the order that I processed them.  This is roughly the
order that they appeared in my mailbox.

> Hannu replied to us he was happy with these, but they are not in the

Sorry, but I don't think I received it.  Digging through my emails
(including the deleted folder) doesn't show it.  Please re-xmit.

> my suggested Recommendation text.

Done.  Please see the next rev., whenever that issues.

> draft-whittle-ivip-drtm in the "rebuttal" link to the reference, that

I'm happy to add the references, but if you want to change the text, please
send explicit changes or new text.

Thanks,
Tony

---

From: Robin Whittle <[email-masked]>
Date: Sun, 07 Mar 2010 20:24:39 +1100

Hi Tony,

Thanks for your reply.  I resent you the Compact Routing critique and
"rebuttal".

> send explicit changes or new text.

OK.  At the end of the first paragraph of the "rebuttal" you have:

  (draft-whittle-ivip-drtm-00)

If you have added the reference material I sent in the previous message,
then please replace this with the xml2rfc incantation:

  <xref target="I-D.whittle-ivip-drtm" />

  - Robin

---

From: Robin Whittle <[email-masked]>
Date: Sun, 07 Mar 2010 21:13:48 +1100

Short version:   Surely we should aim to do better than assume there can
                 be no consensus.

                 If there was a statement which adequately represented
                 the goals, beliefs and arguments for these beliefs of
                 the major bodies of opinion amongst RRG participants,
                 then we could reach consensus that this was a
                 reasonably accurate statement - and recommend the IESG
                 consider these viewpoints.

                 As a start to this process, perhaps Tony or others could
                 write why CEE and Locator / Identifier Separation in
                 general, or ILNP (or another CEE architecture) is
                 the best choice over any CES architecture.  I think
                 this would include reasons why the extra burdens on
                 all hosts and the generally increased delays in
                 establishing communications would be justified by not
                 having to add so much complexity to the routing system
                 as a CES architecture would.

Hi Tony,

>     - Presentation of the recommendation

OK - I understand from this you have no clear idea of what will happen.

> Yes, it will undoubtedly be discussed after the meeting, probably for

OK - but do you have a plan for when you will finalise the RRG Report?
Discussions after that are a different thing from those before you and
Lixia finalising it.

> even be a call for consensus.

Then how could you be sure that no consensus existed?

> Not that I know of.  Of course, if we take too long, the IRTF Chair may step

OK.

> Where the discussions occur, if any, is up to the IESG.

OK.

> would be under new leadership, more appropriate to the subject matter.

OK.

It seems a curious plan to me - to assume there is no consensus, to
assume the meeting will somehow produce a "Recommendation" and then that
the meeting will debate it, but that there will be no test of consensus
on the list or in the meeting.

Surely there's a role for the co-chairs in trying to establish what
things people agree on and disagree on, and how the opinions of the
various individuals can reasonably be grouped, or at least described.

I think there would be little point in developing a Recommendation
which simply reflected the views of a subset of people, when there were
multiple subsets with different, reasonably coherent views - since
there's no way of choosing which subset's views to use for the
Recommendation except by quite arbitrary means.

Even if we are highly divided, I imagine we could reach consensus on a
Recommendation that the IESG consider, for instance, the three major
bodies of opinion, which are irreconcilable.  Then go on to state
common elements, points of difference and agreement - and include
three independent recommendations each group would like to make, with
reasons for each, with references.  Only don't try to make this stuff
fit in 500 words!

    (If anyone thinks 10k words is too long for a Recommendation,
     consider that there are probably 10 times this number of DFZ
     routers affected by the scalable routing problem and its
     potential solutions, and that each one is probably worth $100k
     or so.  That's $1M a word.  Or consider that for every word
     there are 100,000 Internet users or whatever.)

Then we could probably reach consensus that this statement was a
reasonable summary of the major bodies of opinion amongst participating
RRG members.  The Recommendation would not be to adopt a single
architecture, but that the IESG consider the three or whatever types of
approach which are most prominent.  I think this would be a useful
thing to work on and to present to the IESG.

As part of this, perhaps you could write something in favour of CEE and
Loc/ID Separation in general, or in favour of ILNP alone or whatever -
and state why you (and I guess others) believe it is worth changing host
stacks (and apps?) to use this new naming model and so burdening hosts
with more work, more packets, more delays etc. for the purpose of not
having to add so much (or perhaps anything) to the interdomain routing
system (as happens with CES architectures) - and perhaps for other
purposes.  That is for IPv6 only.

Then, if you have any recommendation for IPv4, likewise state what it
is and why it is superior to the alternatives.

At present there is one alternative recommendation for IPv4 and IPv6 -
my msg06162.  But if you write something like what I suggest, then
there would be a second alternative recommendation - and hopefully
other people will do the same and contribute their ideal recommendation
text.

You wrote:

> What I would like to have happen next is nothing but fantasy.

You can keep it a secret fantasy, if you like - which pretty much
ensures it won't happen.

I don't think your position of co-chair should inhibit you from stating
your personal views, hopes, fantasies or whatever.  Indeed I think
being co-chair means you should state these, firstly so everyone knows
what you believe and why - and secondly to set an example for others to
do the same: stating what they believe *and* why this is a better set
of beliefs than the alternatives.

  - Robin

---

From: Tony Li <[email-masked]>
Date: Sun, 07 Mar 2010 11:06:19 -0800

>   <xref target="I-D.whittle-ivip-drtm" />

Done

Thanks,
Tony

---

From: Tony Li <[email-masked]>
Date: Sun, 07 Mar 2010 11:26:43 -0800

Hi Robin,

> Short version:   Surely we should aim to do better than assume there can

That would be nice, but consensus building is a process, not an
instantaneous event.  It begins with debate, to be sure, but generally
involves people compromising to come together to back fewer and fewer
proposals over time.  We've seen none of that.

>> What I expect will happen will look like:
>> 

That's as clear as I can make it.  What were you hoping for?

> Discussions after that are a different thing from those before you and

Hopefully, it's all editorial afterwards.

> Then how could you be sure that no consensus existed?

The discussions that we've had (both publically and privately) have made it
quite clear that everyone is simply going to back their own proposals.

Have you seen folks volunteer to withdraw their proposals and back another?
I can think of only one case.

> on the list or in the meeting.

As I've said before: we would like their to be consensus, but from an IRTF
perspective, it is not a strict requirement.
 
> various individuals can reasonably be grouped, or at least described.

Indeed.  This is, in part, the function of the document.

> Recommendation except by quite arbitrary means.

This is the choice we're left with.  Folks are unwilling to compromise yet
we need to make a choice.
 
> bodies of opinion, which are irreconcilable.

That's a failure to make any decision, which is, itself, a failure.

> thing to work on and to present to the IESG.

We disagree.  Our job is to present the IESG with a path.

Tony

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 08 Mar 2010 09:05:17 +1300

On 2010-03-08 08:26, Tony Li wrote:

> ...Our job is to present the IESG with a path.

Only one path? As far as I can see, the arguments are rather
evenly balanced, and some of the choices are complementary,
or at least could co-exist.

   Brian

---

From: Tony Li <[email-masked]>
Date: Sun, 07 Mar 2010 12:51:03 -0800

> or at least could co-exist.

Yes, a path.  Not necessarily a single proposal, but a clear plan.  From the
charter:

    The group will produce ... a recommendation for a routing and addressing
    architecture.

It doesn't say "a set of possible architectures" or "some vague things that
we could do".  

Again, our job is to make a decision and that's what we're going to do.

Tony

---

From: Robin Whittle <[email-masked]>
Date: Mon, 08 Mar 2010 12:03:24 +1100

Hi Tony,

In "Re: [rrg] Recommendation and what happens next" (msg06184) you
didn't respond to the latter part of my msg06181 where I suggested
you and others who support the adoption of a Locator / Identifier
Separation naming structure should write a detailed argument about
why the burden this places on all hosts is justified by savings in
the routing system.  (LISP is misnamed - it does not involve Loc/ID
Separation.)

I am CCing Ran Atkinson, but this isn't just for you and Ran.  It is
for all the people who believe that the RRG should recommend
development and later adoption of a Core-Edge Elimination
architecture: GLI-Split, ILNP, Name-Based Sockets and RANGI.

I am requesting you write detailed arguments for your positions -
including detailed critiques of arguments against CEE (Locator /
Identifier Separation) and of arguments for CES architectures.

CEE proponents are not the only RRG participants who have failed to
adequately argue their case.  The LISP team have not argued why LISP
is superior to other proposals, contributed critiques of other
proposals or responded to critiques of LISP.

No CEE proponent - except perhaps Lixia, who I think may support CEE
in the long-term future - wrote a critique of any of the CES proposals.

The failure of the proponents of one proposal to write detailed,
constructive, critiques of other proposals has been highlighted as a
problem by k claffy (msg06044) and Eliot Lear (msg06054).

I urge you, Ran and other CEE supporters to argue in detail why your
preferred CEE solution is better than any CES solution.  If you like,
please argue against one particular set of arguments for a particular
CES proposal:

   Recommendation suggestion from RW
   http://www.ietf.org/mail-archive/web/rrg/current/msg06162.html

I can't see what constructive purpose is served by you ignoring
suggestions such as this, which I have made before.  If you have
reasons for not wanting to argue the case for architectural changes
you prefer, then I think you should explain them to the RRG.  You you
are expecting us (msg06184) to debate all these issues and reach
consensus on support for a single proposal - yet you have so far
failed to debate your own preferred position.

There has been essentially no response from CEE proponents to my
previous attempts to debate my argument that the burdens CEE places
on all hosts are not justified by the reduced complexity in the
routing system CEE promises, compared to CES.

   Arguments in favour of Core-Edge Elimination vs. Separation?
   http://www.ietf.org/mail-archive/web/rrg/current/msg05745.html  Jan 22

       Only one CEE proponent responded - Xiaohu Xu (msg05759) and
       he did not argue why the extra burdens on all hosts are
       justified.  He admitted to a problem with CEE (Loc/ID
       Separation): the greatly increased difficulty - or perhaps
       impossibility - of filtering incoming packets based on the
       Identifier (currently an IP address) in the source field.

   Today's "IP addr. = ID = Loc" naming model should be retained
   http://www.ietf.org/mail-archive/web/rrg/current/msg05864.html  Feb 2

   CES & CEE are completely different (graphs)
   http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html  Feb 2

       No replies to either, despite me referring to them in
       subsequent messages.

   Vote [1] Ivip, [2] LISP
   http://www.ietf.org/mail-archive/web/rrg/current/msg06144.html  Feb 26

       Contains critique of all CEE proposals: "Locator / Identifier
       Separation naming model would decrease the speed of session
       establishment and unreasonably burden all hosts with extra
       traffic and responsibilities."

       No replies.

   Recommendation suggestion from RW
   http://www.ietf.org/mail-archive/web/rrg/current/msg06162.html

       Contains more detailed arguments against all CEE proposals.

       No replies.

Here is my suggestion again, in summary and in detail:

As part of this, perhaps you could write something in favour of CEE
and Loc/ID Separation in general, or in favour of ILNP alone or
whatever - and state why you (and I guess others) believe it is worth
changing host stacks (and apps?) to use this new naming model and so
burdening hosts with more work, more packets, more delays etc. for
the purpose of not having to add so much (or perhaps anything) to the
interdomain routing system (as happens with CES architectures) - and
perhaps for other purposes.  That is for IPv6 only.

At present there is one alternative recommendation for IPv4 and IPv6
- my msg06162.  But if you write something like what I suggest, then
there would be a second alternative recommendation - and hopefully
other people will do the same and contribute their ideal
recommendation text.

I don't think your position of co-chair should inhibit you from
stating your personal views, hopes, fantasies or whatever.  Indeed I
think being co-chair means you should state these, firstly so
everyone knows what you believe and why - and secondly to set an
example for others to do the same: stating what they believe *and*
why this is a better set of beliefs than the alternatives.

In addition to stating your arguments in a standalone fashion, I
think it is best to quote and respond to detailed arguments against
your preferred positions, and those in favour of contrary positions.

 - Robin

---

From: Robin Whittle <[email-masked]>
Date: Mon, 08 Mar 2010 13:31:26 +1100

Hi Tony,

> proposals over time.  We've seen none of that.

OK - but where have you encouraged debate, or contributed to it by
arguing for your own preferred position?  I understand you support
ILNP (CEE, Locator / Identity Separation) - which are only practical
for IPv6.  I am not sure what you want for IPv4.

Consensus doesn't involve just compromise - at least in the sense of
people giving up on what they really want, and going along with less
enthusiasm for something they think would be inferior, but not as bad
as other alternatives.  It also involves people changing their minds
about what is practical and desirable - including recognising their
own proposal is not suitable for IETF development or for the
long-term future of the Net.

Quite a few people who contributed proposals to the RRG have
indicated they recognise their proposal is not sufficiently developed
to be considered for IETF development - I think this is true of the
mapping-only proposals and the CES proposal TIDR.  So they may well
be happy to support some other proposal.

Some previously active proposals were not submitted for this phase of
RRG consideration:

    Eliot Lear (msg05274) retired NERD from active development.

    APT was retired because the designers couldn't see how ISPs would
    be motivated to make the initial investment.

    Bill Herrin didn't develop TRRP any more or submit it for RRG
    consideration.

    Christian Vogt dropped Six/One Router and developed Name Based
    Sockets.  I don't know whether he still wants Name Based Sockets
    considered for IETF development, because he hasn't written to the
    RRG since 13 January.

> That's as clear as I can make it.  What were you hoping for?

Since you are one of the co-chairs and you are urging us - indeed
expecting us - by so-far unspecified methods, to develop and in some
way be happy with a single-architecture Recommendation in the next 19
days, I was hoping you would be able to give us more guidance about
how to debate this, including leading by example: by arguing in
detail for your own preferred outcomes.  Lixia has done this for her
AIS (Evolution) proposal.

> Hopefully, it's all editorial afterwards.

You haven't yet stated when you and Lixia intend to finalise the text
of the RRG Report, at which point you will choose what, if anything,
goes in the Recommendation section.  Nor have you explained how you
will choose the text, or guide the mailing list or meeting discussions.

Do you intend to finalise the Recommendation during the meeting, or
afterwards?  If the former, than I am concerned that those not at the
meeting will be at a disadvantage through not being able to be
express our arguments as directly as those who can attend.  If the
latter, do you have any plans for how long after the meeting you will
finalise the text?

> quite clear that everyone is simply going to back their own proposals.

This is a statement about the past.  If this continues for the next
few weeks, then there will be no large enough subset of active RRG
participants with a compatible opinion to form anything like
consensus.  But I doubt that everyone would hold to their own
proposals in the context of a detailed, constructive, debate - a
debate we are yet to have.

We already know that some of the proposals are not real proposals
which could be considered for IETF development - those which just
concern mapping systems:

   Compact routing in locator identifier mapping system
   Layered mapping system (LMS)
   2-phased mapping
   Enhanced Efficiency of Mapping Distribution Protocols in
   Map-and-Encap Schemes

There are three other proposals which are neither CEE or CES:

   hIPv4
   Name overlay (NOL)
   Evolution - Aggregation with Increasing Scopes (AIS)

hIPv4 involve changes to hosts and DFZ routers.  (I don't recall
whether it requires changes to applications).  NOL involves changes
to some routers and to applications.  There's no way these can
compete with CES proposals which involve no host stack or application
changes.

I am yet to read the response to my critique of AIS, but I can't see
how it could cope with the reasonable goals of portability,
multihoming and inbound TE for 10^7 non-mobile networks, much less
mobility for 10^9 to 10^10 MNs.

There are four CEE proposals:

   GLI-Split
   ILNP - Identifier-Locator Network Protocol
   Name-Based Sockets
   RANGI

These are only for IPv6 and are all subject to the critique that they
burden all hosts with extra delays and packets.  Also, they require
all hosts to be changed before there are substantial benefits to
adopters or to scalable routing.

None of the proponents of these architectures have argued why their
proposal should be accepted for development and deployment ahead of
any or all of the CES architectures.  CES works for both IPv4 and
IPv6, involves no host stack or application changes, and (at least
with Ivip and LISP) provides full benefits to adopters irrespective
of the level of adoption, and scalable routing benefits in proportion
to their level of adoption.

That leaves four CES architectures:

   IRON-RANGER
   Ivip
   LISP
   TIDR

TIDR can't solve the routing scaling problem because its "mapping" is
almost identical to advertising individual end-user network prefixes
in the DFZ.  I have argued that IRON-RANGER is not yet sufficiently
developed to understand its security and scaling challenges - and
there's no apparent method of supporting packets sent from
non-upgraded networks.  As far as I know, there have been no
substantial counter-arguments to these critiques.

There is a coherent argument in my msg06162 why only LISP and Ivip
could be considered for IETF development, why the distinguishing
architectural elements of Ivip are superior to those of LISP and why
Ivip or similar should be preferred over all other proposals for IETF
development.

No-one, including the LISP folks or any CEE proponent, has responded
to this yet.  Any constructive debate will involve people reading
opposing views and responding to them in detail.

Your statement:

> The discussions that we've had (both publicly and privately) have
> made it quite clear that everyone is simply going to back their own
> proposals.

may be true of the recent past.  But surely it is your task - a
difficult and perhaps thankless one - to improve on this pattern of
people turning their back on real debate, ignoring other people's
proposals and simply restating their positions, without properly
engaging in debate about why their proposal is better than all the
others.

> I can think of only one case.

Sure - but I am talking about the future, between now and whenever
you finalise the RRG's Recommendation.

> perspective, it is not a strict requirement.

OK - but if you and Lixia include text as a "Recommendation" without
a process which tests support for it - or if there is such a test and
it only gets minority support - then the "Recommendation" won't have
much value or be taken particularly seriously by the IESG or anyone else.

> Indeed.  This is, in part, the function of the document.

I don't recall you or Lixia write anything about what various groups
of people have in common and where they differ.

Classifying the proposals into groups is a way of starting this
process.  I did this in msg06162 and neither of you have commented on
this.

> we need to make a choice.

Your statement is about the past and you seem to have no hope or plan
for improving on this currently bleak situation.  In a recent message:

  Why won't supporters of Loc/ID Separation (CEE) argue their case?
  http://www.ietf.org/mail-archive/web/rrg/current/msg06187.html

I suggested how you, as co-chair, could lead by example in arguing in
detail for the position you support - including arguing in detail
against the arguments (such as msg06162) for opposing positions.

> That's a failure to make any decision, which is, itself, a failure.

Sure - but if that is the best we can do, then surely it would be
better to produce a coherent report on the various positions which
people have adopted, with the arguments each group advances in
support of their position, and the arguments they provide against the
positions of others.

To pretend that the situation involved more agreement than it really
does would be a worse failure.

Also, I think it would be a failure if this situation of isolated,
disengaged, groups of people or individuals persisted due to lack of
hope or guidance from the co-chairs about how to constructively
debate the issues so that people might change their mind, or at least
articulate the reasons for their positions better than they have so far.

> We disagree.  Our job is to present the IESG with a path.

You seem to think the only useful thing the RRG can do is present the
IESG with a single chosen architecture to develop.

You seem to believe that it it will be impossible to get consensus
agreement to any such single architecture recommendation.

Yet you intend to produce such as single architecture recommendation
- and expect RRG members to be happy with this.

You have no apparent plan - and do not lead by example - for debating
the issues, trying to identify commonalities and differences etc.

You have even stated that you don't plan to test for consensus.

I think that is an extraordinary statement to make.  It hardly
encourages people to engage in the RRG process if you are planning to
choose some text as the RRG Recommendation which doesn't result from
consensus and which you don't even plan to test for consensus support.

  - Robin

---

From: Tony Li <[email-masked]>
Date: Sun, 07 Mar 2010 20:28:24 -0800

Hi Robin,

> Identifier Separation) and of arguments for CES architectures.

I respectfully decline.  Thanks, but I have a day job.

Also, we've been over (and over and over) the points here repeatedly.
Unlike some people, I don't believe that endless restatement of the same
points, infinite volume, or downright stubbornness will prove to be
convincing.  I've plead my case.  Mostly in person, sorry Robin, and it
seems to have not resonated with many others.

I'm not here to argue for the sake of argument or to hear the sound of my
own (virtual) voice.  I'm happy to have a reasoned debate about
architectural issues, but I'm not about to have another yes-no-yes-no
argument and I'm not interested in debating mechanisms.

I reject the classification of CEE vs CES.  To me that division is about as
useful as the set of proposals that begin with the letter 'I'.  Sure, it's a
division, but it doesn't really tell us anything insightful about the
solution space. 

Regards,
Tony

---

From: Tony Li <[email-masked]>
Date: Sun, 07 Mar 2010 20:50:34 -0800

Hi Robin,

> arguing for your own preferred position?

Well Robin, I have to admit, I've been far less prolific than you have been.
However, for the last three years, I've been plugging away for an
enlightened debate about the solution space.  No takers.  I've argued for my
position in the past and been slammed with ~"that can't possibly work"~

> I understand you support

I want nothing.  IPv4 is done.  Over.  Cooked. Fully toast.  It will either
enter a black market where we deaggregate and no proposal will help, or we
shift to v6 and v4 is irrelevant.  In either case, we're not in time to do
anything significant for v4.  And we still need a v6 solution, that's
clearly higher priority.

> AIS (Evolution) proposal.

I'm not expecting anyone to develop a recommendation.  I'm not expecting
anyone to be happy with it.

> will choose the text, or guide the mailing list or meeting discussions.

We will present the results at our next meeting.  After that, we will polish
the text and then publish the final document.  We will certainly be happy to
take input on the text, but we will not promise to incorporate any of it.

> latter, do you have any plans for how long after the meeting you will

Again, finalizing the text will be later.  Yes, regardless, you're at a
disadvantage being remote.  Sorry, but the IRTF (unlike the IETF) has no
constraints about the mailing list being the primary mode of operation.  Nor
are there requirements for us to do anything to support remote
participation.  We do try to, but that's mostly thanks to co-location with
IETF.

> debate we are yet to have.

Again, we've been trying to have that debate for three years.

> much value or be taken particularly seriously by the IESG or anyone else.

Again, the IRTF is not bound by a consensus process.  In the preceding years
where we've tried to drive consensus, we've gotten no response.  We're left
making a call without consensus.  If that means that it doesn't have broad
support, well, so be it.  We committed to a deliverable and we're going to
see it through.

> of people have in common and where they differ.

That's because we would never reach consensus on what _that_ text would be
either.

> support of their position, and the arguments they provide against the

Done.

> does would be a worse failure.

No one is claiming agreement.

Tony

---

From: Robin Whittle <[email-masked]>
Date: Mon, 08 Mar 2010 21:36:38 +1100

Hi Tony,

You wrote:

> enlightened debate about the solution space.  No takers. 

Maybe your vision of an enlightened debate doesn't accord with that
of others.  I have been trying to do the same thing and it seems you
and I rarely agree about how to discuss things or about how to solve
the routing scaling problem.  Other people appreciate my efforts to
understand and discuss their proposals - so maybe my idea of how to
debate things is shared more by other people than yours is.

> I've argued for my position in the past and been slammed with
> ~"that can't possibly work"~

Core-Edge Separation AKA Locator / Identity Separation could work,
but only if:

  1 - Everyone moved to IPv6 services, network addresses, routers,
      hosts and applications.

  2 - Essentially all IPv6 hosts stacks are upgraded.  (Some CEE
      architectures require upgraded applications.  I understand
      ILNP doesn't.)

  3 - Upgrade any routers which need it.  (For ILNP I understand
      routers rewriting Locators is optional.  I am not sure how
      important it would be to have this.)

Considering there are Core-Edge Separation alternatives such as LISP
and Ivip which work for IPv4 and IPv6, and don't require changes to
any hosts or most routers, then I agree with the critiques you have
received.

Can you point to mailing list messages or other materials in which
you argued that it could work and that this would be superior to a
CES architecture?  I think this would involve justifying the extra
host burdens and delays of the CEE approach in terms of the reduced
complexity of the routing system and potentially other benefits.

> anything significant for v4.  And we still need a v6 solution, that's

The IPv6 Internet could be unplugged right now and almost no-one
would notice.  All the work, all the pleasure, all the social contact
which occurs on "The Internet" as people know it today would still
happen without a glitch, because the IPv4 Internet is the only one
essentially everyone uses and relies upon.

In the thread:

   IPv4 & IPv6 routing scaling problems
   http://www.ietf.org/mail-archive/web/rrg/current/msg05946.html

a number of people indicated that they considered IPv4 important and
that it was going to be used for a "long time".

LISP or Ivip will allow better utilization of IPv4 address space, in
a scalable manner, as I argue in:

     17.2.3 and 17.5 para 3.

I am curious to know Lixia's view on this.  If one or both co-chairs
don't see IPv4 as in need of a solution, while many other
participants do, then I hope you will try to respect the wishes of
what may AFAIK be the majority of participants who want to devise and
ideally reach consensus on the best approach to solving the IPv4
routing scaling problem as well as IPv6's.  I believe the IPv4
problem is more urgently in need of fixing than IPv6's - and my guess
is that quite a number of other people agree.

Its not just portability, multihoming and inbound TE for fixed
end-user networks.  Mobility is also part of what needs to be done
scalably.  Mobility is prominently mentioned in the RRG Charter, and
the Charter contains nothing to indicate that the IPv4 scaling
problem is either insoluble or less urgent than any scaling problem
IPv6 may have in the future.

> anyone to be happy with it.

Does this mean you are going to develop the recommendation?

> take input on the text, but we will not promise to incorporate any of it.

My interpretation of this is that you and Lixia will write the
recommendation and while you will read or listen to what people have
to say about it, you won't promise to take notice of any of it or to
alter your text according to the views expressed by RRG participants.

In the past, you have stated your intention to achieve consensus on a
recommendation.  Now I understand you don't intend to achieve or even
test for consensus on the recommendation you and Lixia will prepare.

> constraints about the mailing list being the primary mode of operation.  

OK.  http://tools.ietf.org/html/rfc2014

    Since the products are research results, not Internet standards,
    consensus of the group is not required.  Rather, the measure of
    success is the quality and impact of the research results.

    ...

    A Research Group will conduct much of its business via its
    electronic mail distribution list(s).  It is also likely to meet
    periodically to accomplish those things that are better achieved
    in more interactive meetings, such as brainstorming, heated
    altercations, etc.  Meetings may be scheduled as telephone
    conference, video teleconference, or face-to-face (physical)
    meetings.

    ...

    Each Research Group will determine the balance of email and
    face-to-face meetings that is appropriate for making progress on
    its goals.

    ...

    The challenge to managing Research Group meetings is to balance
    the need for consideration of the various issues, opinions and
    approaches against the need to allow forward progress.  The
    Research Group, as a whole, has the final responsibility for
    striking this balance.

    ...

    It is up to the Research Group to decide the authorship of papers
    resulting from Research Group activities.  In particular,
    authorship by the entire group is not required.

    ...

    The Research Group Chair is concerned with making forward
    progress in the areas under investigation, and has wide
    discretion in the conduct of Research Group business.  The Chair
    must ensure that a number of tasks are performed, either directly
    or by others assigned to the tasks.

The statements above give responsibilities to the "Research Group" -
however, the following items indicate the Chair has ultimate
responsibility for deciding how the "Research Group" works:

    The Chair has ultimate responsibility for ensuring that a
    Research Group achieves forward progress.

There's no mention of consensus being required, or of the Chair being
responsible for doing things the way any one or more participants
request.

> Nor

OK.

> Again, we've been trying to have that debate for three years.

I believe you have not contributed much to the debate.

You banned debate about "proposals" for most of the time since 2007.
 (I will provide links to the messages if anyone wants them.)  You
required all people with proposals to submit "Summary and Analysis"
documents in early 2008 - which you and Lixia never commented on or
encouraged debate about.   If your ban and discouragement had been
universally respected, then there would have been much less debate of
any kind - and in my view less progress.

You frequently requested we debate "architectures" - as if this was
somehow different from proposals.  Yet I don't recall an instance of
you leading by example with any such messages.

The CEE / CES distinction is the single biggest architectural
distinction I know of between the various potential solutions.  You
say it isn't significant or helpful, but you don't say why you
believe this - such as by why you disagree with msg05865.

> where we've tried to drive consensus, we've gotten no response.  

I think you have done little or nothing to achieve consensus.  Lixia
at least has participated in a proposal - APT and now AIS
(Evolution).  I understand you support ILNP.  On another mailing list
(arin-ppml/2008-June/010988.html) you stated how important GSE was.

I don't recall where you argued that the extra burdens on hosts and
delays in communications was either insignificant or worth the saving
in routing system complexity.

You haven't critiqued any of the proposals - but Lixia did.

I don't think you have done much to find commonalities and points of
disagreement between participants.  Lixia as done something towards
this by discussing a taxonomy in the past.  She also was co-author of
the paper which did most to establish Core-Edge Separation and
Core-Edge Elimination as valuable terms in this field.

I think you have neither supported the kind of debate which might
result in greater consensus nor debated anything in a manner which
shows by example how to constructively debate the merits of a
proposal, architecture or whatever.

> We're left

OK - you are the co-chairs.

I am surprised you are planning on writing the Recommendation rather
than working with others to do so.  I guess other folks would be
surprised too.

> That's because we would never reach consensus on what _that_ text would be

I don't see how you could be sure of this if you haven't tried.

>> support of their position, and the arguments they provide against the

I agree.  If nothing further happened the recent process of
proposals, critiques and "rebuttals" has been constructive and I
believe of lasting value.  I intend to write up an ID containing, or
pointing to, the fuller discussions about each proposal, including
critiques which didn't make it into the final RRG Report.

> No one is claiming agreement.

OK.  So what you seem to be planning is the co-chairs writing a
Recommendation and everyone else discussing it, with no guarantee
that the co-chairs will use that discussion to alter the Recommendation.

I think the result of that won't have anywhere near as much influence
on the thoughts of people outside the RRG than a Recommendation which
resulted from more engagement and input from the group itself.

This is especially the case regarding the IPv4 routing scaling
problem, which you have indicated you have no interest at all in
trying to help with.

Those people will at least be able to see a bunch of proposals, the
frequently artificially shortened critiques and "rebuttals" and then
look into the RRG archives to see more substantial material and what
other folks had in mind about the best way to solve the routing
scaling problem, including with mass-adoption mobility.

  - Robin

---

From: Robin Whittle <[email-masked]>
Date: Mon, 08 Mar 2010 21:52:26 +1100

Hi Tony,

OK.

> Also, we've been over (and over and over) the points here repeatedly.

Can you point to some messages or other documents where you or others
argue that the delays, extra packets, extra complexity etc. I believe
Loc/ID Separation forces on all hosts are either non-existent or a
price worth paying for saving on the additional routing system
complexity required by the only other obvious solution - Core-Edge
Separation (ITRs, ETRs and tunneling)?

> seems to have not resonated with many others.

OK I don't believe this either.  If I restate something on the list
it is because I perceive it is important and hasn't been debated yet.

Some folks who are involved in the RRG don't get to meet you in
person or talk with you by phone.

> own (virtual) voice.  

Me either.

> I'm happy to have a reasoned debate about

Are you interested in debating anything on the list?

What do you suggest other people should debate?

Can you point to where you lead by example - where you discussed
whatever it is you regard as "architecture" in a manner which, if
others responded in a similar fashion, what you regard as a good
debate would have eventuated?

> division, but it doesn't really tell us anything insightful about the

OK - this is your view.  Like Ran, Joel and Lixia who have also
expressed this view, you haven't written to the list why you regard
this distinction as being of no architectural or practical
significance.

Other people think it is an important and meaningful distinction.  To
see why I think it is architecturally important please see the
messages I link to in msg06187.  My attempt to trace the history of
the terms and to discuss some ambiguities in what is being
"separated" in Core-Edge Separation can be found in msg06110.

I thought you would see this distinction as architectural and
important.  I would appreciate it if you pointed to any other
architectural issues, distinctions, etc. in this field which you
think are more valid, more useful or whatever than than those I
discuss in:

  CES & CEE are completely different (graphs)
  http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html

and also pointed out why the distinctions I mention in this message
are invalid or unimportant.

I would have thought you would agree that the choice of retaining the
current IPv4/6 naming model or replacing it, for all hosts, with a
Locator / ID Separation model is an architectural matter of the
highest significance.  CEE architectures do this and CES
architectures don't.

  - Robin

---

From: Russ White <[email-masked]>
Date: Sun, 07 Mar 2010 20:57:43 -0500

> Again, our job is to make a decision and that's what we're going to do.

One thing that might be _nice_ is to have some idea of what the future
of routing, in general, looks like under each proposal. In other words,
does the proposal allow future changes in the routing system through
incremental deployment of new ideas and techniques, or does it place us
in the position of forcing "flag days?" As architectures become more
complex on the machine side, they also tend to become more brittle in
general.

A second thing might be to address mobility. How does each proposal deal
with host level mobility, since this is obviously a direction in the
Internet at large (whether we like it or not, mobile phones and other
such devices are going to rely increasingly on the Internet, which
may--or may not--place a larger burden on the routing system).

A third thing might be to address whether or not any of the solutions
proposed actually resolves the problem seen in the field--IE, describing
what the problem is and why it exists (I've mostly seen hand waving to
this point about the size of the global table, and projections against
the future size of that table, but nothing that says why the table is
growing at this rate, other than the blanket statement of "dual homing,"
which doesn't really answer the question at all), and how each solution
specifically resolves the problem. Since I don't tend to think the
problem is "dual homing," since that problem is "solvable" in other ways
that don't require a redesign of the entire Internet routing architecture.

Don't know how far we'll go on these sorts of things, but it would
certainly help increase the ability of folks to assess the work if we
could consider them in some way.

Russ

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Mar 2010 08:49:16 -0800

Hi Robin,

About IRON-RANGER below:

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Robin Whittle
> Sent: Sunday, March 07, 2010 6:31 PM
> To: RRG
> Cc: Lixia Zhang
> Subject: Re: [rrg] Recommendation and what happens next
> 
> Hi Tony,
> 
> You wrote:
> 

The points on security and scaling have been addressed
in my latest rebuttal; see:

http://www.ietf.org/mail-archive/web/rrg/current/msg06158.html

> non-upgraded networks.

Non-upgraded IPv4 networks will continue to work as they
always have. Do you mean non-upgraded IPv6 networks?

Assuming the latter, let's consider that today we have
two separate BGP instances - the IPv4 BGP instance and
the IPv6 BGP instance. The IPv6 BGP instance carries a
set of IPv6 prefixes (e.g., 2001:: derivatives) that are
routed in the IPv6 Internet in the "traditional" sense.
What IRON-RANGER is doing is essentially creating a
*third* BGP instance - one that carries a new set of
IPv6 Virtual Prefixes (VPs) that are distinct from the
prefixes carried in the traditional IPv6 BGP instance.

Due to routing scaling issues, IRON-RANGER expects that
growth of this new third BGP instance will flourish while
growth of the traditional IPv6 BGP instance will level
off. In order to support what I think you mean by
non-upgraded networks then, it only requires that IPv6
routers connected to the IRON also participate in the
traditional IPv6 BGP instance. For that matter, not all
IRON routers need to participate, but at least some must. 

> As far as I know, there have been no

Do the above points address your concerns?

Thanks - Fred
[email-masked]

> rrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/rrg

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Mar 2010 09:02:52 -0800

Robin,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Templin, Fred L
> Sent: Monday, March 08, 2010 8:49 AM
> To: Robin Whittle; RRG
> Cc: Lixia Zhang

Hmm, maybe these two IPv6 BGP instances I alluded to don't
even need to be kept separate - as long as Virtual Prefixes
can be kept distinct from non-virtual ones. This I think is
the key point, and would greatly reduce the complexity.

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Tue, 09 Mar 2010 05:15:01 +1100

Hi Fred,

...

> http://www.ietf.org/mail-archive/web/rrg/current/msg06158.html

I don't understand the mechanisms by which potentially very large
numbers of IRON routers repeatedly and continually register with
potentially multiple VP routers.  These IRON routers could be
anywhere and so could the VP routers.  In that message, the most I
can discern about this mechanism is:

   IRON routers lease portions of their VPs as Provider
   Independent (PI) prefixes for customer equipment (CEs),
   thereby creating a sustaining business model. CEs that lease
   PI prefixes propagate address mapping(s) throughout their
   attached RANGER networks and up to VP-owning IRON router(s)
   through periodic transmission of "bubbles" with authenticating
   and PI prefix information. Routers in RANGER networks and IRON
   routers that receive and forward the bubbles securely install
   PI prefixes in their FIBs, but do not inject them into the RIB.

I have a rough idea of what you are trying to achieve with this
mechanism, but I have no idea what the mechanism is - "bubbles"
doesn't give me any understanding of exactly how the IRON routers
would do this securely.  Only when I understand exactly how they
would do this could I estimate the scaling and security problems
which are no-doubt involved.

> always have. Do you mean non-upgraded IPv6 networks?

I mean when you start installing IRON routers, these will collect
packets addressed to "edge" addresses (or whatever the new subset of
"EID" space is called in I-R) from the ISP networks they are
installed in.  Packets sent by hosts in those networks will be
handled according to I-R principles and the recipient networks of
those packets will get the benefits (portability, multihoming and
inbound TE) for those packets.

But what of packets sent from hosts without IRON routers?  AFAIK,
there's no equivalent in I-R to Ivip's DITRs or LISP's PTRs.

> Do the above points address your concerns?

It is 5AM here, and I can't clearly translate what you wrote into an
answer to the concern I mentioned above.  But that doesn't mean it
doesn't address it.

> the key point, and would greatly reduce the complexity.

I am definitely not going to think about this until I get some sleep!

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Mar 2010 11:14:03 -0800

Robin,

> From: Robin Whittle [mailto:[email-masked]]
> Sent: Monday, March 08, 2010 10:15 AM
> To: RRG
> Cc: Templin, Fred L
> Subject: IRON-RANGER scalability and support for packets from non-upgradednetworks
> 
> Hi Fred,
> 

I have been using the term "bubbles" to keep this at a
high level and to avoid getting bound up into talking
about specific mechanisms. But, it sounds like you want
to know more about the specifics, so I can tell you that
a "bubble" is actually an IPv6 Router Advertisement (or
moral equivalent) using the RFC3971, Section 6.1
authorization model. In this model, routers are given
certificates from a trust anchor which authorizes them
to act as routers and to advertise a certain set of
subnet prefixes. Other routers in RANGER use the
certificates to verify authorization.

The scalability part is addressed in the new text I
have added to the rebuttal. What we really need to
avoid is having off-path attackers injecting lots of
bogus RAs that cause IRON routers to have to perform
lots of unnecessary crypto. By shutting down off-path
spoofing, IRON-RANGER ensures that end sites that act
with integrity will not overload correspondents with
crypto-busting DoS attacks. End sites that do NOT act
with integrity on the other hand will be easy to detect
and can safely be blacklisted.

> "EID" space is called in I-R)

Virtual Prefixes (VPs) - but, VPs cover what IRON-RANGER
calls Endpoint Interface iDentifier (EID) prefixes.

> from the ISP networks they are

Considering for now only IPv6 as EID space, an IPv6
host will be connected to a network that is ultimately
served either by an IRON VP or a "native" IPv6 prefix.
If an IRON VP, then routing is naturally via the IRON.
If a "native" IPv6 prefix (e.g., a 2001::/ derivative),
packets will follow default or more-specific routes as
usual but will encounter an IRON VP router if the
destination is covered by a VP.  

> I am definitely not going to think about this until I get some sleep!

OK; rest well.

Fred

---

From: Tony Li <[email-masked]>
Date: Mon, 08 Mar 2010 11:25:46 -0800

Hi Robin,

> debate things is shared more by other people than yours is.

That may well be.  Doesn't make it right, however.

> Does this mean you are going to develop the recommendation?

Yes, that's on the co-chairs.

> alter your text according to the views expressed by RRG participants.

Correct.
 
> test for consensus on the recommendation you and Lixia will prepare.

Yes, that changed.

> I believe you have not contributed much to the debate.

The feeling is mutual.

> you leading by example with any such messages.

Not everything happens on the mailing list.

> I don't see how you could be sure of this if you haven't tried.

We've tried several times to simply agree on what terminology we would use.
All attempts failed.

Tony

---

From: Tony Li <[email-masked]>
Date: Mon, 08 Mar 2010 11:29:25 -0800

Hi Robin,

> Separation (ITRs, ETRs and tunneling)?

Again, I refuse to get into debates about mechanism trivialities.  If we
were going to have an architectural debate, then it should be about the
solution space.  Classes of solutions, not about the engineering details.
We've recently touched on this when we started tried to talk about groups of
proposals.

> Are you interested in debating anything on the list?

Asked and answered.

Tony

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Mar 2010 10:35:25 +1300

On 2010-03-08 14:57, Russ White wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 

There's also the question of which solutions are competitive with
each other, which are compatible with each other, and which simply
defeat each other.

    Brian

---

From: Scott Brim <[email-masked]>
Date: Tue, 09 Mar 2010 13:44:19 -0500

> may--or may not--place a larger burden on the routing system).

The routing system does not deal with endpoint mobility directly and
cannot make many predictions about how it will be handled.  However,
each proposal does set up the framework in which mobility has to be
designed, and can constrain how mobility can be done.  It would be good
if each proposal listed the assumptions it makes, and the constraints it
puts on, both endpoint and network mobility.

Scott

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 9 Mar 2010 14:22:32 -0800

Scott,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Scott Brim
> Sent: Tuesday, March 09, 2010 10:44 AM
> To: Russ White
> Cc: RRG

With IRON/RANGER, the hybrid routing system handles network
mobility without causing a ripple effect in the BGP. Endpoint
mobility as you say is not handled by the routing system
directly, but is rather handled by an adjunct mechanism. We
have been thinking that HIP would be the natural adjunct
mechanism to not only handle host-level mobility but also
to give a true loc/ID split.

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Wed, 10 Mar 2010 13:18:19 +1100

Hi Fred,

> and can safely be blacklisted.

I wasn't even thinking about attackers.  I don't know anything in
detail about RFC 3971 and I don't think you should require readers to
interpret a document such as this, which has nothing to do with I-R,
in the context of what little they know about I-R - and then to
figure out enough detail about how it would work to convince
themselves that this is scalable.

To improve on this section of the I-R description, I suggest you
nominate some target numbers for the maximum number of IRON routers
there will ever be, how many individual prefixes of end-user network
"edge" space there could be in total, the maximum number of these
that any one IRON router might need to handle, how many VPs there
would be, how many individual "edge" prefixes there might be in a VP,
how many VP routers there might be per VP prefix, how often each IRON
router will register each of its "edge" prefixes with the one or move
VP routers . . . and then describe in detail how the system works,
with one or more examples.  This would enable people to understand in
good physical detail how your system works, without having to read
RFC 3971 and imagine how you would use it in I-R.  The reader needs
to be able to estimate the number of packets flowing in the various
roles, the length of the packets, the computational resources
required at each router, the security measures as you mentioned - but
in greater detail - and then to figure out who is paying for all this
and why they would do so.  They would also want to see the whole
thing works OK when individual routers fail, become unreachable, are
rebooted etc.

The only way I could develop my partial understanding if I-R was to
do a lot of imagining, then ask detailed questions, and then have you
provide answers - in a recursive process until I thought I had a good
enough understanding of the proposal.   I don't have time to do this
for I-R or other proposals now.  However, if you have a clear idea of
how it would work, and what your targets are, you should be able to
write a description which anyone can read and understand, without
having to imagine anything or ask too many further questions.

There may well be a way of doing it - and I think for anyone involved
in CES architectures, I think I-R certainly is worth understanding.
All I am saying now is that the current state of the proposal leaves
important scaling problems unanswered.  Its actually more than that -
it doesn't explain this registration process in anything like
sufficient detail for people to understand how it would work.

> calls Endpoint Interface iDentifier (EID) prefixes.

So do the VP routers act like Ivip's DITRs and LISP's PTRs?  To make
this work without unreasonably extending the path between the sending
host and the IRON router which connects to the destination network,
you would need multiple VP routers - say 10 or 20 or so, depending on
how keen you were to minimise path lengths.  This number needs to be
part of the explanation regarding operation and scaling properties I
suggested above.

> destination is covered by a VP.  

OK - this confirms what I assumed above, about the VP router(s)
acting as DITR(s) or PTR(s).

I recall that the VP router advertises its VP in the IRON overlay
network.  Does it advertise the VP in the DFZ too?  It would need to
if it were to receive packets sent from hosts in non-upgraded networks.

>>> IRON routers need to participate, but at least some must.

Sorry, I still can't follow this and I don't have time to ask more
about it.

 - Robin

---

From: Robin Whittle <[email-masked]>
Date: Wed, 10 Mar 2010 13:58:14 +1100

Short version:      TL:   Again, we've been trying to have that
                          debate for three years.

                    RW:   I believe you have not contributed much to
                          the debate.

                    TL:   The feeling is mutual.

                 Tony gives no explanation for his dismissal of my
                 efforts - including 5 of the 15 critiques in the
                 RRG Report.

                 I give reasons for my negative assessment of his
                 contribution to debates - as briefly as possible.

Hi Tony,

> That may well be.  Doesn't make it right, however.

My impression is that an IRTF research groups is a less formal and
more free-wheeling arrangement than an IETF WG.  In msg06204 I quoted
some pertinent parts of RFC 2014.

Responsibility is generally given to the Research Group.  But the
Chair has wide discretion about how everything is done and is
ultimately responsible for the Group making progress.

By your own account, the Group hasn't made much progress.

We have achieved consensus on almost nothing.  We have no agreed
statement of the the problem or of the goals for a solution. We have
no agreed taxonomy of potential solutions.  We can't agree on whether
IPv4 is worth working on.  We can't agree whether all hosts should be
required to move to IPv6 and do more work, suffering more delays when
establishing communications or changing the (Locator) IP address (CEE
= Loc/ID Separation) rather than add some things to the routing
system (CES) and leave IPv4 and IPv6 host stacks and applications
untouched, and with no further burdens and few, if any extra delays
in establishing communications.

If you are right about how the Group should be run, are right about
Locator / Identity Separation being the way forward, and are right
about IPv4's fate:

   I want nothing.  IPv4 is done.  Over.  Cooked. Fully toast.  It
   will either enter a black market where we deaggregate and no
   proposal will help, or we shift to v6 and v4 is irrelevant.  In
   either case, we're not in time to do anything significant for v4.

then in the fullness of time this will be widely recognised and all
will be well.

OK.

OK.

>> test for consensus on the recommendation you and Lixia will prepare.

I would appreciate it if you announced such changes clearly on the
mailing list.  The only way I and I guess other people discovered
this was by me questioning you in recent days.

I think both you and Lixia failed to inform those who rely on the
mailing list what your plans are and how they have changed.

>> I believe you have not contributed much to the debate.

I have explained my reasons for thinking you made little contribution
- firstly a simple absence of you debating things at all, or in any
depth and secondly you banning or discouraging debate on "proposals"
(AKA "candidate architectures") while failing to lead by example by
debating whatever it is you wanted us all to debate, which you
described as "architecture".

Your negative assessment of my contribution is without any such
explanation.

I wrote critiques of, and discussed, every proposal but my own and
ILNP.  (I also discussed ILNP on the list and offlist with Joel, but
I never wrote an actual critique.  Still, ILNP is covered by my
critique of all CEE architectures.)  I recall that the proponents of
all these 13 proposals wrote on the list that they appreciated my
efforts and found the critique and/or the discussions which followed
helpful.

The draft RRG Report contains 5 critiques I wrote.  In the case of
LISP, an alternative was offered by Noel:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05747.html

If I hadn't worked on these 4 proposals: hIPv4, Compact Routing, TIDR
and the formidable IRON-RANGER (which I devoted a week of my life to)
the RRG Report, for which you and Lixia are primarily responsible,
would not have critiques for these proposals.

How many of the proposals have you read?  Since you accepted the
proposals, you have only commented on ILNP.

I wrote about the architectural distinction between CEE and CES
(msg05865) and you dismissed this (msg06221) without any explanation
for why the things I mentioned (including Loc/ID Separation) are are
not both architectural and important.

I tried to describe the constraints we face due to the need for
widespread voluntary adoption:

  http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

and you showed no interest - not even a criticism.

I regard your unexplained dismissal of my contributions to the RRG -
and to the report you and Lixia are primarily responsible for - as
ungracious, unfounded and unbecoming of the position of IRTF Chair.

> Not everything happens on the mailing list.

Can you cite where in the meetings you lead by example?  The audio
archives are available.

Since you exhort us all to debate things on the mailing list the way
you want them debated - something to do with "architecture", which I
think we are doing very well despite your disapproval - then I think
it is your responsibility to lead by example, on the mailing list.

> We've tried several times to simply agree on what terminology we would use.

Can you point to any messages where you proposed some terminology we
should adopt?

See your recent msg06221 for how you dismiss, without giving any
reasons, a serious attempt to discuss terminology and architectural
distinctions between proposals.

My impression of your position is:

   Loc/ID Separation is doing it right.

   IPv4 is dead, toast etc. etc.

   ILNP is the only current proposal you support.

   You don't like reading or writing things longer than a few
   paragraphs.

   You are generally uninterested in what you regard as "engineering
   details" - but find that many other folks think that what you
   regard as such is "architectural" or at least important.

   You frequently don't feel it is necessary to explain the reasons
   behind your disapproval of other people's work - at least in a
   level of detail which allows the other people to debate those
   reasons.  (See note for a recent example.)

   When someone points out that you don't seem to be leading by
   example in the mailing list debate you want us to have, rather
   than admit this, you feel it is adequate to assert that you
   are doing this in some unspecified venues other than the list.

   You are highly confident that you know what is best for the future
   of the Internet.

   You have not encountered much, or anything, in this scalable
   routing phase of the RRG which has altered your view of what is
   "right".

Would you disagree with this, or add anything to it?

Can you mention other people who share your vision of how the
Internet should develop?

 - Robin

Here is an example of your lack of explanation when disagreeing with
someone - in this case refusing to debate something because you
consider it to be "engineering" rather than "architecture":

  http://www.ietf.org/mail-archive/web/rrg/current/msg06221.html

You don't explain why "engineering details" are "mechanism
trivialities".  Nor do you explain why the CEE/CES distinction I
believe is architecturally important is not so.  You explain the
reasons why you believe forcing all hosts to adopt the Locator /
Identifier Separation naming model is not "architectural".

I think this is a stark example of a pattern of dismissing views
contrary to your own in a manner which is uncommunicative - and which
I find hard to distinguish from being disrespectful to the people who
work hard to contribute these ideas to the Group you co-chair.

---

From: Tony Li <[email-masked]>
Date: Tue, 09 Mar 2010 22:54:38 -0800

Hi Robin,

> ungracious, unfounded and unbecoming of the position of IRTF Chair.

You're right, I apologize.

Tony

---

From: Robin Whittle <[email-masked]>
Date: Wed, 10 Mar 2010 18:02:17 +1100

Here's a revised last few words for a paragraph in my previous
message in this thread:

I regard your unexplained dismissal of my contributions to the RRG -
and to the report you and Lixia are primarily responsible for - as
ungracious, unfounded and unbecoming of the position of co-chair.

and some additions:

Is it just my contributions you have such a dim view of?  Can you
nominate some people whose work you think has made a useful
contribution to the RRG?  There's been 6,243 messages, plus the
presentations at meetings, IDs and other documents.

Since you don't promise to alter your and Lixia's Recommendation
based on input from other people, do you actually want input for this
purpose?

Could you give an indication of any aspects of what you are planning
to recommend which might be affected by what an RRG member might
write on the list or say in forthcoming meeting?

  - Robin

---

From: Robin Whittle <[email-masked]>
Date: Wed, 10 Mar 2010 23:27:49 +1100

Hi Tony,

> You're right, I apologize.

Thanks for this.  BTW, I meant to write "co-chair".

  - Robin

---

From: Scott Brim <[email-masked]>
Date: Wed, 10 Mar 2010 11:44:50 -0500

> to give a true loc/ID split.

Suppose the generic question everyone should answer is "how does the
proposed system constrain or promote specific approaches to endpoint and
network mobility?".  You would say something like: "IRON/RANGER does not
introduce any constraints on endpoint or network mobility approaches, or
make one more appropriate than another."?

thanks ... Scott

---

From: Scott Brim <[email-masked]>
Date: Wed, 10 Mar 2010 12:25:22 -0500

> "separated" in Core-Edge Separation can be found in msg06110.

CES/CEE is a useful dimension for comparison but (1) it's not binary,
despite our wishes, and (2) there are other dimensions that are also
important.  We just couldn't figure out how to use them for effective
comparisons.  See the various taxonomy attempts.

Scott

---

From: Robin Whittle <[email-masked]>
Date: Thu, 11 Mar 2010 11:55:12 +1100

Short version:   It seems I have been unable to tempt people to
                 read and respond to msg05865, so here are the
                 distinguishing points between Core-Edge Elimination
                 and Core-Edge Separation architectures as briefly
                 as they can be mentioned.

Hi Scott,

> comparisons.  See the various taxonomy attempts.

I am not suggesting that all proposals are either CES or CEE.  In
msg06219, after listing the "mapping only" proposals, I wrote that
these three were neither CES or CEE:

   hIPv4
   Name overlay (NOL)

I don't think any of these can achieve the scaling benefits we need
for, say 10^7 non-mobile end-user networks and 10^9 to 10^10 mobile
devices.

The four CEE proposals are:

   GLI-Split

The four CES proposals are:

   IRON-RANGER
   Ivip
   LISP
   TIDR

There is a very clear distinction between these two sets.  Please see

for a complete explanation of the following summary.

All the CEE architectures, in order to achieve substantial benefits
to adoptors and to scalable routing have these in common.

  1 - All CEE architectures require all hosts to use the Locator /
      Identifier Separation naming model.  (Not just hosts in
      end-user networks which want portability, multihoming and
      inbound TE.)

  2   All CEE architectures require changed stacks.  Some also
      require changed applications.  Those which support unmodified
      IPv6 applications are pretty dodgy, I think - because these
      applications assume that the IP addresses they deal with are
      both identifiers and locators, and that when sending a packet
      to a host with a given IP address, the routing system will
      never deliver it to any other host than the one with this IP
      address.

  3 - CEE architectures are only practical for IPv6, for multiple
      reasons but always including the fact that CEE requires each
      multihomed end-user network to use at least twice the number of
      addresses (host locators) from the global unicast address
      space.  Each of its ISPs needs to provide the number of
      addresses the network needs.

  4 - So these architectures can't use IPv4 applications or IPv4
      stacks.  They use either IPv6 applications or require
      rewritten versions of these.  Some apps would require a minor
      rewrite and some would require a lot more work, including in
      some cases a major change to the actual protocols the app
      uses.

  5 - All CEE architectures require hosts to do more work - including
      typically both hosts in an initial communication establishment
      doing ID->Loc lookups via global mapping systems with one or a
      few authoritative servers.  Such lookups are likely to be slow
      and involve higher risks of lost packets.

      This will typically delay the establishment of all
      communication sessions.  The delays and the burdens on hosts
      are worse for devices relying on slow, potentially unreliable,
      links such as 3G - so the new burdens are especially bad for
      many mobile devices.

  6 - Some CEE architectures require sending longer packets between
      hosts.

  7 - Have a really undesirable relationship between adoption levels
      and substantial benefits to adopters (all, or almost all of
      their communications have the benefits of portability,
      multihoming and inbound TE).

           ^
         B  |        *      Core-Edge Elimination (CEE)
         e  |       .*
         n  |      ..*      .  Proportion of communications
         e  |     ...*         with the benefits.
         f  |    ....*
         i  |   .....*      *  Actual total benefits generally
         t  |  ......*         only arise when all, or almost
            | .......*         all, communications have the
            |........*         benefits.
            0--------->
              Effort ~= level of adoption

  8 - As a result of the above, only produce substantial scaling
      benefits after all, or nearly all hosts adopt it:

            ^
         B  |        *      Core-Edge Elimination (CEE)
         e  |        *
         n  |        *
         e  |        *
         f  |        *
         i  |        *
         t  |        *
            |        *
            |        *
            0--------->
              Effort ~= level of adoption

  9 - While no CEE architecture involves tunneling, some involve
      routers rewriting Locators of packets and some involve "split-
      horizon" DNS tricks.  AFAIK (and I haven't yet read all the
      responses to my CEE critiques) none of them appear to be
      suitable for use with ULA addresses.

Whether or not any of these CEE architectures would work as intended
I am not yet sure.  If they did work as intended, they could
certainly solve the routing scaling problem in terms of portability,
multihoming and inbound TE for non-mobile networks.  I do not believe
any of them are suitable for providing mobility.

CES architectures have these features in common, assuming they could
actually work and solve the scaling problem, which is not true of
TIDR - and which I am not yet convinced is true for IRON-RANGER.

  1 - No changes to the current naming model, in which the IP address
      plays the roles of both Identifier and Locator.

  2 - Therefore, requires no changes to host stacks or applications.

  3 - Works fine with IPv4 and IPv6 (for Ivip and LISP at least - I
      think some of IRON-RANGER is IPv6-specific).

  4 - LISP and Ivip at least can support the TTR Mobility
      architecture and so provide scalable mobility - for both IPv4
      and IPv6, without changes to host stacks (except for extra
      tunneling software in the MN).

  5 - Do not require hosts to do more work.  Significant extra delays
      at one or both ends of a communication establishment will
      sometimes, perhaps frequently, occur with LISP-ALT - due to
      ITRs dropping packets until they receive mapping information
      via the ALT global query system.  Ivip with DRTM typically
      involves insignificant delays since authoritative query servers
      are "nearby".  (See draft-whittle-ivip-drtm.)

  6 - With LISP's PTRs and Ivip's DITRs, adopters get full benefits
      of portability, multihoming and inbound TE for all their
      communications irrespective of the level of adoption:

            ^
         B  |*********      Core-Edge Separation (CES)
         e  |*********
         n  |*********
         e  |*********
         f  |*********
         i  |*********
         t  |*********
            |*********
            |*********
            0--------->
              Effort ~= level of adoption

  7 - Because of this, substantial scalable routing benefits accrue
      in direct proportion to the level of adoption:

            ^
         B  |        *      Core-Edge Separation (CES)
         e  |       **
         n  |      ***
         e  |     ****
         f  |    *****
         i  |   ******
         t  |  *******
            | ********
            |*********
            0--------->
              Effort ~= level of adoption

  8 - Apart from Ivip's Modified Header Forwarding arrangements,
      CES architectures involve encapsulation for tunneling packets
      from ITRs to ETRs (IRON-RANGER doesn't have ITRs and ETRs, but
      it still requires encapsulated tunneling).  There are some
      problems with this - but they do not appear to be prohibitive.

What other dimensions can you suggest?

The above dichotomy seems to be significant in many respects,
starting with the basic architectural question of whether hosts
retain the currently used naming model, which saves hosts a lot of
work and results in communication establishment without lookups - or
whether hosts (all hosts) are required to adopt Locator / Identifier
Separation.

Do you think the above CEE/CES distinctions are invalid or
unimportant?  The all look really important to me.

None of the people who have dismissed the architectural or practical
importance of the CEE/CES distinction have explained their reasoning
in any way.  To do so, I think they should argue against the
distinctions listed above.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 13:53:09 -0800

Robin,

In catch-up mode, your suggestion of a need for more
discussion of scaling properties is a good one, and
something I think that can be accommodated in the
next IRON update.

Thanks - Fred

> Sent: Tuesday, March 09, 2010 6:18 PM
> To: RRG
> Cc: Templin, Fred L
> Subject: Re: [rrg] IRON-RANGER scalability and support for packets from non-upgradednetworks
> 
> Hi Fred,
> 
> You wrote:
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 14:18:01 -0800

Scott,

> From: Scott Brim [mailto:[email-masked]]
> Sent: Wednesday, March 10, 2010 8:45 AM
> To: Templin, Fred L
> Cc: Russ White; RRG
> network mobility?".

That is a very good question.

> You would say something like: "IRON/RANGER does not

That is also correct. Although I named a specific
endpoint mobility approach, there is nothing in
IRON/RANGER that would introduce any unfavorable
constraints to other approaches.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 14:29:41 -0800

Hi Robin,

A few comments below:

> Sent: Wednesday, March 10, 2010 4:55 PM
> To: RRG
> Cc: Ran Atkinson; Scott Brim
> Subject: Re: [rrg] Why won't supporters of Loc/ID Separation (CEE)argue their case? (Summary of
> differences)
> 

I am convinced that IRON-RANGER solves the scaling problem,
but I will work on some additional scaling analysis to also
convince you and other readers.

>       think some of IRON-RANGER is IPv6-specific).

IRON-RANGER used to speak of using IPv6 neighbor discovery
as the means for locator liveness testing, dissemination
of routing information, secure redirection, etc. However,
the VET and SEAL mechanisms are being revised to instead
use a different mechanism called the SEAL Control Message
Protocol (SCMP) for tunnel endpoint negotiations that occur
*within* the tunnel sublayer and are therefore not visible
to either the outer IP protocol nor the inner network layer
protocol. Hence, the inner network layer protocol could be
anything, including IPv4, IPv6, OSI CLNP, or any other network
layer protocol that is eligible for encapsulation in IP.

>          B  |*********      Core-Edge Separation (CES)

IRON-RANGER calls them as ITEs/ETEs because it is possible
to also configure a tunnel endpoint on a host and not just
on routers. In terms of routers, the IRON-RANGER ITE/ETE
are exactly equivalent to what the other proposals are
calling as ITR/ETR.

Thanks - Fred

---

From: Scott Brim <[email-masked]>
Date: Thu, 11 Mar 2010 17:36:40 -0500

> That is a very good question.

I guess RRG will not get around to asking about how each proposal
impacts mobility, but when the IETF does, this should be answered for
each proposal.

---

From: Robin Whittle <[email-masked]>
Date: Fri, 12 Mar 2010 11:17:42 +1100

Hi Fred,

> something I think that can be accommodated in the

Also, in "Re: [rrg] Why won't supporters of Loc/ID Separation ..."
you wrote:

> IRON-RANGER used to speak of using IPv6 neighbour discovery

OK.  I hope you will be able to explain these things not just in
terms of high-level concepts, but to give examples of how the whole
thing would actually work on a large scale.

For instance, how many IRON routers are there in an IPv4 I-R system,
and how many individual EID prefixes?  Then, how do these IRON
routers, for each of these EID prefixes continually and repeatedly (I
guess every 10 minutes or less) securely inform a given number of VP
routers they are the router, or one of the routers, to which packets
matching a given EID prefix should be tunneled.  Since there could be
multiple VP routers for a given VP, and the IRON routers don't and (I
think) can't know where they are, how does this process work securely
and scalably?

If the VP routers act like DITRs or PTRs by advertising their VP in
the DFZ, then in order to make them work well in this respect - to
generally minimise the extra path length taken to and from them
compared to the path from the sending host to the proper IRON router
- I think you need at least a dozen of them.   This directly drives
the scaling problems in the process just mentioned where the IRON
routers continually register each of their EID prefixes with the
dozen or so VP routers which cover that EID prefix.

Your IDs tend to be very high level and tend to specify external RFCs
for how you do important functions in I-R.  Yet those RFCs say
nothing about I-R itself.  I think your I-Ds generally need more
material telling the reader specifically how you use these processes
in I-R.   Then, for each such process, have a detailed discussion
with real worst-case numbers to show that it is scalable at every
level for some worst-case numbers of EID prefixes, IRON routers etc.
- as well as secure against various kinds of attack.

>>       CES architectures involve encapsulation for tunneling
>>       packets from ITRs to ETRs (IRON-RANGER doesn't have ITRs and
>>       ETRs, but it still requires encapsulated tunneling).  There
>>       are some problems with this - but they do not appear to be
>>       prohibitive.
>

OK.  In Ivip the sending host can have in "ITR" function - though it
is not a router and this "ITR" function doesn't advertise routes to
the MABs (Mapped Address Blocks) inside the host.  It does however
only handle packets sent by the host's stack which have destination
addresses matching any of the MABs.  I am sticking with "ITR" and
"ETR" in Ivip, to remain compatible with LISP - and because I think
they are easier to pronounce than "ITE" and "ETE".

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 12 Mar 2010 14:48:49 -0800

Hi Robin - see below for some follow-up:

> Sent: Thursday, March 11, 2010 4:18 PM
> To: RRG
> Cc: Templin, Fred L

OK if you are talking about an architectural description,
but please note that both VET and SEAL are already full
functional specifications that can be used by software
developers to produce real code. 

> and how many individual EID prefixes?

Let's suppose that each VP is an IPv6 ::/32, and that
the smallest unit of PI prefix delegation from a VP is
an IPv6 ::/56. In that case, there can theoretically be
up to 4B VPs in the IRON RIB and 16M PI prefixes per VP.
In practice, however, we can expect to see far fewer than
that until the IPv6 address space reaches exhaustion
which many believe will be well beyond our lifetimes.

Still thinking (very) big, let's try sizing the system
for 100K VPs; each with 100K ::/56 delegated PI prefixes.
That would give 10B ::/56 PI prefixes, or 1 PI prefix
for every person on earth (depending on when you sample
the earth's population). Let's look at the scaling
considerations under these parameters:
 
> Then, how do these IRON

Each IRON router R(i) discovers the full map of VPs in
the IRON through participation in the IRON BGP. That
means that each R(i) would need to perform full database
synchronization for 100K stable IRON RIB entries that rarely
if ever change. This doesn't sound terrible even for existing
core router equipment. As you noted, it is also possible that
a given VP(j) would be advertised by multiple R(i)s - let's
say each VP(j) is advertised by 2 R(i)s (call them R(x) and
R(y)). But, since the IRON RIB is fully populated to all
R(i)s, each R(i) would discover both R(x) and R(y) that
advertise VP(j).

Now, for IRON router R(i) that is the provider for 100K PI
prefixes delegated from VP(j), R(i) needs to send a "bubble"
to both R(x) and R(y) for each PI prefix. That would amount
to 200K bubbles every 600 sec, or 333 bubbles/sec. If each
bubble is 100bytes, the total bandwidth required for updating
all of the 100K PI prefixes is 260Kbps. Now, let's say that
each PI prefix is multihomed to 2 providers, then we get 2x
the message traffic for 520Kbps total for the bubbles needed
to keep the 100K PI prefixes refreshed.  

> dozen or so VP routers which cover that EID prefix.

I don't understand why the dozen - I think with IRON VP
routers, the only reason for multiples is for fault tolerance
and not for optimal path routing, since path optimization will
be coordinated by secure redirection. So, just a couple (or a
few) IRON routers per VP should be enough I think?

> for how you do important functions in I-R.

You may be speaking of IRON/RANGER, but the same is not
true of VET/SEAL. VET and SEAL are fully functional
specifications from which real code can be and has been
derived.

> Yet those RFCs say

Does the analysis I gave above help? If so, I can put
it in the next version of IRON.
 
> they are easier to pronounce than "ITE" and "ETE".

I'm not sure about this - an {Ingress/Egress} Tunnel
*Router* is a router that happens to terminate tunnel
endpoints. On the other hand, an {Ingress/Egress}
Tunnel *Endpoint* is a tautologically a tunnel
*endpoint* - so, why not call it as such?

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Sat, 13 Mar 2010 14:11:00 +1100

Short version:    Exploring the scalability of IRON-RANGER's
                  "bubble"-based registration system - every
                  10 minutes the two IRON routers of the two
                  ISPs send a registration packet to however
                  many VP (Virtual Prefix) Iron routers there
                  are for the VP which covers the I-R PI
                  prefix in question.

                  I think the scaling properties of this
                  system look bad - and I can't yet see how
                  the IRON routers can discover the IP addresses
                  of all the VP routers.

Hi Fred,

> developers to produce real code. 

I think I-R needs to be described in a way that someone who is up to
speed on scalable routing in general can read one or perhaps two I-R
documents and have a good idea of how the whole thing is going to
work - including with respect to scaling and security.  This doesn't
require exact bits in headers, but that could be part of it.  I think
 it needs to be pretty-much self-contained rather than requiring
people to read other documents which are not part of I-R.

> which many believe will be well beyond our lifetimes.

OK.  Still, depending on how the address space was allocated - or at
least that subset of the address space covered by I-R's VPs - there
could be high numbers, approaching 16M perhaps, of I-R PI prefixes
per VP.

> considerations under these parameters:

OK, I think this is a good scenario to discuss.  I assume that the
VPs can be of various sizes, so some VPs could be a longer prefix,
covering less space, if there are a larger number of I-R PI prefixes
within that part of the address space.

As far as I know, you don't need VPs covering the entire advertised
subset of global unicast address space.  However, for worst-case
scaling discussions I think it is good to assume this.

> the IRON through participation in the IRON BGP. 

I recall that some IRON routers handle VPs and others don't.  As I
wrote earlier, assuming VP routers advertise the VP in the DFZ, not
just in the I-R overlay network, then they are acting like LISP PTRs
or Ivip DITRs.  In order for them to do this in a manner which
generally reduces the path length from sending host, via VP router to
the IRON router which delivers the packet to the destination, I think
that for each VP something like 20 or more IRON routers need to be
advertising the same VP.

I interpret your previous sentence to mean that all the IRON routers
are part of the IRON BGP overlay network, and that each one will
therefore get a single best path for each VP.  That will give it the
IP address of one IRON router which handles this VP.  It won't give
it any information on the full set of IRON routers which handle this VP.

> That
> if ever change. 

I am not sure what you mean by "full database synchronization".  Only
a subset of IRON routers advertise a VP, and each IRON router would
get a best-path to a single IRON router out of potentially numerous
IRON routers which were advertising a given VP.  So any one IRON
router would not be able to use the IRON BGP overlay system to either
discover the IP addresses (or best paths) to all IRON routers, or to
all the IRON routers which advertise VPs, assuming that some VPs were
advertised by more than one IRON router.

> This doesn't sound terrible even for existing

I don't see how this would occur.  A given IRON router receives best
paths for each VP, so for VP(j) it will get a best path to (and IP
address of) either R(x) or R(y).

> to both R(x) and R(y) for each PI prefix. 

Its no-doubt a relief to less muscle-bound scalable routing
architectures that the routers of IRON-RANGER are hurling about
merely "bubbles" rather than something with greater impact!

> That would amount to 200K bubbles every 600 sec, or 333
> bubbles/sec.  If each bubble is 100bytes, the total bandwidth
> required for updating all of the 100K PI prefixes is 260Kbps.

I am not sure each registration "bubble" would only be 100 bytes of
protocol-level data.  You need to specify, for IPv6:

  1 - The IP address of the IRON sending the registration (16 bytes).

  2 - The prefix the IRON router is registering (18 bytes).

  3 - Nonces and other stuff which invariably accompany messages
      such as this (10 to 20 bytes?).

  4 - Authentication material, such as a digital signature for the
      above, including the public key of the signer (the
      IRON router itself?) and a pointer to one or more PKI CAs or
      whatever so the VP router can ascertain that this really is
      the public key of the signer.  These will be FQDNs - lets
      say 50 bytes or so.

Maybe you could get the whole thing into 100 bytes.  Then add the
IPv6 header - 40 bytes - and a UDP header 8 bytes - and we are up to
about 150 bytes already.  Add in L2 headers - Ethernet is 46 octets -
 and we are up to 200 bytes.  Multiply by 8 and this is 1600 bits.

  1600 x 333 = 532,800 bits/sec ~=0.5Mbps

This is the bandwidth of incoming packets to R(x) and likewise for
R(y) in your description.   This is assuming a two IRON routers
("200k bubbles every 600 sec") per I-R PI prefix.

But your description varies from mine already in two other important
respects.

Firstly, if these VP-advertising routers are to operate properly like
DITRs or PTRs, they needs to be a lot more than 2 of them per VP.
Let's say 20.  Maybe 10 would be acceptable, maybe more - but 20 will
do.  Let's call them RVP(j, 0) to RVP(j, 19) where, in your example:

  R(x) == RVP(j, 0)
  R(y) == RVP(j, 1)

Secondly, I don't see how R(i) could discover the IP addresses of
more than one of this set of 20 routers.

In my model, if it could be shown how routers such as R(i) which
handle the 100k I-R PI prefixes in VP(j) could discover all the 20
routers RVP(j, 0) to RVP(j, 19), then each of these 20 routers has
this incoming bandwidth.

> Now, let's say that each PI prefix is multihomed to 2 providers,
> then we get 2x the message traffic for 520Kbps total for the
> bubbles needed to keep the 100K PI prefixes refreshed.

You already assumed two IRON routers per I-R PI prefix in your
260kbps figure above, so there's no need to double at again to 520kbps.

2 ISPs seems a reasonable figure, which was already part of my
calculations.

Each provider has an IRON router which handles a given I-R IP prefix,
and each such IRON router is sending bubbles to all the VP routers
(though I don't yet understand how these VP routers would be
discovered - and I am assuming there are 20 of them while you are
assuming there will be 2 of them).

My figure is 532kbps ~= 0.5Mbps incoming bandwidth per VP router.

> few) IRON routers per VP should be enough I think?

Secure redirection works when an IRON router sends the initial packet
to a VP router, but it doesn't apply when the sending router is that
of a non-upgraded network.  To support generally low stretch paths
from those sending networks to the IRON router which is currently the
desired one for forwarding packets to the destination network, I
think you need a larger number.  20 is a rough figure, assuming a
global distribution of sending hosts and IRON routers which handle
the I-R PI prefixes - as is required for real portability.

If all the IRON routers for the I-R PI prefixes of a given VP were in
Europe, then it would suffice to have all the VP routers also in
Europe - so depending on the need for robustness and load sharing,
perhaps you wouldn't need 20 or them.  Maybe 5 would do.  But
generally, for this kind of scaling discussion, I think we need to
assume the goal of global portability of the new kind of address
space, with sending hosts likewise distributed globally.

So I think that for a VP containing 100k I-R PI prefixes, there are
going to be 20 such VP routers, and each is going to get a continual
1Mbps stream of registration packets.

This is not counting the work that VP router needs to do in order to
establish the authenticity of those registrations.  As far as I know,
it could only do this by looking up PKI CAs (Certification
Authorities) on a regular basis to ensure the signed registrations
were valid.

There are serious scaling problems per VP router in handling 333
signed registrations per second. That's a lot of crypto stuff to do
just to check the signatures - and a lot more work and packets going
back and forth for regularly checking that the public keys provided
are still valid.

There is also the scaling problem of there being 20 or so of these VP
routers, so the entire Internet needs to handle 20 x 0.5Mbps = 10Mbps
continually just to handle the registration of these 100k I-R PI
prefixes.  Each such prefix requires 100 bits per second in continual
registration activity - 5 bits per second per VP router per I-R PI
prefix.  For each VP router, 5 bits per second on average comes from
each of the typically two IRON routers which are registering a given
I-R PI prefix.

Checking this: If there was a single VP router and a single IRON
router registering an I-R PI prefix, the IRON router would send 1600
bits every 600 seconds. This is 2.66 bits a second.  Since there are
20 VP routers, the figure per IRON router per I-R PI prefix is 53bps.
 Since there are two such IRON routers per I-R PI prefix, each such
IRON router sends 106bps per I-R PI prefix.  With 100k of these I-R
PI prefixes per VP, this is about 10Mbps.  This checks out OK.

I think this is an unacceptable continual burden of registration traffic.

Also, this is just for 10 minute registrations.  I recall that the 10
minute time is directly related to the worst-case (10 minute) and
average (5 minute) multihoming service restoration time, as per our
previous discussions.  I think that these are rather long times.

> specifications from which real code can be and has been

Yes - SEAL is a self-contained protocol, but I still found it hard to
navigate my way within the one document.

> it in the next version of IRON.

This is the sort of example I am hoping you will add.  But first I
think there are two questions I raised which would need to be
resolved before your example would be realistic according to my
understanding of I-R:

  1 - How does an IRON router discover all the IRON routers
      advertising a VP?  The I-R BGP overlay network does not
      provide this, as far as I know.

  2 - Allow for 20 or so routers each advertising the one VP,
      for the purposes of supporting packets from non-upgraded
      networks.

Assuming 2 is accepted, and 1 is somehow achieved, we now have, for
each of the 20 VP routers, 0.5Mbps of registration traffic.  That's a
lot of traffic and a lot of crypto processing to do.

It is no-doubt more efficient than the ~100k or so extremely
expensive BGP routers of today's DFZ fussing around comparing notes
about 300k prefixes.  However, I don't think it scales as well as an
alternative:

  http://tools.ietf.org/html/draft-whittle-ivip-arch
  http://tools.ietf.org/html/draft-whittle-ivip-drtm

which doesn't have such continual flows of registration, mapping etc.
data, unrelated to the traffic flowing to a given micronet, or to
changes in the ETR to which the micronet is mapped.

> *endpoint* - so, why not call it as such?

I am not suggesting you adopt "ITR" and "ETR" instead of "ITE" and
"ETE" - which I agree are more apt terms.  I was just explaining why,
for now, I will stick with "ITR" and "ETR" for Ivip.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 15 Mar 2010 11:21:39 -0700

Robin,

See below for some follow-up:

> Sent: Friday, March 12, 2010 7:11 PM
> To: RRG
> Cc: Templin, Fred L

There is room in a future update to IRON to improve on this.

> could be high numbers, approaching 16M perhaps, of I-R PI prefixes

Well, this is a tunable knob of course. We could for
example set the length for VPs to ::/36, ::/40, etc.
to reduce the number of PI prefixes per VP.

The tradeoff is in managing a RIB containing a large
number of VPs (which are likely to be quite stable) vs.
managing a large number of PI prefixes per VP (which
require periodic keepalives to maintain). So, given a
routing protocol that can maintain a large number of
VPs in a relatively static topology it seems like a
proper balance of PI prefixes per VP can be found.

> within that part of the address space.

The length of the VPs is a tunable. It may be that there
can be VPs of varying lengths, but I chose to discuss as
all VPs having the same length for simplicity.

> I recall that some IRON routers handle VPs and others don't.  As I

Not quite. All IRON routers by definition connect to the
IRON. So, all IRON routers discover all VPs in the IRON,
and *some* IRON routers also connect to the DFZ. Those
that connect to the DFZ advertise one or a few very short
prefixes (e.g., 4000::/3) that cover the set of all VPs
in the IRON.

> that for each VP something like 20 or more IRON routers need to be

No; those IRON routers that also connect to the DFZ
advertise very short prefixes into the DFZ; they do
not advertise each individual VP into the DFZ else
there would be no routing scaling suppression gain.

> it any information on the full set of IRON routers which handle this VP.

Here, it could be that my cursory understanding of BGP
is not matching well with reality. Let's say IRON routers
A and B both advertise VP1. Then, for any IRON router C,
C needs to learn that VP1 is reachable through both A and
B. I was hoping this could be done with BGP, but I believe
this could only happen if BGP supported an NBMA link model
and could push next hop information along with advertised
VPs. Do you know whether this arrangement could be realized
using standard BGP?

If we are expecting too much with BGP, then I believe we can
turn to OSPF or some other dynamic routing protocol that
supports an NBMA link model. In discussions with colleagues,
we believe that the example arrangement I cited above can
be achieved with OSPF.

> advertised by more than one IRON router.

What we need here is a dynamic routing protocol that
supports an NBMA link model, and the IRON is treated
as a gigantic NBMA link on which all IRON routers are
attached. Maybe BGP won't fill the bill for that, but
other dynamic routing protocols such as OSPF show some
promise.

> address of) either R(x) or R(y).

As above.

> merely "bubbles" rather than something with greater impact!

No worries; they are harmless, and not at all weapons
of war.

>   1 - The IP address of the IRON sending the registration (16 bytes).

You mean in the data portion of the bubble or in the header?
For IPv6-over-IPv4, the bubble does not need to include an IPv6
header; it need only include the IPv4 header, since VET stateless
address mapping allows the IPv6 link-local address to be discovered
by knowing only the IPv4 address. I can't see why an IPv6 address
would also be required in the data portion of the bubble if it can
already be inferred from the IPv4 header?

>   2 - The prefix the IRON router is registering (18 bytes).

Not necessarily 18 bytes; prefix plus length is all that
is needed. For a ::/32, that would be 4 bytes of prefix
plus 1 length byte = 5 bytes. Since IPv6 likes to do
things in blocks of 8, however, let's round up to 16
to be safe.

>       such as this (10 to 20 bytes?).

The SEAL header with a sequence number that also
serves as a nonce is used for this - the SEAL
header plus sequence number length is accounted

>       the public key of the signer.  These will be FQDNs - lets

I honestly do not know how much this would be. I will
take your 50 byte estimation.

> about 150 bytes already.

No IPv6 header; only an IPv4 header (20 bytes) plus a SEAL
header (8 bytes) plus possibly also a UDP header (8 bytes)
for a total of 36.

Add in L2 headers - Ethernet is 46 octets -

I guess you are counting everything from the preamble to the
end of the interframe gap? I come up with 42 (when 802.1Q header
is added), but I'll use your 46 to be conservative.

>  and we are up to 200 bytes.  Multiply by 8 and this is 1600 bits.

I have (36 + 16 + 50 + 46) = 148. So, call it 150 to be
safe, and the guesstimate is midway between your 200 and
the 100 I said initially.

>   1600 x 333 = 532,800 bits/sec ~=0.5Mbps

I get 1200 * 333 = 399,600 bps ~=0.4Mbps

> DITRs or PTRs, they needs to be a lot more than 2 of them per VP.

No, because all that needs to be injected into the DFZ is
one or a few very short prefixes (e.g., 4000::/3). It doesn't
matter then which IRON router is chosen as the egress to get
off of the DFZ, since that router will also have visibility
to all VPs on the IRON.

> more than one of this set of 20 routers.

As above, it is only 2-3 IRON routers per VP; not 20.

> the I-R PI prefixes - as is required for real portability.

Again, DFZ routers on the non-upgraded network would select
the closest IRON router that advertises, e.g., 4000::/3 as
the router that can get off the DFZ and onto the IRON. So,
it would not be the case that all VPs would be injected into
the DFZ.

> 1Mbps stream of registration packets.

Not 20; only 2 or 3. And, it would be less than 1Mbps per
VP router.

> back and forth for regularly checking that the public keys provided

Crypto overhead can be greatly relaxed if the IRON router
performs crypto only for the initial prefix registration
then accepts bubbles without performing the crypto for
subsequent prefix refreshments. This is because, using
SEAL, there are synchronized sequence numbers for blocking
off-path injections of bogus bubbles.

> PI prefixes per VP, this is about 10Mbps.  This checks out OK.

You are off by a factor of 10 here, because there only needs
to be 2 VP routers per VP.

> previous discussions.  I think that these are rather long times.

Well, let's touch on this a moment. The real mechanism
used for multihoming service restoration is Neighbor
Unreachability Detection. Neighbor Unreachability
Detection uses "hints of forward progress" to tell if
a neighbor has gone unreachable, and uses a default
staletime of 30sec after which a reachability probe
must be sent. This staletime can be cranked down even
further if there needs to be a more timely response to
path failure. This means that the PI prefix-refreshing
"bubbles" can be spaced out much longer - perhaps 1 every
10hrs instead of 10min. (Maybe even 1 every 10 days!)

In this way, the PI prefix registration process begins
to very much resemble DHCP prefix delegation.

> navigate my way within the one document.

The IRON document has a lot of room to add more
descriptive text on the architecture. But, the
mechanisms are already specified in VET and SEAL.

>       provide this, as far as I know.

We believe that OSPF with NBMA link model (or equivalent)
could be used.

>       for the purposes of supporting packets from non-upgraded

We don't need 20; we only need 2-3. And, the bubble
interval (aka the "lease lifetime" can probably be
pushed out by a factor of ~100.

> lot of traffic and a lot of crypto processing to do.

Crypto is not needed on each and every bubble;
only on the first bubble.

> changes in the ETR to which the micronet is mapped.

I think we have learned a few things about the scaling,
and there are solutions. Consider now the bubble interval
as being analogous to the DHCP lease lifetime, and scaling
can be greatly improved for (much) longer bubble intervals.

> for now, I will stick with "ITR" and "ETR" for Ivip.

OK - Fred

---

From: Robin Whittle <[email-masked]>
Date: Tue, 16 Mar 2010 22:40:18 +1100

Short Version:    Fred describes a method for handling packets sent
                  from by non-upgraded networks - roughly similar
                  to Ivip's DITRs and LISP's PTRs.  However there
                  are unresolved questions about the commercial
                  arrangements for running these.

                  With this arrangement, my prior assumption that
                  there be 20 or so routers advertising a given VP
                  (because I had understood these were performing the
                  DITR/PTR functions) does not apply.

                  This reduces the the scaling difficulties I had
                  previously mentioned.

                  But there is not yet a clearly defined method of
                  registering with the 2 or 3 VP routers.  We discuss
                  some of the scaling problems and workarounds for
                  them - but a fuller discussion will depend on
                  Fred's choice of registration mechanism.  He is
                  contemplating replacing the overlay network's BGP
                  with OSPF, but maybe there's a way of doing it
                  while retaining BGP.

Hi Fred,

OK.

OK.

OK.

> prefixes (e.g., 4000::/3) that cover the set of all VPs

OK - so these are the I-R equivalents of Ivip's DITRs (Default ITRs
in the DFZ) and LISP PTRs.  In my previous message, I assumed that VP
routers were also advertising their VPs in the DFZ.  I recall I got
this from something you wrote, but it doesn't matter now.

But what are the scaling properties of these routers I will refer to
as being "DITR-like"?

Who runs them?  They are doing work, handling packets addressed to
very large numbers of I-R end-user network prefixes, who are the
parties which benefit.  So I think there needs to be an arrangement
for money to flow from those end-user networks, in rough proportion
to the traffic each DITR-like router handles for each end-user
network.  This is handled in Ivip, but with DITRs which advertise
specific subsets of the MABs (Mapped Address Blocks):

  http://tools.ietf.org/html/draft-whittle-ivip-arch-04#section-8.1.2

I suggest you devise a business case for these "DITR-like" routers -
and give them a name.

They are going to be busy, depending on where they are located, the
traffic patterns, how many of them there are etc.   So they need to
be able to handle the cached mapping of some potentially large number
of I-R end-user network prefixes.

> there would be no routing scaling suppression gain.

I think there would be, since each VP covers multiple individual
end-user network prefixes.  If there are 10^7 of these prefixes, and
on average each VP covers 100 of them, then there are 10^5 VPs and we
have excellent routing scalability, saving 9.9 million prefixes from
being advertised in the DFZ while providing 10 million prefixes for
end-user networks who use them to achieve portability, multihoming
and inbound TE.

> VPs. Do you know whether this arrangement could be realized

Sorry, I can't reliably tell you what can and can't be done with BGP
- I don't try to do anything special with it with Ivip.

Still, if you assume that something could be done with BGP, consider
the potential scaling problems.  Somehow, for every one of X VPs, and
 for every Y IRON routers which handles a given VP, then you want
each IRON router to learn via BGP the address of every one of these
VP-advertising routers, and which VPs each one advertises.  This is
(X * Y) items of information you are expecting BGP to deliver to
every IRON router - so every BGP router needs to handle this
information.

The scaling properties of this would depend on how you get BGP to do
it, and how many VPs there are, and how many IRON routers advertise
the same VP.

> we believe that the example arrangement I cited above can

OK . . . so you are considering using OSPF on the I-R overlay network
rather than BGP.  I can't discuss that without doing a lot of reading
- which I am not inclined to do.  But see below where I propose
methods of doing the registration within the limits imposed by BGP.

> No worries; they are harmless, and not at all weapons

Good!

> already be inferred from the IPv4 header?

I am definitely not going to try to think about mixed IPv4/v6
implementations of I-R.  I can handle thinking about purely IPv4 and
purely IPv6.

OK.

OK.

OK.

OK.

OK.

OK.

Since you have what to me is a new "DITR-like" router plan for
supporting packets send from non-upgraded network, there is no need
for the larger number of VP routers as I assumed in my previous
message.  As long as you have two or three, that should be fine, I think.

There are two reasons an IRON router M might need to know about which
other IRON routers A, B and C advertise a given VP:

 1 - When M has a traffic packet.  (M is either an ordinary IRON
     router and advertises the I-R "edge" space in its own network
     or it is a "DITR-like" router advertising this space in the
     DFZ.)  M needs to tunnel the packet to one of these VP routers.

     The VP router will tunnel it to the IRON router Z it chooses as
     the best one to deliver the packet to the destination network
     and will send a "mapping" packet to M which will cache this
     information and from then on tunnel packets matching the
     end-user network prefix in the "mapping" to Z (or some other
     IRON router like Z, if there were two or more in the "mapping").

     In this case, M needs only the address of one of the A, B or C
     routers.  Ideally it would have the address of the closest one -
     but it doesn't matter too much if it has the address of a more
     distant one.  That would involve a somewhat longer trip to the
     VP router, and perhaps a longer or shorter trip from there to Z.
     (This would typically be shorter than the path taken through
     LISP-ALT's overlay network.)

     After M gets the "mapping", it tunnels traffic packets to Z - so
     the distance to the VP router no longer affects the path of
     traffic packets.

     In this case, BGP on the overlay would be perfectly good - since
     it provides the best path to one of A, B or C - typically that
     of the "closest" (in BGP terms).

 2 - When M is one of potentially multiple IRON routers which
     delivers packets to a given end-user network - packets whose
     destination address matches a given end-user network prefix P.

     M needs to "blow bubbles" (highly technical term from this
     R&D phase of IRON-RANGER) to A, B and C.  The most obvious
     way to do this is for M to be able to know, via the overlay
     network the addresses of all VP routers which advertise a given
     VP.  There may be two or three or a few more of these.  They
     could be anywhere in the world.

     BGP does not appear to be a suitable mechanism for this, since
     its "best path" basic functions would only provide M with
     the IP address of one of A, B and C.

     You could do it with BGP, by having A, B and C all know about
     each other, and with all three sending everything they get to
     the others.  This is not too bad in scaling terms for two,
     three of four such VP routers.

     Then, M sends its registration to one of them - whichever it
     gets the address of via the BGP of the overlay network - and
     A, B and C compare notes so they all get the registration.

     I will call this the "VP router flooding system".

     Later I suggest another alternative which would also work
     with BGP.

If you adopted something like the above-mentioned "VP-router flooding
system" I think you can retain BGP for the overlay network.  This
will tell each IRON router a best-path to one of the potentially
multiple routers which advertise a given VP.  If there are three such
routers for a given VP, known as A, B and C, and if for a given IRON
router, the BGP overlay network gives it a best path to B, then all
is well.  B will tend to be closer than the others.  If B dies or
becomes unreachable, this will cause the BGP overlay network to
withdraw the best path to B and then provide a best path to A or C
instead.

OK.

> it would not be the case that all VPs would be injected into

OK - as per what to me is a new "DITR-like router" arrangement for
handling packets sent from non-upgraded networks.

OK.

> off-path injections of bogus bubbles.

   (I had to Google "bogus bubbles" - what a great little phrase!
    There's no obvious domain name or rock-band or DJ using it.)

I agree, there may be some way of reducing the crypto overhead with
these sequence numbers.  But sooner or later, there would need to be
a check of the PKI to ensure the signature was made with a public key
which is still valid.

> to be 2 VP routers per VP.

Yes - with my new understanding of the "DITR-like" routers.

> 10hrs instead of 10min. (Maybe even 1 every 10 days!)

OK, I am not sure if I ever knew the details of "Neighbor
Unreachability Detection" - but shortening the time for these
mechanisms raises its own scaling problems.

Can you give some examples of how this would work?

> to very much resemble DHCP prefix delegation.

I will pass on this for the moment.

At present, I can see these choices for this registration mechanism:

  1 - Keep BGP as the overlay protocol and use my proposed "VP router
      flooding system".

  2 - Retain your current plan of each IRON router like M needing to
      know the addresses of all the routers handing a given VP (A, B
      and C) which BGP can't do.  So you could:

      2a - keep BGP and add some other mechanism.  Maybe M sends a
           message to the one of A, B or C it has a best path to,
           requesting the full list of all routers A, B and C which
           handle a given VP.  When M gets the list, it sends
           registration "bubbles" to the routers on the list.  This
           needs to be repeated from time-to-time to discover
           new VP routers.

      2b - use something different from BGP which provides all the
           A, B and C router addresses to every IRON router, such as
           M.  This needs to dynamically change as A, B and C die and
           are restarted, or joined by others.

OK.

OK.

OK.

OK.

> can be greatly improved for (much) longer bubble intervals.

OK - but you still need to design a registration mechanism before we
can think in detail about scaling.

OK.

   - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 16 Mar 2010 15:14:33 -0700

Hi Robin,

Thanks for your follow up, and I will try to keep my
responses brief (below):

> Sent: Tuesday, March 16, 2010 4:40 AM
> To: RRG
> Cc: Templin, Fred L

The more I think about it, the more these specialized
VP routers are really just Default Mappers, i.e., the
similar to those discussed in APT. On the IRON, they
advertise "default", and on the DFZ they advertise one
or a few short prefixes (e.g., 4000::/3) that cover all
of the VPs in use on the IRON.

> of I-R end-user network prefixes.

In the case of IPv6, I think whether the IRON Default
Mappers (IDMs) will be very busy depends on how large
the IPv6 DFZ becomes. In my understanding, the IPv6 DFZ
is not very big yet. So, if most IPv6 growth occurs in
the IRON and not in the IPv6 DFZ the packet forwarding
load on the IDMs might not be so great.

> end-user networks who use them to achieve portability, multihoming

That's good, but I think I'd still rather have the
IDMs only advertise the highly-aggregated short prefixes.

> methods of doing the registration within the limits imposed by BGP.

I will think about both routing alternatives more. But, if
we use OSPF in the IRON overlay, routing would work the same
way as at any other layer of RANGER recursion. The list of
IDMs could be kept in the DNS under the special domain name
"isatapv2.net" which I have set aside for this purpose. All
other IRON routers can discover the list of IDMs by simply
resolving the name "isatapv2.net".

> > No worries; they are harmless, and not at all weapons

The term "bubbles" came from teredo (RFC4380). Maybe we can
think of a better term to use for IRON-RANGER?

> implementations of I-R.  I can handle thinking about purely IPv4 and

I choose to think of mixed IPv4/IPv6 for at least three
reasons:

1) We already have global deployment of IPv4, and that won't
   go away overnight when IPv6 begins to deploy.

2) IPv4 is fully built-out, so new growth will come via IPv6.

3) IPv6 addresses can embed IPv4 addresses such that there
   is stateless address mapping between an EID nexthop and
   an RLOC.

>      I will call this the "VP router flooding system".

This is a nice idea. If I get what you are suggesting, each
IRON router that advertises the same VP (e.g., VP(x)) would
need to engage in a routing protocol instance with one
another to track all of the PI prefix registrations. The
problem I have with it is that that would make for perhaps
10^5 or more of these little routing protocol instances as
well as lots and lots of manually-configured peering
arrangements between the IRON routers that advertise VP(x).

For these reasons, I believe it is better for IRON router
M to know about all three of A, B and C and direct bubbles
to each of them. I think we can achieve this using OSPF
with the NBMA link model in the IRON overlay.

Please note: the EID-based IRON overlay is configured over
the DFZ, which is using BGP to disseminate RLOC-based
prefix information. So, it is BGP in the underlay and
OSPF in the overlay - weird, but I think it works.

> Can you give some examples of how this would work?

I want to go back on this notion of extended inter-bubble
intervals, and return to something shorter like 600sec
or even 60sec. There needs to be a timely flow of bubbles
in case one or a few IRON routers goes down and needs to
have its PI prefix registrations refreshed.

>            are restarted, or joined by others.

Right - I am still leaning toward OSPF with its NBMA
link model capabilities. The good news is that the
IRON topology itself should be relatively stable, so
not much churn due to dynamic updates.

> can think in detail about scaling.

Let's forget about the DHCP lease lifetimes analogy
for a bit and get back onto the assumption that the
inter-bubble interval is the mechanism that keeps
PI registrations refreshed in a timely fashion.

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Wed, 17 Mar 2010 11:30:27 +1100

Short version:   Further discussion on "DITR-like" routers, now
                 called "IRON Default Mappers" (IDMs), how an IRON
                 router registers an end-user network EID prefix
                 with multiple VP routers, and whether to use OSPF
                 rather than BGP for the IRON overlay network.

Hi Fred,

> of the VPs in use on the IRON.

This is different from what I understood from your previous message.

I understood there was a subset of IRON routers which we call "VP
routers".  Each such "VP router" advertises in the IRON network (the
tunnel-based overlay network between all IRON routers, currently
implemented with BGP) a VP (Virtual Prefix).  There are typically two
or perhaps more such routers advertising a given VP.  Each such VP
router may also advertise other VPs, but for this discussion, let's
think of IRON routers A, B and C all advertising VP "P" on the IRON
network - and for the purposes of discussion not advertising any
other VPs.

After your msg06274, when I wrote msg06278, I understood that the VP
routers also advertised their VPs in the DFZ, and that this was the
mechanism by which I-R supported packets sent by hosts in
non-upgraded networks.  It doesn't matter now why I thought this.

>From the most recent pair of messages, your msg06305 and my msg06315,
I thought that this role, which I described as "DITR-like" was
performed by a subset of IRON routers which advertise one or a few
prefixes in the DFZ which covers the entire I-R "edge" subset of the
global unicast address space.  This was on the basis of your:

   >> Firstly, if these VP-advertising routers are to operate
   >> properly like DITRs or PTRs, there needs to be a lot more than
   >> 2 of them per VP.
   >

   > it would not be the case that all VPs would be injected into

I assumed that these "DITR-like" routers were not necessarily VP routers.

Here is my understanding on what you just wrote:

> VP routers

I think you mean the "DITR-like" routers are VP routers. Later you
refer to these as "IRON Default Mappers (IDMs)".  I had assumed they
either were not VP routers, or that they need not be VP routers.

> are really just Default Mappers, i.e., the

APT's DMs certainly advertised into their routing systems of the
networks they were located within.  I recall they advertised a set of
prefixes covering all the "edge" end-user (EID) prefixes of any
end-user network which was using an ISP in the same APT island.
There could be multiple APT islands - sets of APT-adopting ISPs which
were linked by direct BGP links and therefore which were able to
share all their mapping information, which was carried over those
direct BGP links.   (If this was extended with tunnels, then all APT
adopting ISPs would be part of a single global APT island, and this
would enable EID space to be split more finely than the IPv4 /24
limit.  With separate islands, any EID prefixes longer than /24 would
need to use the same island.)

I recall that the DMs also advertised these covering prefixes to
neighbouring ISPs - AKA "advertising them in the DFZ".  But this
assumes the DMs were border routers, which I recall was not
necessarily the case.   So based on what I remember about APT, in an
single APT island, the subset of DMs which were BRs would act in much
the same way as LISP's PTRs or Ivip's DITRs, except that with Ivip's
DITRs each such DITR normally only advertises a subset of the total
Ivip "edge" space, while these APT DMs would advertise it all.

If there was a single global APT island, then all the DMs which were
BRs would advertise in the DFZ the complete set of APT "edge" space.
 I understand from what you just wrote that "these specialised VP
routers" (IDMs, below) in I-R are also BRs and that each one also
advertises the complete set of "edge" address space in the I-R system.

However, this part:

> On the IRON, they advertise "default"

makes no sense to me.  I don't recall any IRON router advertising
"default" on the IRON overlay network.  I understand that a VP router
advertises its one or more VPs.

> load on the IDMs might not be so great.

This would only be true if you could convince most networks adopting
IPv6 to adopt I-R at the same time.

OK.

OK.

> think of a better term to use for IRON-RANGER?

OK.  I don't think "bubbles" is appropriate for the registration
methods you have described so far, or that I have suggested.

>    go away overnight when IPv6 begins to deploy.

I agree.

> 2) IPv4 is fully built-out, so new growth will come via IPv6.

I don't agree with this at all.  I think there's plenty of scope for
more growth in the IPv4 Internet.  Fig. 11 at:

  http://www.potaroo.net/tools/ipv4/

shows 130 /8s worth of space is currently advertised.  Fig. 5 shows
this in more detail.  Of the /8s to to 223, a handful can't be used
(127, 0 maybe).  There are still a bunch of /8s which are
unadvertised.  As time progresses, this space will be too valuable to
use internally, probably inefficiently - so I expect quite a lot of
that will be made available and advertised too.

Then there are ways of using space more efficiently, as Ivip, LISP
and probably IRON-RANGER could do, by slicing and dicing it into much
smaller chunks than is possible with the /24 limit on prefixes in the
DFZ.

I think that most growth in Internet usage will occur in the IPv4
Internet for at least the rest of this decade.  The only time it
would make sense to use IPv6 instead of direct IPv4 or IPv4 behind
NAT would be for some service where it wasn't important to be able to
connect to IPv4.  At present, you couldn't sell any such service. I
guess that it may be possible to do this for large IP cell-phone
deployments where there are enough IPv6 services available to do a
reasonable subset of what people want in a hand-held device, and
where tunneling to a server which provides behind-NAT IPv4
connectivity would also be possible.

>    is stateless address mapping between an EID nexthop and

Can you explain this with an example?  I can't clearly envisage what
you mean.

If I am to keep up with mixed IPv4/IPv6 IRON-RANGER, you will need to
explain things with detailed examples.

> arrangements between the IRON routers that advertise VP(x).

Something like this - but I am not sure what you mean by "routing
protocol instance".  I understand that the two or three VP routers
for any one VP "P" do need to cooperate and share their various
registrations.  You could either create a fresh protocol to do this,
or push into service some existing protocol, including perhaps a
routing protocol.

You haven't specified anything other than manual configuration for
how an IRON router becomes a VP router.  VP routers have extra
workload, so whoever runs such a router must have a reason to do
this, probably involving payment of money in some way from the
end-user networks whose EID prefixes are covered by this VP.

If there are two or three IRON routers acting as VP routers for a
given VP, then some organisation is responsible for that VP, is
collecting payments as described above and is therefore the one
organisation driving the existence of these two or three VP routers.
 So manual configuration seems OK to me - I don't think there needs
to be a fancy automated system by which one VP router for a given VP
"P" would auto-discover any other VP router for "P" in the whole I-R
system.  However, these VP routers for the one VP do need to work
together to share registrations, and to quickly detect when one or
more of the set becomes unreachable.

> with the NBMA link model in the IRON overlay.

OK - but I guess that means not running BGP.  I don't know anything
about OSPF or its scaling properties.  BGP has no central
coordination - something which is understandably attractive to many
people.  Does OSPF have central coordination, single points of
failure etc.?

> OSPF in the overlay - weird, but I think it works.

Yes the DFZ uses BGP and the overlay uses . . . originally I-R used
BGP (a separate instance of BGP in each such router).  Also, IRON
routers don't need to be DFZ routers and in many or most cases are
not DFZ (BR) routers - but they all communicate via tunnels which are
carried between networks via the ordinary Internet (using the DFZ).

I guess these tunnels between IRON routers will need to be manually
configured, since they are typically between physically and
topologically nearby routers.

> have its PI prefix registrations refreshed.

OK - I will stay tuned for further details.

> not much churn due to dynamic updates.

OK.  Since the IRON routers have their own IP addresses and are
generally in networks multihomed by existing BGP techniques, then any
outages don't affect the IRON routers' IP addresses or their
tunneling arrangements.  There would still be transitory breaks in
connectivity, before the BGP multihoming arrangements kick in.  If
you could ignore those by some means in the overlay's routing system
(BGP or OSPF) then yes, the IRON routers should be pretty stable.

OK.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 17 Mar 2010 14:56:14 -0700

Hi Robin,

Responding again:

> Sent: Tuesday, March 16, 2010 5:30 PM
> To: RRG
> Cc: Templin, Fred L

OK.

> I assumed that these "DITR-like" routers were not necessarily VP routers.

Correct; these routers (IDMs) may also be VP routers on
the IRON but need not be. So, we have three classes of
IRON routers: 1) VP routers, 2) IDMs, and 3) both.

> either were not VP routers, or that they need not be VP routers.

The latter - IDMs need not also be VP routers, but they
could be.

> advertises its one or more VPs.

Yes; this is new. By having the IDMs connected to the DFZ
advertise "default" on the IRON, other IRON routers that do
not connect to the DFZ can discover a nearby IDM that can
reach the non-upgraded IPv6 Internet.

> IPv6 to adopt I-R at the same time.

Well, now is the time to put forward the case for
handling new IPv6 growth in the IRON instead of in
the IPv6 DFZ. Otherwise, once growth in the IPv6
DFZ takes off and we start to see significant PI
addressing and multihoming, we will eventually
end up in the same boat we are in with the IPv4
DFZ today.

> methods you have described so far, or that I have suggested.

OK. How about Channel Queries (CQs)?

> that will be made available and advertised too.

OK, but how bad would it be if we just let IPv4 address
depletion run out under the current system, then jack up
to IPv6 in parallel to handle PI addressing and multihoming?

OK.

> connectivity would also be possible.

I agree that the IPv4 Internet is not only not going away
but also continuing to grow. But, I still think that users
will want to have both IPv4 (behind NAT if necessary) and
IPv6 as we move forward from here.

> Can you explain this with an example?  I can't clearly envisage what

I mean, if the IPv6 EID FIB includes entries with a next-hop
address such as: 'fe80::5efe:V4ADDR' (i.e., an IPv6 address
with embedded IPv4 address), then V4ADDR can be statelessly
extracted as the RLOC address of the ETR.

> or push into service some existing protocol, including perhaps a

We haven't brought the Virtual Router Redundancy Protocol (VRRP)
into discussion yet [RFC5798], but we might want to consider
looking at this as a way of providing fault tolerance for VP
routers. I'm not sure whether VRRP would also support load
balancing between the multiple routers, but it seems like
fault tolerance is the dominant consideration.

Using VRRP also reduces the "fanout" of VP-advertising routers
to just a single RLOC address, and so makes for less complexity
in ferrying CQs around the IRON.

> end-user networks whose EID prefixes are covered by this VP.

Yes. End-users have to pay either a one-time or
recurring cost for their PI prefixes.

> more of the set becomes unreachable.

VRRP maybe?

> people.  Does OSPF have central coordination, single points of

In this case, central coordination would be through
maintenance of the domainname-to-RLOC mappings for
the FQDN "isatapv2.net". In other words, when a new
IDM comes into existence its RLOC address gets added
to the DNS RR's for "isatapv2.net". In the same way,
when an existing IDM is decommissioned its RLOC address
is removed.

Currently, "isatapv2.net" is registered to me. Do you
trust me to maintain it properly? :^}

> topologically nearby routers.

No manual config needed; the IRON is just a gigantic NBMA
link, and can use automatic tunneling the same as for VET
and ISATAP.

> OK - I will stay tuned for further details.

Bringing VRRP into the consideration could have a
contributing factor to how long the bubble (er, CQ)
interval needs to be.

> (BGP or OSPF) then yes, the IRON routers should be pretty stable.

With VRRP, probably even moreso.

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Thu, 18 Mar 2010 13:50:43 +1100

Short version:   Continuing discussions:

                    Is OSPF suitable for the I-R overlay?  BGP is
                    highly decentralised, but I am not sure this is
                    the case with OSPF.

                    Fred suggests  Virtual Router Redundancy Protocol
                    (VRRP) RFC 5798 for the task of "DEL" routers
                    registering their EID prefixes with a handful of
                    VP routers.  My initial impression is that this
                    won't work because it requires multicast - which
                    I think would be impossible or at least
                    unscalable on a global overlay network as I-R
                    requires.

                    I give some names to the various roles I think
                    IRON routers might have, and consider what
                    combinations of roles might be valid.

Hi Fred,

Continuing our interesting conversation on the design of IRON-RANGER,

> IRON routers: 1) VP routers, 2) IDMs, and 3) both.

I understand the population of IRON routers can be classified
according to their roles.  I am giving some new names to these roles.

I think it is important to invent names for concepts in a new design,
otherwise we have to use phrases which are longer, and might be
written in different ways when they really refer to the same thing.

   DEL Delivers packets to one or more nearby end-user networks, and
       so needs to register the one or more EID prefixes this
       involves with VP routers (VPRs).  (Typically 2 VPRs, but
       I tend to think of 2, 3 or 4 or so for robustness.)

   LFR Local Forwarding Router.  Advertises a few prefixes covering
       all I-R "edge" space in the local routing system of the
       network it is located in. For the purposes of discussion, I
       will assume this is an ISP network, but it could be the
       network of a large end-user network which has its own AS and
       participated in the interdomain routing system.

       This router is not advertising anything outside the ISP
       network - so it is not "advertising I-R edge space in the
       DFZ".

       Any packets addressed to I-R edge space (that is to any
       EID prefix used by an I-R-using end-user network) will
       go to this router rather than to the ISP's BRs.  So this
       router needs to tunnel it to one of the VPRs for the VP this
       EID address is within.  That VPR doesn't have to be the
       "closest" of the two or more VPRs, but if you use BGP in the
       overlay network then it typically will be the "closest" in BGP
       terms - which would be desirable, to reduce path lengths and
       delays for the one or more initial packets which go via the
       VPR.  That VPR will tunnel the packet to the IRON router
       playing the DEL role.  The VPR will also send mapping to
       this LFR-role router which will subsequently tunnel further
       traffic packets whose destination address matches the EID
       prefix in the mapping to one of the IRON routers which are
       playing the DEL role for this prefix.

       So this LFR role involves the IRON router knowing the address
       of at least one VPR for every VP.  It finds this out, in the
       current design, from the BGP best path it gets from the I-R
       overlay BGP system.  However, it doesn't tunnel the packet via
       that overlay system - it tunnels it via the Internet.

       (This means the VPR must use its Internet IP address for its
       VP advertisements on the overlay network.)

   IDM IRON Default Mapper.  As for LFR, except this router is a
       BR of the ISP and advertises all I-R edge space to
       neighbouring ISPs - that is, to the DFZ.

       As you wrote below, IDMs also advertise "default" on the
       I-R overlay ("on the IRON") - but I don't understand the
       purpose of this.

   VPR Virtual Prefix Routers.  These advertise one or more Virtual
       Prefixes (VPs) on the I-R overlay.  Each VP covers multiple
       (thousands to millions in principle) individual EID prefixes,
       each of which is used by an end-user network via one or more
       IRON routers playing the DEL role.

       See LFR above for a description of the responsibilities of
       VPR routers regarding traffic packets.

       The VPR role also involves accepting registrations from DEL-
       role routers for all the EIDs covered by the VP.  The
       mechanisms for doing so are currently undefined, since we
       have been discussing the inability of the BGP overlay system
       to tell each DEL router the addresses of all VPRs for the
       VP which matches the DEL router's EID prefix.  The new
       arrangements may involve (at least I suggested this as a
       possibility) the VPRs for a given VP working together to
       share registration information.  I figure they will be run
       by the one organization, or at least this VPR role for a given
       VP will be controlled by the organization which runs the VP -
       so they will presumably be coordinated in some way.  This
       would probably mean they don't need to automatically discover
       the other VPR-role routers which are handling a given VP.

As best I understand it, any IRON router can perform the DEL role -
its just a matter of somehow configuring it to initiate the
registration process for an EID prefix for an end-user network it
can deliver packets to.

As far as I know, IRON routers are typically not DFZ routers and are
(to a rough approximation) not BRs - so they typically perform the
LFR role as well.   (A BR could still perform the LFR role, by not
advertising the I-R edge space to other ASes - only within its own
network.)  Its just that a router which is not a BR can't advertise
the edge prefixes in the DFZ, and so can't perform the IDM role.

However, a subset of IRON routers are BRs and are also configured to
perform the IDM role.  While a router could perform purely this IDM
role and not advertise the edge prefixes locally, I will assume this
would not be typical.

A VPR need not be a BR.  It need not perform any other roles, but I
guess it typically would perform some, such as DEL.

Assuming that all IRON routers will, or could, perform the DEL role,
here are the various combinations:

   LFR?   IDM?   VPR?   BR?

 0 -      -      -      Maybe    Just playing the DEL role.

 1 -      -      VPR    Maybe    Also playing the VPR role.

 2 -      IDM    -      Yes      Just playing DEL and IDM roles  -
                                 but for some non-obvious reason not
                                 advertising I-R edge space to local
                                 routing system.

 3 -      IDM    VPR    Yes      As for 2, but also VPR role.

 4 LFR    -      -      Maybe    DEL and accepting packets from the
                                 local network too.

 5 LFR    -      VPR    Maybe    As for 1, but also accepting packets
                                 from the local network.

 6 LFR    IDM    -      Yes      As for 2, but also accepting packets
                                 from the local network.

 7 LFR    IDM    VPR    Yes      As for 3, but also accepting packets
                                 from the local network.

OK.

> reach the non-upgraded IPv6 Internet.

Assuming all IRON routers are IPv6 routers, why would they need to
find another IRON router via the overlay network which could deliver
packets to any IPv6 address?

I think the reasoning for this must come from your mixed IPv4 / IPv6
plans, which I have tried to avoid thinking about so far.

Can you explain more about your vision for this?

> end up in the same boat we are in with the IPv4

OK.  But I still prefer Ivip for IPv6 since it will be able to give
end-user networks, or their appointees, real-time control of
tunneling behavior.  This will be advantageous for real-time
responsive inbound TE and for quickly getting all traffic packets to
the newly selected TTR (Translating Tunnel Router) in TTR Mobility -
so the MN can quickly drop the tunnel it made to the previous TTR.

> OK. How about Channel Queries (CQs)?

I don't see any "channels" and it doesn't look like a "query".

In my nomenclature, it is a DEL router registering an EID prefix (I
think this is the term you use in I-R) with a VPR because this VPR is
one of the typically two or more VPRs which handle this VP.

What about "EID Registration Message" - ERM?

> to IPv6 in parallel to handle PI addressing and multihoming?

I am interested in exploring the IRON-RANGER design - including for
mixed IPv4 and IPv6, because I find this stuff generally interesting
and a good way to learn about scalable routing.  Maybe at the end of
this process I might think that IRON-RANGER is practical and in some
ways desirable compared to Ivip or LISP, which are the only two I
consider potentially practical or desirable at present.  (msg06219)

A likely outcome is that this process will prompt me to think of
improvements to Ivip - since previous improvements to Ivip came from
thinking about other proposals, not from a conscious effort to
improve Ivip.

However, I think it is wildly unrealistic to assume that IPv4 will
die or become anything but *the* Internet everyone relies upon for a
very long time, perhaps forever.  I am not saying this is a good thing.

If you can articulate your vision for mixed IPv4 and IPv6 IRON-RANGER
operation, I can go along with it.  But I don't believe at all that
IPv6 will take over from IPv4 for most end-users before 2020.  As I
mentioned, there's still a lot of unused advertised space - and (I
assume) unused unadvertised - global unicast IPv4 address space.

I can't envisage a situation where it will be better to sell ordinary
(non-mobile) users purely an IPv6 service, without even behind-NAT
IPv4 connectivity, than to sell them a service which is either a
single global unicast IPv4 address or behind-NAT IPv4.

Mobile users could be different, since many functions and services
suitable for hand-held cellphone-like devices could be done via IPv6
- and since there would always be an option to tunnel through IPv6 to
an IPv4 NAT box so people can run client-style IPv4 applications on
their MN when they want to.

>> smaller chunks than is possible with the /24 limit on prefixes in the

So to me, a successful implementation of IRON-RANGER would be as good
as Ivip or LISP in enabling really high levels of address utilization
in IPv4.  This will considerably extend the ability of IPv4 to handle
new users, including new end-user networks which need real global
unicast space (not behind-NAT) because they are running servers.

> IPv6 as we move forward from here.

At present, there's only one scenario in which I can imagine there
being a real demand among non-mobile customers for IPv6.  Let's say
that one or more large mobile phone companies decides to make their
new, or existing, 3G systems work with each MN having its own global
unicast IPv6 address (or perhaps /64).   This would enable direct
host-to-host connectivity between any of these MNs.  (Though carriers
typically want to avoid this, to stop people running VoIP and instead
to use their voice call services, for which they charge more than
they can for basic IP connectivity).

Now let's say there are hundreds of millions or billions of these
MNs, each with its own global unicast IPv6 address.  That address
could be stable as long as the MN is in the one carrier network.  If
it roams to another network, it would probably get another address.
However, the TTR Mobility system would fix this - and give each MN
its own permanent /64, no matter how it connected to the Net, as long
as it is via IPv6.  (I do not currently plan any connections between
Ivip or TTR Mobility for IPv4 and IPv6 - best to keep them as
separate systems.)

In this situation, people on non-mobile networks would have a genuine
reason to get native IPv6 connectivity.  Firstly, they might want to
sell or give services to these MN users.  Secondly, from home, they
might want to run a web-cam, file sharing, VPN or whatever which the
MN could access directly, on a host-to-host basis, without mucking
around with IPv4.

So I can imagine this trend happening - but only once there are a
substantial number of ordinary users with native IPv6 connectivity.
I guess this is most likely to occur with cell-phones.

> extracted as the RLOC address of the ETR.

So the "mapping", which the LFR-role and IDR-role routers get from
the VP router is actually telling them to tunnel subsequent traffic
packets to an IPv4 address?   That would only work if every LFR-role
and IDR-role router had IPv4 access - unless you were to establish
special routers to act as gateways for delivering to IPv4 addresses,
which is not out of the question.

Also, an IPv6 VPR would need to be able to do the same thing - tunnel
an IPv6 traffic packet to a DEL-role router which is actually on an
IPv4 address, but which is nonetheless delivering packets to an
end-user network which uses an IPv6 EID.

This could be done, I guess, but there are messy PMTUD problems to
solve.  I prefer not to think about such things, but for now can
imagine you might want to do this, and that you could devise a way of
doing it.

> fault tolerance is the dominant consideration.

I agree - fault tolerance is more important than load balancing at
this stage of the design, though some form of load balancing might be
possible and desirable too.

I don't want to try to read this RFC in order to imagine how it might
work with I-R, so if you can describe how it would work, that would
be good.

> in ferrying CQs around the IRON.

But if all VPRs are on the one IP address, this would radically alter
the nature of the overlay network.  Also a single router might be VPR
for multiple VPs - so I can't see how this would work.

A quick look into this RFC:

  http://tools.ietf.org/html/rfc5798#section-5.1.1.2

indicates that it relies on multicast.  I think VRRP is intended for
multiple routers in a single local network, where multicast could be
done.  I can't imagine how you could scalably implement multicast on
the I-R overlay network.

I think this illustrates our differing design approaches.  I think
you tend to view the subsystems from a very high level - and it if
looks like one might do the trick, you consider it.  I immediately
want to know whether it is possible to do such things, and in this
case, it took me a few minutes with a protocol I had never heard of
to find a "lower level" detail which seems to preclude its use in the
way you intend.

I am not suggesting my approach is always the best - because I think
it is important to brainstorm ideas and think loosely for a while.
Too much "no, it can't be done" thinking too soon results in there
being nothing to explore.

> recurring cost for their PI prefixes.

OK - but what about the costs of running the IDMs, which will handle
widely varying traffic loads from one EID to the next, with these
loads generally having little correlation with the amount of space in
the EID?

>> more of the set becomes unreachable.

Since it appears to involve multicast, maybe not.

It shouldn't be too hard to develop a protocol by which a handful of
VPRs work together.  Maybe some existing protocols can be used as
part of this.

> trust me to maintain it properly? :^}

Sure!

Whoever runs it needs to have some fancy way of recognising or
rejecting attempts to register whatever needs to be in this branch of
the DNS.

But how is OSPF structured?  BGP is flat and egalitarian, with links
between nearby routers all that is required - and of course care
about which prefixes are advertised.

You could chop the whole BGP-based interdomain routing system into
two or more pieces and they would keep running, just fine - although
of course each only with a subset of the prefixes.

I quick look at:

  http://en.wikipedia.org/wiki/OSPF

and the IPv4 RFC:

  http://tools.ietf.org/html/rfc2328#page-19

indicates that a large OSPF network is organised into various areas.
 How would you do this for the IRON-RANGER overlay network?  Don't
OSPF and ISIS require more centralised administration, such as to
structure the whole system into sub-systems and to give certain
routers particular roles, on which other routers depend?

I haven't read the OSPF article, but my impression is that it is a
valuable resource, with Wbenton:

  http://en.wikipedia.org/wiki/User:Wbenton-test

contributing many things, not least a formidable table and diagram of
interdependencies between RFCs.  The diagram looks like it needs it
own routing protocol!

> link, and can use automatic tunneling the same as for VET

But it is important for IRON routers to run their new BGP instance
with neighbouring IRON routers which are generally physically or
topologically close.  Otherwise, the "distance" metrics in the
overlay network won't resemble the real "distance" to the other
routers, and your routers playing the LFR or IDM role won't
automatically discover the address of the "closest" VPR for a given VP.

These tunnels surely need to be manually configured - and that
defines the membership in the I-R overlay network and its structure
for the purposes of its BGP (or OSPF?) control plane.

> contributing factor to how long the bubble (er, CQ)

I regard the whole question of registering EIDs with VPRs as being
undecided until you propose an exact mechanism.

> With VRRP, probably even moreso.

Or with your own purpose-designed protocol involving one, two or a
few more IRON routers in their DEL-roles registering the one EID with
two or maybe a few more VPRs.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 18 Mar 2010 12:15:39 -0700

Hi Robin,

Thanks for the continued discussion:

> Sent: Wednesday, March 17, 2010 7:51 PM
> To: RRG
> Cc: Templin, Fred L

This clearly needs further investigation, which I will
be working from my end. I fully believe there is a good
fit to be found.

>                     unscalable on a global overlay network as I-R

The multicast required is only link-local multicast on
a shared link that connects the multiple VP routers.
The "link" could be an ordinary physical link, or it
could even be a mesh of tunnels of some form. But, the
multicast scope is link-local only - it is not IRON-wide
nor Internet-wide multicast. I think VRRP fits the purpose.

>                     combinations of roles might be valid.

I don't want to add lots of new acronyms for this, as it
rapidly becomes too complex to follow while the situation
itself is really quite simple. Consider that all IRON
routers are IRON border routers (IBRs), in that they
connect zero or more EID-based enterprise networks to
the IRON. Each IBR:

  - participates in the IRON overlay routing protocol
  - advertises zero or more VPs into the routing protocol
  - connects zero or more EID-based enterprise networks to
    the IRON
  - may or may not connect the IRON to the DFZ

I choose to view this latter category as "gateways" from
the IRON to the DFZ, so I will call these as IBGs. So,
we now simply have only IBRs and IBGs.

>        VP advertisements on the overlay network.)

These are roles belonging to IBRs.

>        I-R overlay ("on the IRON") - but I don't understand the

This is the role of the IBG, which is really just an IBR
that also happens to be a "default mapper". Hence, I
would like to deprecate the term "IDM".

>        the other VPR-role routers which are handling a given VP.

Again, any IBR can also be a "VP Router".

> registration process for an EID prefix for an end-user network it

Right.

> the edge prefixes in the DFZ, and so can't perform the IDM role.

What I am calling "Border Router (BR)" is any router that
can be used for getting off the IRON and onto either an EID-
based enterprise network or onto the DFZ. In this sense,
any router that "sinks" EID-addressed packets that do not
belong to either an EID-based enterprise network nor the
DFZ is also considered as a BR.

> role and not advertise the edge prefixes locally, I will assume this

All IRON routers are BRs (IBRs). Some IBRs are also
gateways for getting off the IRON and onto the DFZ.
These are called IBGs.

>  7 LFR    IDM    VPR    Yes      As for 3, but also accepting packets

This gets way too complex, and I believe is greatly
simplified by what I said above.

> packets to any IPv6 address?

Because all IBRs have full knowledge of all VPs advertised
in the IRON, but only some IBRs have knowledge of prefixes
advertised within the DFZ. This latter class is known as
IBGs, and they advertise "default" into the IRON.

> Can you explain more about your vision for this?

My reasons for thinking so strictly about mixed IPv4
and IPv6 was the nice property of stateless address
mapping when only an IPv6 address is known and not the
corresponding IPv4 address. However, with a routing
protocol now in use in the IRON we have state - so, my
rationale no longer applies. With this in mind, IRON
applies equally well for IPv6-EID/IPv6-RLOC, IPv4-EID/
IPv4-RLOC and IPv6-EID/IPv4-RLOC (however, I need to
think more about IPv4/IPv6).

> so the MN can quickly drop the tunnel it made to the previous TTR.

I will have to finally take the time to understand Ivip.
I will try to do so soon so I can converse with you on
more even terms.

> What about "EID Registration Message" - ERM?

I was thinking "Prefix Control Messages (PCMs)", but I like
yours slightly better. I will give it more thought.

> their MN when they want to.

For many of the reasons you have mentioned, I am going
to back down and say that IRON-RANGER can be agnostic to
whatever IPvX/IPvY protocol combination gets used. I
still believe that the expanded address space of IPv6
will eventually steer new growth toward IPv6, but I
won't be so brave as to guess a timeframe for this.

Still, one of the salient features of IRON-RANGER is
support for IPv6 transition.

> unicast space (not behind-NAT) because they are running servers.

Can do.

> I guess this is most likely to occur with cell-phones.

I honestly don't know what the drivers will be, Robin,
but I still believe (and I still believe that the *IETF*
believes) that IPv6 is where we need to go in the long
run. Again, however, I agree with you that IPv4 will
still be around for a very long time.

> which is not out of the question.

Public IPv4 RLOCs that are routable within the IPv4 DFZ
is what I am suggesting.

> imagine you might want to do this, and that you could devise a way of

SEAL should help.

> possible and desirable too.

VRRP says that load balancing is possible, but AFAICT
leaves it out of scope.

> work with I-R, so if you can describe how it would work, that would

I touched on this above, which is just about as deep
as my understanding goes. In an nutshell, with VRRP
each router shares the same IP address, and each
router maintains synchronized state. One of the
routers is chosen as the primary, and the others
are designated as backups. If the primary fails,
one of the backups takes over sort of like an
uninterruptible power supply.

> for multiple VPs - so I can't see how this would work.

No, it doesn't alter the overlay network in any way.

> done.  I can't imagine how you could scalably implement multicast on

No - not multicast over the I-R overlay network;
link-local multicast on an underlying link.

> being nothing to explore.

I'm somewhat amazed by this assessment. I am very much
a "bottom-up" designer by nature, as can be seen in VET
and SEAL. Higher-level architecture descriptions are not
my strongest suit, but I guarantee you that everything I
describe has a path toward something that can actually
be implemented.

> loads generally having little correlation with the amount of space in

Somehow this cost has to be factored into EID prefix
registry business sector's cost of doing business.
After all, if all the EID prefix registries did was
run VP routers and serve up EID prefixes, then the
IRON would be detached from the DFZ and kept apart
from a huge set of content on the Internet. So, it
seems like each EID prefix registry should also be
required to stand up an IBG.

> Since it appears to involve multicast, maybe not.

I'm pretty sure it will work.

> VPRs work together.  Maybe some existing protocols can be used as

I really don't want to require any adjunct protocols
that aren't already standardized.

> routers particular roles, on which other routers depend?

My understanding is that the set of designated routers
determines each OSPF area. The name "isatapv2.net" is
essentially the list of designated routers for the entire
IRON as a single area. But admittedly, I need to do a
deeper dive into OSPF to prove that this is feasible.

> interdependencies between RFCs.  The diagram looks like it needs it

I appreciate all of these links, and will go chase
them down.

> automatically discover the address of the "closest" VPR for a given VP.

Do you mean distance as in hopcount? Because, every IRON
router is a neighbor on the link - i.e., hopcount is 1
always.

> for the purposes of its BGP (or OSPF?) control plane.

Automatic tunneling is the goal I am working toward.

> undecided until you propose an exact mechanism.

The mechanism is periodic transmission of signed router
advertisements with credentials that prove ownership of
the advertised prefixes. These are what I have formerly
called "bubbles", but as discussed above we should
probably try for a new name.

> two or maybe a few more VPRs.

I'd really prefer not to do that if at all possible.
I think VRRP fits.

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Fri, 19 Mar 2010 12:38:20 +1100

Hi Fred,

I will try to use for your simpler arrangement for naming the IRON
routers, but I still find it easiest to think and write in terms of
some roles I mentioned, which have no exact name in your arrangement.

When I referred to "BR" I mean a router in an AS which also connects
to other ASes - so I was referring to its status in the interdomain
routing system, not whether it was a "Border" router from the
perspective of the I-R overlay network.  You use "BR" to refer to the
router's status within the I-R network.

I only spent a few minutes looking at VRRP - and without trying to
understand how it uses multicast, I thought maybe multicast of any
kind would be difficult or impossible on the I-R overlay network.  I
wasn't suggesting this was definitely the case.

You wrote, in part:

> Consider that all IRON

OK.  But those which advertise VPs have many extra responsibilities,
so I think it is important to think of something like a VP Router, or
the VP role or something as a distinct entity which some IBRs or IBGs
have and which the rest don't.

> based enterprise network or onto the DFZ.

"getting off the IRON" implies to me that packets sent along the I-R
overlay would be handled by this IRON router and forwarded to the
EID-based enterprise network (what I referred to as and "EID-using
end-user network") or towards some arbitrary host on the Internet
(IPv4 and/or IPv6?) via forwarding to other routers (the DFZ).

I don't understand this because the I-R overlay network doesn't
handle traffic packets.  It only handles BGP best path communications
between the IRON routers so each IRON router can discover the IP
address (the Internet address and I-R overlay addresses are the same)
of routers which advertise in the I-R overlay some particular prefix.
 The only IRON routers in the I-R overlay which advertise anything
into the overlay are those which advertise a VP.  If two or more
advertise a VP, then with BGP, every IRON router will get a best path
to one of these VP routers, and so find out the IP address of one of
them.

No traffic packets are sent over the I-R network.  When an IRON
router performing the IBG or IBR role tunnels a packet to an IRON
router which advertises a VP in the overlay, the tunnel is via the
Internet.

So I don't understand what you mean by:

   > getting off the IRON and onto either an EID-
   > based enterprise network or onto the DFZ.

(But see potential explanation a few paragraphs down.)

> In this sense,

I don't see how this would be needed.  No IRON router receives
traffic packets on the overlay - the overlay is simply a mechanism by
which IRON routers discover the "nearest" IRON router which is
advertising a VP.  As I wrote previously (and as is quoted below in
points 1 and 2), there are two reasons for doing this.  One is to
tunnel a traffic packet to that VP router.  The other is to register
an EID prefix with that VP router and with the other VP routers which
advertise the VP which covers this EID prefix.

> gateways for getting off the IRON and onto the DFZ.

Oh - you mean the IBG advertises all I-R "edge" prefixes, (perhaps as
one or a few prefixes in IPv6, though probably many would be required
for IPv4) and that this means it acts like an Ivip DITR or LISP PTR -
accepting traffic packets sent by other ASes which lack their own
IRON routers and then tunneling them firstly to an VP router, and
then (after the VP router sends back "mapping") to an IRON router
which can deliver the packet to its destination network.

I wouldn't describe this as "getting off the IRON and onto the DFZ" -
except in terms of the flow of advertisements of routes.  I tend to
think more in terms of the flow of packets rather than the flow of
information about routes to particular prefixes.

> simplified by what I said above.

Yes, but now I have to write "and IRON router which advertises a VP"
rather than "a VPR" which means the same thing - or an "IRON router
which delivers packets to an EID-using enterprise network" rather
than  a "DEL" router.

> in the IRON, 

Yes, but with BGP in the overlay, they only get a best path to one of
the multiple routers which advertise a VP.

> but only some IBRs have knowledge of prefixes
> advertised within the DFZ. 

I don't think any IRON routers need to know what prefixes are
advertised in the DFZ, since they don't forward packets to DFZ routers.

> This latter class is known as

Yes, but this is for the purpose of being like an Ivip DITR or LISP
PTR, as described above.  They don't forward packets to DFZ routers
so as far as I know, they don't need to know what prefixes their DFZ
router neighbours are advertising best paths for.

> rationale no longer applies. 

OK - but isn't "stateless address mapping" what is contemplated below
for these?:

    IPv6-EID/IPv4-RLOC
    IPv4-EID/IPv6-RLOC

> With this in mind, IRON

IPv6-EID/IPv6-RLOC  This is what would happen if I-R was purely
                    for IPv6.

IPv4-EID/IPv4-RLOC  This is what would happen if I-R was purely
                    for IPv4.

IPv6-EID/IPv4-RLOC  I understand this as the ability of the mapping
                    (which the VP router always has, as developed
                    the potentially multiple "DEL" router
                    registrations for a given EID prefix, and which
                    it sends as "mapping" - AKA route redirection -
                    to any IRON router which tunnels a traffic packet
                    to this VP router) to tell IRON routers to
                    somehow tunnel IPv6 traffic packets to an IRON
                    router (this is where I want to use my DEL term)
                    which will deliver them to an EID-using
                    enterprise network which is an IPv6 network, but
                    which presumably doesn't have native IPv6
                    connectivity since this delivering role IRON
                    router ("DEL"!) is on an IPv4 address.

                    So this is to support isolated IPv6 networks
                    which use I-R "edge" (EID) space and which
                    receive their incoming packets via an IPv4
                    service.  (Or potentially a multihomed such
                    network where one or more of its "DEL" routers
                    is on IPv4, rather than IPv6.)

IPv4-EID/IPv6-RLOC  This would be an I-R "edge" using (EID)
                    network which lacked IPv4 connectivity (at
                    least for this particular "DEL" router) and
                    so which accepted incoming packets via such
                    a router on an IPv6 address.

                    Maybe a MN doing IPv4 applications when it is
                    physically connected only to an IPv6 network.

                    Or a non-mobile network which for some reason
                    only has an IPv6 service, but wants to run
                    IPv4 space, using I-R "edge" EID space.

> I will try to do so soon so I can converse with you on

OK - I would really appreciate this.

OK.

> support for IPv6 transition.

OK.  I understand there is potential value in the two crossover cases
mentioned above:

   IPv6-EID/IPv4-RLOC
   IPv4-EID/IPv6-RLOC

but for now I am imagining Ivip development being completely separate
for IPv4 and IPv6.  Before finalising any protocols, I would be keen
to investigate possible interworking - so I am not saying it is
impossible or a bad idea.  Just that in this stage of development I
am keeping the two systems entirely independent.

>> unicast space (not behind-NAT) because they are running servers.

OK.  Ivip will be able to slice and dice IPv4 space into 1, 2, 3, 4,
5, 6, 7 or any integer number of IPv4 addresses in a single micronet.
 It is not restricted to "prefixes".

LISP and IRON-RANGER work in prefixes, but can still do 1, 2, 4, 8
IPv4 addresses, which will be pretty much as good for finely slicing
the space and mapping it to wherever it is to be used.

I predict the vast majority of IPv4 micronets (EID prefixes for LISP
and I-R) will be less than 256 IPv4 addresses.  If you look at the
huge preponderance of /24 currently advertised in the DFZ:

  http://bgp.potaroo.net/as2.0/bgp-active.html

  /19	 18191 	  5.74%
  /20	 22214 	  7.00%
  /21	 22356 	  7.05%
  /22	 28914 	  9.12%
  /23	 28732 	  9.06%
  /24	166028 	 52.35%

I conclude that the great majority of end-user networks find 256 IP
addresses sufficient.  Since this is the smallest number of addresses
they can advertise in the DFZ (and have the best-paths propagated
through the whole DFZ) I believe it is reasonable to assume that many
would be happy with 128 addresses, 64, 32 or whatever.  Perhaps quite
a few would be happy with 1 or 4 addresses.

At present, these /24s only take up a fraction of a percent of the
IPv4 advertised address space, so arguably they are not wasting much
space now, by being forced to be 256 addresses when less would suit
the needs of the advertising networks.  However, a good scalable
routing solution will be catering to many more end-user networks than
those which currently advertise their own prefixes in the DFZ.
Assuming the new kind of "scalable" "edge" space of LISP, Ivip or I-R
has few or no performance problems, then it could be widely used and
be used in just the right quantities required, without wasting much
space.

> still be around for a very long time.

None of us know anything about the future - we have to make do with
educated guesses.

I agree we should plan for widespread IPv6 adoption.  I am only
arguing against assumptions such as:

   IPv6 widespread adoption will being real soon now.

   IPv4 usage is near its peak - so there's no need to plan for
   it to be more widely used, to solve its scaling problem etc.

For an example of the latter position, and implicitly the first, see
Tony Li's recent message:

  http://www.ietf.org/mail-archive/web/rrg/current/msg06192.html

     IPv4 is done.  Over.  Cooked. Fully toast.  It will either
     enter a black market where we deaggregate and no proposal
     will help, or we shift to v6 and v4 is irrelevant.  In
     either case, we're not in time to do anything significant
     for v4.  And we still need a v6 solution, that's clearly
     higher priority.

> Public IPv4 RLOCs that are routable within the IPv4 DFZ

Yes, but if you are able to specify this in the mapping sent by a VP
router to an IRON router which is accepting traffic packets and
tunneling them (initially to the VP router, and then to whichever
"DEL" role router it decides to from the mapping sent by the VP
router in response to this initially tunneled packet), then it will
only work if all these IRON routers (IBR and IBG in your terminology)
can tunnel packets to any IPv4 "RLOC" address.  Yet these are all
IRON routers which are on IPv6 addresses.

They would need either direct IPv4 connectivity to do this, or a
means of forwarding the packet, in tunneled form, to some other IPv6
router which could send them to the IPv4 address of the "DEL" role
IRON router.

Neither of these things are in the current design, as far as I know.

OK.

OK.

I imagine you would want load balancing with generally the "nearest"
VP router being used, when an IBR or IBG router tunnels the initial
one or more traffic packets (before it gets "mapping" from the VP
router.

The use of both nearest and the load sharing would generally make the
system work better.  Also, when one VP router dies, if you had three,
it would only affect on average 1/3 of the IBR and IBG routers.

Load sharing would be vital for scaling purposes - even if VRRP
somehow handled the robustness problem, there's no way you want all
the initial packets for any set of EID prefixes having to go to just
one physical router in the Net.

> uninterruptible power supply.

I can't see how you can have this shared IP address arrangement for
IRON routers which are going to be in different places, and therefore
 in different parts of the topology.  The diverse placement is
essential for robustness - and for the goals of load-sharing with
generally shorter paths.

In the I-R overlay system, each IRON router which advertised a VP
does so giving its IP address as being the same as the IP address it
uses in the Internet.  This is because IRON routers playing your IBR
or IBG roles tunnel traffic packets directly (via the Internet, not
the I-R overlay network) to the VP router.

So I don't see how you could have multiple such routers, which must
be on different IP addresses on the Internet, behaving on the I-R
overlay as if they all had the one IP address.

> No, it doesn't alter the overlay network in any way.

I just wrote about how I can't see how it could work.  At some stage,
 if you adopt VRRP, I guess you will explain exactly how it will work
in the context of the I-R overlay.

> link-local multicast on an underlying link.

I haven't read the VRRP RFC, but I don't understand how you could
scalably do any multicast on the I-R overlay network.  It is a bunch
of IRON routers, using their Internet IP addresses, with tunnels
between them purely (at present, as best I understand it) for the
purpose of handling BGP messages in this overlay network.  No other
packets flow in the overlay network itself.  So I am not sure what
"link-local" would mean in this context

  http://en.wikipedia.org/wiki/Link-local_address

It means within a local, physical, IPv6 network, or within
169.254.0.0/16 for IPv4.

I understand there is an I-R overlay BGP instance for IPv6 and
another one, involving the same (or mainly the same) routers, for
IPv4.  The participating routers use their Internet IP addresses on
these overlay networks, so I can't see how either IPv4 or IPv6
"link-local" addressing or multicast could be done.

> describe has a path toward something that can actually

If I was going to suggest VRRP as a solution, I would also point out
how its mechanisms would work in the intended scenario.  Since it is
not at all obvious how you can scalably do any kind of multicast on
the overlay network, and since VRRP apparently relies on multicast,
and on some other things such as the routers sharing the one IP
address, I would have accompanied plans for VRRP with an explanation
for how these things could in fact be done in the overlay network.

But that's fine - I regard this as a brainstorming phase of design,
so its OK to consider things just because they look like they might
do the job, without assuming that they really can do it.  Even if
they can't, it might lead to a line of thought which turns out to be
of lasting value.

> required to stand up an IBG.

Yes - IBGs are the equivalents of Ivip DITRs and LISP PTRs.  I think
they will need to monitor traffic on these and charge the destination
networks accordingly.  Ivip anticipates this, but so far LISP and
IRON-RANGER don't.

OK.

> that aren't already standardized.

I agree, but maybe there's nothing already in existence which will do
the job, or do it as efficiently as a purpose-built protocol.

OK.

OK.

> router is a neighbor on the link - i.e., hopcount is 1

I meant the "distance" metrics of BGP - each best-path offered by a
neighbouring router is assessed according to the number of ASes it
contains, and then, subject to local policy, the one with the lowest
number of ASes is usually chosen - with this being offered to all the
neighbours, with an additional AS added (or a few, to make it less
attractive, if this is desired).

I assume that in the BGP overlay each IRON router uses as its AS the
AS it is within on the Internet.

> Automatic tunneling is the goal I am working toward.

But if you have 100k IRON routers, how does any one IRON router
decide which small subset of these to create tunnels to?

You can't have all 100,000 IRON routers tunneling to the 99,999 other
IRON routers.

If you are going to use BGP to provide each IRON router with a
best-path to the "nearest"  VP router of several VP routers
advertising a given VP, then you need these tunnels to be with
topologically nearby routers.   I can't imagine how you could do this
automatically.

Also, if I ran an AS with one or more IRON routers, I would want to
manually configure which IRON routers in other ASes each one tunneled
to, rather than trusting some automagic system to do this.  There
will be real flows of BGP messages over those tunnels, and I would
want most or all of them to be with ASes I had a zero-cost peering
arrangement with.

> probably try for a new name.

OK - but depending on your choice of BGP or OSPF I understand you
will devise a mechanism, potentially using VRRP.

> I'd really prefer not to do that if at all possible.

OK - if you can make VRRP do the trick, then of course that is
preferable to devising a new protocol.

Thanks for the continuing discussion.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 19 Mar 2010 12:21:38 -0700

Hi Robin,

> Sent: Thursday, March 18, 2010 6:38 PM
> To: RRG
> Cc: Templin, Fred L

OK; I speak more to this below.

> router's status within the I-R network.

Actually, the IRON BR is a router that connects an enterprise
network to a different enterprise network - whether within
the same AS or in a different AS.

> wasn't suggesting this was definitely the case.

The assumption I have been making is that there is
a shared link in an underlying network over which the
VRRP multicasts are exchanged. This would not affect
the IRON in any way. But, I believe we can relax the
role of VRRP to where it can be considered as only
an optimization and not a fundamental underpinning
of the architecture.

> have and which the rest don't.

Right; that's why I say an IBR can service _zero or more_
VPs.

> handle traffic packets.

Not so - the IRON is used both for control plane and forwarding
plane, and both use ITE->ETE tunneling.

> It only handles BGP best path communications

The IRON is used for:

  - BGP exchanges of VPs so that the next hop addresses
    of all IBRs that hold VPs can be determined
  - initial data packets sent to an IBR that holds a VP
    that covers the EID destination address prefix
  - secure redirection from the VP-holding IBR to an
    IBR that has a more-specific route to the final
    destination
  - subsequent forwarding plane traffic from an IBR's
    ITE to the ETE of an IBR that has a more-specific
    route to the final destination

> router which advertises a VP in the overlay, the tunnel is via the

The IRON is an overlay over the Internet, and it is used
for both control and forwarding plane.

>    > based enterprise network or onto the DFZ.

Hopefully what I said above helped.

> advertise the VP which covers this EID prefix.

Let's say an IBR holds the VP 4000::/32. Of this VP, the
IBR has delegated many ::/56's to customers but many others
remain undelegated (e.g., 4000::/56). Now, suppose the IBR
receives a packet with destination address 4000:0:0:0::1.
Since this destination matches an undelegated prefix, the
IBR must know enough to discard the packet and send back
a "network unreachable" message of some sort. This is all
I am meaning to say when I said that the IBR "sinks" the
packet.

Correct.

> information about routes to particular prefixes.

Flow of packets is what I am talking about. Data packets
traverse the IRON the same as for control message.

> which delivers packets to an EID-using enterprise network" rather

OK, but I really don't want a long list of disassociated
acronyms which are hard for the reader to remember. I'd
really rather stick with a single "base" TLA and have
variants of it based on function. From now on, I will
use IRON Router (IR) as the base acronyom and extend
it with parenthesized functions as follows:

  IR(VP) - an IR that advertises one or more VPs into
           the IRON BGP RIB

  IR(GW) - an IR that connects the IRON to the DFZ

  IR(EID) - an IR that connects a customer EID network

  IR(RR) - an IR that is a BG Route Reflector in the IRON

Note that it is possible for a single IBR to serve
*more than one* of these roles, but I think that it
is sufficient to discuss one function at a time so
I don't think we will need even more complex TLAs.

> the multiple routers which advertise a VP.

The way the IRON BGP is going to work is as follows. There
will be a well-known FQDN, e.g., "isatapv2.net", which
resolves to a list of IP addresses - possibly many. Each
IP address is the RLOC of an IR(RR) per RFC4456. Quoting
from RFC4456, Section 6:

   "In a simple configuration, the backbone could be divided into many
   clusters.  Each RR would be configured with other RRs as Non-Client
   peers (thus all the RRs will be fully meshed).  The Clients will be
   configured to maintain IBGP session only with the RR in their
   cluster.  Due to route reflection, all the IBGP speakers will receive
   reflected routing information."

and this is precisely the arrangement that will be used
for the IRON "backbone". In particular, upon startup each
IBR(RR) resolves the name "isatapv2.net", and forms a full
mesh of BGP sessions with all other IBR(RR)'s.

Then, when each IBR(VP) starts up it likewise resolves the
name "isatapv2.net" and forms a BGP session with one or
a few IR(RR)s that are "nearby" (the IR(VP) can decide
for itself what constitutes "nearby"). The IR(VP)s then
advertise their VPs into the IRON, and the IR(RR)s ensure
that all VPs are reflected to all IR(VP)s.

After all of the VPs have been disseminated to all of the
IR(VP)s, then the IRON-RANGER data packet forwarding and
route optimization can be coordinated in the forwarding
plane. That's it.

> advertised in the DFZ, since they don't forward packets to DFZ routers.

IR(GW)s forward data packets from the IRON into the DFZ.

> router neighbours are advertising best paths for.

Yes they do. Again, the IRON is used for both the control
and forwarding planes.

> OK - but isn't "stateless address mapping" what is contemplated below

Yes, because an IPv4-RLOC address can fit inside of
an IPv6-EID address.

>     IPv4-EID/IPv6-RLOC

No, because an IPv6-RLOC address cannot fit inside
of an IPv4-EID address.

>                     (which the VP router always has, as developed

VP router == IR(VP) in my nomenclature.

>                     the potentially multiple "DEL" router

DEL == IR(EID) in my nomenclature.

OK.

> am keeping the two systems entirely independent.

OK, but from a legacy perspective IRON-RANGER is descended
from ISATAP which deals only with IPv6-EID/IPv4-RLOC.
(ISATAP is itself descended from mechanisms such as
RFC2529, RFC4213, etc.)

>   http://bgp.potaroo.net/as2.0/bgp-active.html

All right. I think I understand the concept of IPv4
micronets now. IRON-RANGER can handle this too.

> router which could send them to the IPv4 address of the "DEL" role

Direct connectivity to the IPv4 Internet is required and assumed.

> Neither of these things are in the current design, as far as I know.

I'll try to make the assumption explicit.

> one physical router in the Net.

As I mentioned from the beginning, I am going to relax the
requirement for VRRP as an underpinning of the architecture.
Instead, VRRP can be used as an optimization if desired, but
the architecture will allow for a single VP to be advertised
by multiple IR(VPs). That would satisfy the scenario you
are describing above.

>From now on, for the base case there will be multiple IR(VP)s
for each VP; each with its own IP address. However, each IR(VP)
could also be cloned using VRRP, so there may also be more than
one IR(VP) per IP address for fault tolerance.

> in the context of the I-R overlay.

Again, I am relaxing the use of VRRP to only being
an optimization.

> "link-local" addressing or multicast could be done.

I touched on this above, so it would only serve to
confuse to try to explain it in more detail here.

OK.

> networks accordingly.  Ivip anticipates this, but so far LISP and

Again with the new terminology, IBG == IR(GW). But,
I'm not sure about any business model that would charge
the end-users specifically for the use of the IR(GW).
I think the business model is for charging the end-users
for the use of the VP, and then amortize the cost of
operating an IR(GW) into the VP costs passed on to
the end-user.

> the job, or do it as efficiently as a purpose-built protocol.

The question is how does an IR(EID) forward EID Prefix
Advertisements (PAs) to multiple IR(VP)s? The answer I
am planning to use is "one at a time". In other words,
I am going to ask the IR(EID) to forward each PA to
each and every IR(VP) that advertises a matching VP.
So, no purpose-built protocol between the IR(VPs).
Does that work for you?

> > deeper dive into OSPF to prove that this is feasible.

OSPF looks like a dead end due to its requirements for
using areas for very large networks. I want everything
to be one big, flat network with a backbone of IR(RR)s.
I think BGP with Route Reflectors fits the bill well.

> AS it is within on the Internet.

The IRs actually use IBGP within the IRON, so everything
is within a *single* AS. (Yes, that's a very big AS, but
that's the way it is.) IR(GW)s use EBGP within the DFZ.

> decide which small subset of these to create tunnels to?

By resolving the name "isatapv2.net".

> You can't have all 100,000 IRON routers tunneling to the 99,999 other

No; the 100k IRs tunnel to one or a few IR(RR)s among
perhaps 100 or so IR(RR)s around the world. Each IR(RR)
is fully meshed to all other IR(RR)s.

> topologically nearby routers.   I can't imagine how you could do this

When each IR that needs to form a tunnel with an IR(RR)
comes up, it checks through the list of all IR(RR) IP
addresses and picks one or a few that are "nearby". It
is up to each IR to decide what is "nearby".

> want most or all of them to be with ASes I had a zero-cost peering

How about all IRs within the same AS using IBGP? That
would keep the AS peering costs at zero always.

> will devise a mechanism, potentially using VRRP.

Right. The IR(EID) will forward a separate PA to each
IR(VP) that advertised a VP that covers the EID prefix.

> preferable to devising a new protocol.

We touched on this several times above. VRRP is an
option, and devising a new protocol is not required.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 19 Mar 2010 16:17:17 -0700

Robin,

I haven't seen anyone commenting on your Ivip proposal
('draft-whittle-ivip-arch'). I myself am just now getting
around to taking a cursory look, but here are some initial
observations. I'm sorry to say that I was only able to get
through to the end of Section 5 on this pass. I will try
to review more later.

Fred

--- begin comments ---

1) In your abstract, you say: "Both involve modifying the
   IP header and require most DFZ routers to be upgraded."
   Somehow, it seems like a very big assumption to think
   that most or even some existing DFZ routers could be
   upgraded to recognize new IP header formats. Indeed,
   it goes against one of the RFC1955 principles of "no
   changes to *most* routers". I will check in the rest of
   the document for what you mean by this, but I am
   concerned when I see what looks like a non-starter in
   the abstract.

2) In the Introduction, it seems that a significant amount
   of the architecture actually appears in other documents.
   I will check further to see if the base document conveys
   enough without having to dig up all of the others.

3) 2nd paragraph in Section 2 says: "...each with a common
   mapping to a single ETR". Why can't the SPI's map to
   *multiple* ETRs?

4) 4th paragraph of Section 2, it appears that MAB in
   Ivip means the same thing as VP in IRON-RANGER?

5) 5th paragraph Section 2, it appears that a MABOC is
   the same as an IRON-RANGER VP provider?

6) 6th paragraph Section 2, "Multihoming end-user networks
   would typically contract a separate company...". Why
   would they not instead directly negotiate the mappings
   with the MABOC themselves?

7) 7th paragraph, Section 2, "...which is typically a
   subset of all MABs in the Ivip system." IRON-RANGER
   gateways each advertise the full set of VPs, but I
   suppose could also be configured to advertise only
   a partial set as well. Is it not possible for the
   DITR to advertise the full set, or do you for some
   reason think it would be best only to advertise a
   partial set?

8) 8th paragraph, Section 2, with the query service, it
   looks like Ivip is standing up its own SPI to RLOC
   mapping service apart from any routing systems or
   the DNS? Would deploying this service be on the
   same level of complexity as for the global DNS?

9) 10th paragraph, Section 2, "Each QSR uses a DNS-based
   mechanism and an additional protocol to discover...".
   Does this mean that the QSR uses the MAB as an FQDN
   in a DNS query? That could mean quite a few MABs in
   the DNS. I think the main difference between this and
   the IRON-RANGER use of BGP to disseminate VP to RLOC
   mappings is that BGP can more readily cope with dynamic
   updates if some RLOCs fail and/or if the VP moves to
   a different RLOC. Also, each IR has full knowledge of
   all VPs in its RIB, so there are no per-packet lookups
   needed. There are probably more differences as well.

   That said, there may well be a use for keeping the
   MABs/VPs in *both* the BGP RIB *and* the DNS. The
   BGP RIB could be used as the full information data
   base that can be accessed in real time, and DNS could
   be used as a check to ensure that the MAB/VP is actually
   "registered" to the ETR thatclaims to own it.

10) 11th paragraph, Section 2, "MABOCs will typically
   charge their customers for each mapping change." This
   arrangement seems like it might be very costly to highly
   mobile SPI prefix holders - why not just a flat-rate?

11) 12th paragraph, Section 2, I am not (yet) seeing an
   exact mechanism whereby the real-time updated full mapping
   database is managed.

12) 16th paragraph, Section 2, I think in your description
    here you are assuming that EIDs are routable within the
    same scope as RLOCs. That would mean a pure-IPv4 or
    pure-IPv6 deployment and not an IPv6/IPv4 one?

13) 18th paragraph, Section 2, I will wait until it comes
   up in later sections, but the notion of modifying the IP
   heard and changing *most* DFZ routers seems onerous.

14) Section 4, paragraph 1, I have doubts about the
   ability to deploy what is being called "Modified Header
   Forwarding".

15) Section 4, paragraph 1, I have strong doubts about the
   use of the sending host's address as the outer headers
   source address. First, ICMP messages coming from within
   the tunnel would not be recognized by the sending host.
   Second, it only works when both the inner and outer IP
   protocols are the same version, i.e., it doesn't work
   for mixed IPv6/IPv4. Also, the BR source address
   filtering can still be done if the inner source is
   different than the outer source, because the outer
   source indicates the previous hop. This is very useful
   for IPv6-within-IPv4 encapsulation, since the previous
   hop can be constructed as an IPv6 link-local address
   that embeds the IPv4 source address.

16) Section 4, paragraph 2, a large part of the motivation
   for MHF seems to be to avoid PMTUD issues. But, PMTUD
   issues can IMHO be handled through the use of SEAL
   with segmentation/reassembly turned off, but with the
   ability to discover and weed out degenerate links.

17) Section 5.3, paragraph 2, it surprises me to see that
   aspects of the mapping system are outside the scope of
   Ivip. Does it mean that there needs to be some adjunct
   protocols or mechanisms at work in a manner that can
   plug into Ivip? I guess I don't really understand this
   business of separation.

18) Section 5.4, paragraph 1, the path MTU determination
   mechanism seems to require actively looking for a reply
   to an explicit probe before the MTU can be adjusted.
   SEAL uses all packets as implicit probes, so there is
   no need to wait for an explicit probe reply and MTU
   limitations can be discovered asynchronously.

19) Section 5.4, paragraph 3, "...and ETRs do not
   communicate at all." - This seems to preclude the
   ability for the ETR to inform the ITR of any error
   conditions, e.g., if the ITR thinks that the ETR has
   the correct mapping when in fact it really doesn't.
   So, there seems to be a need for the ETR to at least
   convey error information to the ITR.

20) Section 5.4, final paragraph, the use of vanilla
   IP-in-IP encapsulation may not interact well with
   load balancing and/or ECMP in the network.

21) Section 5.5, this section seems to assume no
   "recursive re-encapsulation", i.e., where there could
   be a path "ITR(1)->ETR(1)->ITR(2)->ETR(2)-> ... etc."

22) Section 5.7, I am highly skeptical of the MHF approach
   and/or the use of a non-IP protocol type. I just don't
   see it reasonable to expect the majority of DFZ routers
   to update.

23) Section 5.8, I think the goal of unmodified hosts is
   correct, but should be tempered by "hosts are unaffected
   by the use of an Ivip-like routing service in the network.
   I say this, because it is possible that hosts will want
   to upgrade to support adjunct mechanisms (e.g., HIP,
   MIP) that are unrelated to the network routing service.

24) Section 5.11, paragraph 3, IRON-RANGER also does
   do source address filtering at the ETR. That is why
   each potential ITR needs to securely prove to the ETR
   that it is authorized to source packets from a particular
   EID prefix.

25) Section 5.11, paragraph 4, I do not believe the "inner
   same as outer" source arrangement makes any difference
   if the source address can be spoofed. This can be fixed
   by a new mechanism in SEAL whereby the ITR and ETR
   synchronize sequence numbers so that there is no
   chance of an off-path spoofing. Note also that the
   "inner same as outer" approach only works for same
   inner and outer IP protocol, and only works when
   both addresses are globally routable.

26) Section 5.12, about full or parital knowledge of MABs,
   if the number of MABs can be kept manageable (e.g.,
   order of 100k or less) there should be no reason for
   each DITR to not keep full knowledge. The number of
   expected MABs may be different for IPv4 (where a
   significant portion of the address space has already
   been delegated) than for IPv6 (where a significant
   portion of the address space has not already been
   delegated).

-- end comments ---

---

From: Robin Whittle <[email-masked]>
Date: Wed, 24 Mar 2010 20:44:21 +1100

Hi Fred,

Thanks very much for taking a look at:

  http://tools.ietf.org/html/draft-whittle-ivip-arch-04

and for your questions.

> ('draft-whittle-ivip-arch').

(Med) Boucadair wrote some comments on the 03 version, which was a
fresh rewrite:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05661.html

I responded (msg05714) to some of what he wrote but have not yet
replied to all he wrote, because I concentrated on critiquing other
proposals and developing the DRTM approach to Ivip, which
ivip-arch-04 refers to.

> I myself am just now getting

>    concerned when I see what looks like a non-starter in

This is only for the two "Modified Header Forwarding" approaches
which are alternatives to encapsulation:

  ETR Address Forwarding  (IPv4)
  http://tools.ietf.org/html/draft-whittle-ivip-etr-addr-forw-00

    I may alter this to make a whole new header, of the same
    length as the IPv4 header, but with a different type.  That
    way I can use the "evil" bit and so get 31 bits of ETR
    address instead of 30.  Perhaps I could find another bit
    to get all 32.

  Prefix Label Forwarding (IPv6)
  http://www.firstpr.com.au/ip/ivip/PLF-for-IPv6/

The idea is for these to be used in the long-term - 10 to 20 years
from now.  If we could use them earlier, such at introduction, that
would be great - but I am assuming this won't be possible.

Its best to keep these in mind as long-term upgrades to Ivip - since
over long periods of time it will be easy to have all DFZ and other
routers have the new functionality.

Most of the Ivip documentation assumes encapsulation.  To-do: take
this stuff out of the abstract, to shorten it and to avoid people
mistakenly thinking that Ivip requires upgrades to all DFZ routers.

>    enough without having to dig up all of the others.

I tried to make it so you can read Ivip-arch and then Ivip-drtm,
referring to Ivip-glossary if you need to.

The Modified Header Forwarding stuff doesn't matter - it is best to
regard that as a long-term efficiency upgrade.

>    mapping to a single ETR". Why can't the SPI's map to

  http://tools.ietf.org/html/draft-whittle-ivip-arch-04#page-7

For multihoming:

Once the end-user network (or some other organization they appoint)
has real-time control of the tunneling of all the ITRs which are
handling traffic for a given micronet of SPI space, there's no need
for the ITRs to be given a list of multiple ETRs and then to have to
choose between them.

All the CES (Core-Edge Separation) architectures other than Ivip
(LISP, APT, TRRP, TIDR and IRON-RANGER) assume that this real-time
control can't be achieved.  Then, in order to achieve multihoming
service restoration faster than the ability of the mapping system to
get new mapping information to all the ITRs which need it, it is
necessary for the mapping to be more complex.  This requires more
storage in the ITR - but the worst problem is that the ITRs are now
required to choose between ETR addresses on the basis of which ETRs
enable the destination network to be reached.  This scales badly and
is difficult or impossible for an ITR to test, unless it has a
specific address within the destination network to test - which none
of these architectures currently provide.

Also, by providing real-time direct control of ITR tunneling
behaviour from outside the Ivip system, this externalises all the
testing and decision making process.  It requires end-user networks
to detect reachability and make multihoming service restoration
decisions - which means they can do it however they like, without any
need to be restricted by whatever limited functionality we can build
into all ITRs.  A typical multihomed end-user network would pay a
dedicated "Multihoming monitoring" company to test reachability via
its two or more ETRs and to control the mapping accordingly in
real-time.

Another advantage is in scaling.  An ETR might have 10k ITRs sending
packets to it, but it doesn't want to have to tell all 10k ITRs in
some way that the packets have been delivered.  With Ivip, it is up
to the end-user network what the nature of the reachability testing
is, so there are no scaling problems at all.

For load sharing:

You could have multiple ETR addresses for the one micronet.  That
would involve variable-length mapping and extra stuff to do with
giving each ETR some percentage of the total traffic, on average.   I
could do this, but at present I am keeping the mapping and the ITRs
simple by having a single ETR address.  Load sharing via real-time
inbound traffic engineering can still be done, provided the load is
spread over two or more micronets - which will typically be easy to
achieve.  Even a single host could have two IP addresses, and each
could be covered by a different micronet, which could be mapped to a
different ETR address.  Real-time changes to mapping of these
micronets will directly steer traffic via the two or more ETRs and
therefore two or more ISPs.

   (To-do: explain this approach to inbound traffic engineering
    more clearly and make it easier to find in Ivip-arch.)

>    Ivip means the same thing as VP in IRON-RANGER?

They are approximately equivalent.  Both a MAB and an VP are sections
of "edge" space which are administered by a single organization, and
which typically cover the space used by many end-user networks.

In I-R, I think, at least for IPv6, you would have all the edge space
in a single short prefix.  Then your IR(GW) routers (the equivalent
of Ivip's DITRs and LISP's PTRs) would advertise just this short
prefix into the DFZ.   But this raises questions of each such IR(GW)
router doing work for multiple beneficiaries - so who runs the IR(GW)
routers? (As I discussed in my recent message:  IRON-RANGER - Route
Reflectors & DNS?)

In IPv6 Ivip, the SPI space and so the MABs could be intermixed with
ordinary prefixes.  There's no need to make an existing prefix into a
MAB, since there's oodles of fresh space to use instead.

For Ivip with IPv4 and IPv6, generally a single DITR will advertise a
subset of the MABs - because whoever operates this DITR is being paid
to do so by a subset of the total number of MABOCs (MAB Operating
Companies).

With IRON-RANGER in IPv4, you would have your VPs scattered all over
the place, like Ivip MABs, I think, so it would be important not to
have so many VPs, each of which needs to be separately advertised in
the DFZ, as to overly burden the DFZ control plane.

If there were ever 100k MABs, that would be a considerable burden on
the DFZ control plane - but as long as each MAB was serving the needs
of many end-user networks, this is still achieving routing
scalability.  I intend a typical MAB in IPv4 to be a /16 or so and to
provide space for tens of thousands of end-user networks, each with
one or more micronets of one or more IPv4 addresses each.

>    the same as an IRON-RANGER VP provider?

Yes.

>    with the MABOC themselves?

The end-user network (EUN) has ultimate authority over the mapping of
its SPI space - and how that space is broken into individual micronets.

If the EUN has a single site with two ISPs, and therefore two ETRs
ETR-A and ETR-B for its one or more micronets, then in theory, it
could manually or automatically control the mapping of these
micronets to achieve multihoming service restoration.  That will be
allowed, of course, but I expect it would be easier and better to pay
another organisation - a "Multihoming Monitoring" company - to
continually probe reachability of their network via the two ETRs, and
to change the mapping from the usually used ETR-A to ETR-B in the
even that the network was not reachable via ETR-A.  This company's
automatic systems could also switch the mapping back when ETR-A was
working again.

For the EUN itself to detect reachability reliably and then to change
its own mapping may be possible, but when there are network failures
going on, it is probably easiest to let another company not affected
by those failures control the mapping.  Also, the multihoming
monitoring company can test reachability from its network of probing
servers all over the world, which would be better than what most EUNs
could do on their own.

Of course the MABOC might provide these "Multihoming Monitoring and
Mapping Control" services, as part of the package by which the EUN
leases their SPI space.   But it doesn't have to be done by the MABOC
- it can be done by any organisation the EUN gives proper credentials
to.  The EUN would have an arrangement with the MABOC to get a
username and password which enable the Multihoming Monitoring company
to control mapping for a defined subset (perhaps all) of its SPI
space - and the EUN could easily revoke these credentials.

>    reason think it would be best only to advertise a

A DITR could advertise the whole set of MABs.  Someone running a DITR
would probably only want to do so if they had a contract with all the
MABOCs so they were paid for the traffic the DITR handled for each
MAB.  In order for the MABOCs to be happy with this, the usage
information would need to be broken down into traffic per micronet,
so the MABOC could in turn charge their EUN customers according to
the traffic which was being handled.

This probably would not need to be accounting for every byte or
packet - I guess a sampling system would be generally good enough.

>    same level of complexity as for the global DNS?

With Ivip before DRTM (Distributed Real Time Mapping) - which I only
finalised two weeks ago - there was a single, globally coordinated,
but still somewhat distributed mapping system.

With DRTM, there is no one mapping system.  Each MABOC needs to run
DITRs - either directly, or pay some other company to run DITRs for
them.  The DITRs typically need to be scattered around the Net.  Its
up to the MABOC how those DITRs get their real-time mapping changes.
 This need not be part of Ivip, but there could also be standards on
at least one way of doing it.  Since it is an internal matter for the
MABOC, and for any company it hires to run its DITRs, the problem of
secure, real-time distribution of the full set of mapping changes for
each MAB is assumed to be solvable without too much fuss.  There are
only one or two organisations involved, and there are a finite (5 to
50 or so) DITR sites to get the mapping to.  This is a challenge, but
not so difficult as to present a fundamental objection to Ivip or a
barrier to any companies which wanted to do this.

The Internet can get a packet from almost anywhere to almost anywhere
else in about 0.2 seconds, with a pretty high reliability - such as
99% or more on average.  This is pretty good material for making a
real-time mapping distribution system, especially just within one or
two companies.  Also, for this finite number of sites, a DITR-site
company might want to use private network links in order to avoid
potential problems with DoS flooding or any other problems which
makes the Internet difficult for traffic which must be highly
reliable.  Private network links would be more expensive, but there
is only a finite number of DITR sites and they are all money-making
concerns.

Where Ivip standards are required is for ISPs and other large
networks which want to run their own caching ITRs.  They will run one
or more QSRs (Resolving Query Servers), and we need a standardised
protocol by which QSRs can query the new QSA (Authoritative Query
Servers) which will run at each DITR site.   Each QSA only handles
queries for whatever subset of MABs the DITR site handles - but it is
"full-database" for those MABs, getting real-time updates and so not
requiring any delays to look up the mapping anywhere else.  Each QSR
may need to query hundreds or perhaps even thousands of these QSAs,
and this protocol obviously needs to be standardised so all ISPs can
use the one protocol with all the QSAs of all the DITR-companies.

Ivip standardises the protocol between these QSAs and QSDs - and
likewise the protocol between ITRs and QSRs, which can optionally
involve one or more levels of intermediate QSCs (Caching Query Servers).

In a full deployment, the complexity of the entire system of ITRs,
QSCs, QSRs and QSAs, with the real-time pushing of all mapping
changes to QSAs, and real-time pushing of map updates to querying
devices (QSRs, QSCs and ITRs) which may need it, would be at least as
complex and important as the DNS system.  It would be at least as
distributed too.  There is no central system within the whole thing,
except for the DNS arrangements by which each QSR finds two or more
(typically) nearby QSAs for each MAB.

I just suggested you use DNS for a similar purpose in IRON-RANGER.

>    "registered" to the ETR thatclaims to own it.

I won't try to fully describe the DNS-based mechanism here.  I
haven't had time to write it up in draft-whittle-ivip-drtm-01, so
please refer to it in the RRG archives:

   http://www.ietf.org/mail-archive/web/rrg/current/msg06128.html

   "Stage 2 needs a DNS-based system so TRs (QSRs) can find
   DITR-Site-QSDs (QSAs)"

On boot up, each QSR walks through the DNS quickly discovering the
start and end of each MAB in order.  If this was going to take a
while, it could start at four places in the address range and move
forward from each until the whole global unicast space was covered.
The DNS arrangements I have planned will enable a single query to
find that a given address is either the start of a MAB, and then what
the end address is, or that it it is the start of a MAB-free zone,
and then what the starting address of the next MAB (or potentially
the next MAB-free-zone) is.  So the QSR should be able to walk
through these pretty quickly.

The DNS reply for each MAB will also provide addresses of all the
QSAs which handle this MAB.  The QSR will choose two or so of these -
ideally based on closeness, which can be done with a little probing
with ping or traceroute.

>    mobile SPI prefix holders - why not just a flat-rate?

Everyone still seems to think that mobility involves frequent mapping
changes.  This is true for LISP-MS - but that won't be practical for
a number of reasons.  (See the end of:
http://www.firstpr.com.au/ip/ivip/lisp-links/#critiques )

TTR Mobility does not require a mapping change whenever the MN gets a
new address or access network:

  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

The mapping might stay unchanged for a year.  It is not absolutely
necessary to change the mapping, but if the MN moves more than about
1000km or so, then it would be good to tunnel to a TTR which is
closer to its new locality.  Once that is done, the mapping can be
changed to the new TTR.

>    exact mechanism whereby the real-time updated full mapping

This is described in:

Version 01 doesn't have everything, but it has most of the DRTM
system clearly described.  This is explained in terms of the staged
development of services.  DRTM is not a single thing which has to be
built before anything can happen.  It is an approach to providing
portability, multihoming and inbound TE with essentially no ISP
involvement or investment (ISPs only need to forward packets with SPI
source addresses, since EUNs can run their own ETRs).  For TTR
Mobility, DRTM involves mobility without ISPs doing anything at all.

>     pure-IPv6 deployment and not an IPv6/IPv4 one?

I am currently planning Ivip to be two completely independent systems
for IPv4 and for IPv6.  Closer to the time of deployment, we would
obviously consider some kinds of linkage, interdependence etc.  Maybe
there will be a role for Ivip in supporting interworking, transition
etc.

>    heard and changing *most* DFZ routers seems onerous.

Modified Header Forwarding is not at all required for Ivip to be
widely used.  It should be a long-term option to be adopted whenever
we can upgrade all the relevant routers.

>    ability to deploy what is being called "Modified Header

In the long-term future - 10 to 20 years - it shouldn't be hard to
have new protocol (or modified IPv4 header) functionality built into
routers.  The new approaches are not particularly complex, and for
IPv4 only concern the FIB.  For IPv6, there is also some involvement
of the RIB.  In neither case is there any change to the BGP behaviour
of the router, or to the BGP control plane.

>    that embeds the IPv4 source address.

I don't understand the last sentence.  I am not at present trying to
mix IPv4 and IPv6.

This is just the summary of architectural choices.  There is more
explanation at:

  http://tools.ietf.org/html/draft-whittle-ivip-arch-04#section-7.7

The idea is to support existing ISP BR source address filtering
without significant expense or configuration in each ETR.  This
should work nicely, and it would be impossible to do it in any
reasonable fashion if the outer address was that of the ITR.

I explore how LISP might do it, and it seems to be impossible:

  http://www.ietf.org/mail-archive/web/rrg/current/msg06219.html
  17.6.9 - ETR support for ISP border router source address filtering

assuming an ETR in an ISP is also receiving packets tunneled from
ITRs inside that ISP - in which case the filtering applied by the
ISP's BR should not be applied to the decapsulated packets, since
they don't arrive from outside the ISP's network.  The same would be
true for IRON RANGER, I think.

(More below on ETRs supporting source address filtering.)

The sending host won't recognise PTBs from routers in the tunnel -
but nothing short of host modifications could make that work.  My
approach to Path MTU Discovery is intended to cope with all this:

  http://www.firstpr.com.au/ip/ivip/pmtud-frag/

>    ability to discover and weed out degenerate links.

I don't intend to do fragmentation and reassembly for any traffic
packets - other than the few an ITR sends in a special dual packet
format for probing PMTU to a given ETR.  I don't intend to support
DF=1 IPv4 packets longer than some figure like 1440 bytes or a little
less that we could actually encapsulate and send between any ITR and
ETR without fear of fragmentation.  This will involve restricting the
placement of all ITRs and ETRs in IPv4 so they have a PMTU of at
least some chosen figure (below 1500 bytes) from the DFZ, which is
assumed to have some other PMTU somewhat below 1500 bytes.

I think the encapsulation overhead, particularly with IPv6, is more
than sufficient motivation to avoid encapsulation in the long-term.
Being able to forget about PMTUD problems is an additional motivation.

>    plug into Ivip? I guess I don't really understand this

At present, there is nothing in Ivip to say how a MABOC and any
company it hires to run its DITRs - and therefore the QSAs for its
MABs - can or must get mapping in real-time to each DITR site.
There's no point in specifying how it "must" happen, since these one
or more companies can always make private arrangements which are not
visible to anyone else.

It would be desirable to have at least one standardised method of
transmitting the mapping in real-time to DITR sites, and for ways
that a MABOC could transmit its mapping to one or more DITR-site
companies.  It would be highly desirable to provide a standardised
protocol by which any EUN could interact with its MABOC (directly or
through some other organisation) to split its SPI space into
micronets, control the mapping of those micronets and get a username
and password (or use some other delegation system) so another
organization it appoints could control the mapping of one or more
micronets.

Over time, I expect these protocols could be developed.  Regarding
real-time mapping within a system of DITR sites, perhaps the
"Replicator" system I developed for the original version of Ivip
could be used:

  http://tools.ietf.org/html/draft-whittle-ivip-fpr-01

None of this is a part of the main Ivip proposal.

The key point is that with DRTM, the totality of the real-time
mapping system is broken into any number of parallel systems - for
each set of DITR sites - with no need to interwork between them, that
it is reasonable to assume each such system can be realised without
any prohibitive security or scaling problems.

DRTM enables the entire system to be made of a potentially very large
number of independent pieces, such that there is no need for a single
system at all - thereby avoiding a bunch of scaling problems and
concerns over single points of failure, centralized administration etc.

>    limitations can be discovered asynchronously.

Yes, but these probes - a long and a short packet - are only sent at
particular instances where the traffic packet would be in the Zone of
Uncertainty between the currently established upper and lower limits
of what the ITR knows about the PMTU to a given ETR.  Each such probe
will typically reduce the gap between these, until there is no gap.
Then, there is no need for a probe, except occasionally - such as
after a few to ten minutes - to see if the PMTU has changed.

>    convey error information to the ITR.

There may need to be a system by which an ETR or any other device can
complain it is getting a bunch of packets, apparently tunneled by an
ITR, which it doesn't want.  The correct response, if any, would be
for the EUN whose micronet this is - or whoever is currently
authorised to control the mapping - changing the mapping accordingly.

That wouldn't be done by the ITR - but I agree there needs to be some
mechanism for manual or automatic reporting of what appears to be
erroneously tunneled packets.

>    load balancing and/or ECMP in the network.

I agree - I need to research this more.  If this is a concern, then
Ivip's encapsulation approach would use IP and then UDP in some way,
still with the outer source address being that of the original
sending host, but with some UDP port arrangement to keep the ECMP/LAG
systems happy.  This would increase the encapsulation overhead somewhat.

I mention this in section 17.6.8 of:

  http://www.ietf.org/mail-archive/web/rrg/current/msg06219.html

and it is a To-do to add this to the Ivip-arch ID.

>    be a path "ITR(1)->ETR(1)->ITR(2)->ETR(2)-> ... etc."

There won't be anything like this.  If there was, the packet would
come back to ETR(1).

There are two basic ways an ETR could handle the decapsulated packet:

  1 - The ETR will recognise the destination address of the tunneled
      packet as one which it has special knowledge of - and will have
      some means of delivering the packet to the destination network
      without it going to an ITR.  This may be via some direct link
      or a tunnel.  If the ETR is at the site of the destination
      network, then it is only expecting to get tunneled traffic
      packets addressed to this network.

  2 - If the ETR is at an ISP, and doesn't have a specific mechanism
      to get the packets to the destination network, then if the
      ISP's network's routing system has a more-specific route for
      the address space of this end-user network (which would need
      to be covered by a prefix, since routes are always for prefixes
      rather than integer numbers of IPv4 addresses) so the packets
      would be forwarded according to that more-specific route,
      despite also matching a shorter, less-specific, prefix of the
      MAB, which is also advertised in that routing system.  (Or
      if no MABs are advertised, then the default route which leads
      to the DFZ.)

If the ETR is a TTR, then it tunnels the traffic packet by whatever
means to the MN.  The MN establishes a two-way tunnel to the TTR,
including from behind NAT or from an SPI address.

>    see it reasonable to expect the majority of DFZ routers

As mentioned below, over a decade or two it could be done with very
little cost or trouble.

>    MIP) that are unrelated to the network routing service.

OK - HIP or some other Core-Edge Elimination (Locator / Identifier
Separation) architecture would not be affected by anything such as
Ivip, which deals just with IP addresses.

>    that it is authorized to source packets from a particular

I don't recall this being part of our IRON-RANGER discussions.  I am
not sure how you could do this scalably and securely.

>    if the source address can be spoofed.

More on BR source address filtering and ETR support for this:

Let's say there is an ISP with various prefixes of its own and
customer networks which advertise their PI space solely through this
ISP.  One of these prefixes is 44.44.44.0/20.

The ISP sets up its BRs to drop any packet which arrives from another
AS with a source address matching any one of these prefixes -
including this one.  I understand this is typically done with TCAM
(tens and hundreds of thousands of parallel address comparators on a
single expensive, power-hungry, chip).

Let's say an attacker wants to send a packet to some end-user network
(EUN) which is using an ETR inside this ISP's network (including
perhaps an  ETR at this EUN's site, and that the attacker wants to
make it seem as if the packet came from 44.44.44.44, or any other
address in one of the prefixes the ISP had BR source address
filtering for.

The attacker is assumed to be outside the ISP's network.  The BR
filtering stops the attack succeeding for any ordinary packet
entering the ISP's network from another AS ("from the DFZ").

With Ivip's arrangement, the attacker can't succeed, because if it
sends the packet to the ETR with an outer source address other than
the inner source address 44.44.44.44, then the ETR will drop the
packet.  If the attacker tries to send a packet to the ETR with the
outer source address being 44.44.44.44, then it will never get to the
ETR, since the packet will be dropped by the BR.   This will work
fine, and will still allow the ETR to handle packets tunneled from
ITRs in the ISP's site, such as those which really do originate from
the 44.44.44.0 /20 network.

In LISP, the BR filtering doesn't look at the inner packet, so the
ETR receives it from the attacker, in a form which is
indistinguishable from a packet arriving from any ITR outside the
ISP's network.  To gain the same protection on decapsulated packets
which the BR filtering achieves on conventional packets, each LISP
ETR would need to separately filter on the source address of the
decapsulated packet, which is inordinately expensive for more than a
handful of prefixes - and then only apply this to packets whose outer
addresses were not from any of the addresses inside the ISP's
network.  The first task is too expensive and the second is equally
expensive and probably impossible, since it may not be possible to
know all the prefixes within the ISP on which an ITR could reside.

>    This can be fixed
>    chance of an off-path spoofing.

Yes, but how can the ETR function in IRON-RANGER - the IR(EID) role -
know that any particular ITR is "authorised" to tunnel packets whose
source address is of any particular network?  I can't imagine a
scalable way of doing this, much less a scalable way where the
IR(EID) function could do this quickly, without delaying the delivery
of the initial packets, and without making a CPU- and bandwidth-
intensive target for a DoS attack.

>    Note also that the
>    inner and outer IP protocol,

Yes, it wouldn't work with IPv4 packets being tunneled in IPv6
packets or vice-versa - but Ivip doesn't involve such arrangements.

>    and only works when

I think the nature of the source address doesn't really matter.  If
the BR filter accepts an outer source address, then as long as the
inner source address is the same, then the ETR will deliver it to the
destination network.  Of course BRs doing such filtering would
typically drop a packet if its source address was not globally
routable - so the system I have for Ivip will automatically support
this, without the ETR needing to decide what is globally routable or
not.  Likewise, it will allow packets sent from a non-globally
routable address inside the ISP network, which is exactly what it
should do.

>    portion of the address space has not already been

I agree.  The IPv4 MABs are going to pop up in all sizes all over the
address space, because there is a crowded address space.  In IPv6,
there can be MABs, if desired, all in some currently unused short
prefix, and even if there were a million of them, each one could
support billions of end-user networks.

In this way, perhaps each MABOC would have a single MAB.  So there
might be a few hundred or a few thousand at most.

With IPv4, there could be 100k MABs, but there are unlikely to be
anywhere near that number of MABOCs.  There is definitely not going
to be thousands or even hundreds of separate global sets of DITRs,
even if there were tens of thousands of MABOCs, because economies of
scale would quickly kick in and it would be cheaper and more
effective for almost all these MABOCs to pay an existing DITR company
to handle their one or more MABs, rather than each one establish its
own global set of DITRs.

Thanks again for your questions.

  - Robin

---

From: Robin Whittle <[email-masked]>
Date: Wed, 24 Mar 2010 20:58:47 +1100

For reasons unknown I have been unable to get my reply to Fred's
message accepted by the list server.

It is here:

   Re: [rrg] Comments on 'draft-whittle-ivip-arch'
   http://www.firstpr.com.au/ip/ivip/misc/RRG-msg-2010-03-24-b.txt

 - Robin