Subject: RE: (ngtrans) shipworm and anycast Was: Review of draft-ietf-ngtrans-shipworm-03.txt

From: "Christian Huitema" <[email-masked]>
Date: Wed, 30 Jan 2002 08:34:17 -0800

Thomas,

Thanks for the detailed comments. 

The use of an anycast address is really a consequence of two
requirements, stateless shipworm relays and /64 prefix. 

Stateless relays are desirable for at least two reasons, reliability and
scaling. Relays can be made stateless if the mapped IP address and
mapped IP port are embedded in the teredo address, and if the source
address is guaranteed to be accepted by the NAT on the path. As a
consequence, all relays that serve a given teredo user must be
interchangeable, i.e. route the packet with the same IPv4 source
address. Since the relay that will process a given packet will be
selected based on the IPv6 prefix, all the relays that advertise the
same IPv6 prefix must be ready to use the same IPv4 source address.
This, by itself, does not imply that all relays in the world should use
the same IPv4 address and advertise the same IPv6 prefix; it merely
implies that we have groups of equivalent relays, and those relays
within the same group share the same IPv4 address and advertise the same
IPv6 prefix.

The next step in the reasoning is a function of the size of the prefix
that we want to give to the Teredo node. There are really two
interesting types of length, lower or equal to /64, or larger than /64.
If the prefix is lower than a /64, then you can potentially use a single
Teredo prefix for a whole subnet, in much the same way as 6to4; the
Teredo function could, for example, be implemented in a home gateway. If
the prefix is larger than /64, then we run afoul of the requirement of
RFC2374; in practice, this means that each node who want to use such an
address will have to get its own prefix.

If we want a /64 prefix and stateless relays, then we need a /16 service
prefix for any group of equivalent relays. At this point, we have three
choices:

1) Consider that a /16 prefix is a limited resource, and allocate just
one prefix and one IPv4 address world-wide;

2) Use large groups of relays and allocate many /16 prefixes and many
separate IPv4 addresses to many groups;

3) Forgo the /64 requirement and let groups of relays use arbitrary
length prefixes and arbitrary IPv4 addresses.

Option 1 is the current draft. Option 3 really means that a group of
relays is mapped to an IPv6 provider, that this provider runs Teredo
cloud using a network prefix which is a subnet of his local address
pool, and presumably an IPv4 address in its own address pool. Option 2
is an intermediate between (1) and (3), in which there are a set of /16
prefixes to be used by regional coalitions.

Both option (2) and (3) require some amount of client configuration, to
select the appropriate cloud of servers. From an operational point of
view, manual configuration is not an option; we need something
automatic. The classic method for discovery is DHCP, but this won't work
here, since we have to assume that the client is located behind a non
reconstructed NAT that does not know about the service -- if it knew, it
would do something else, e.g. 6to4. This leaves us with a choice between
either using a single IPv4 anycast address and somehow redirecting to
the specific IPv4 address of the appropriate cloud, or using a DNS based
solution, e.g. looking the SRV record associated to "Teredo over UDP" in
the local domain. 

-- Christian Huitema

> -----Original Message-----
> From: Thomas Narten [mailto:[email-masked]]
> Sent: Tuesday, January 29, 2002 7:42 PM
> To: Christian Huitema
> Cc: Brian E Carpenter; [email-masked]; NGtrans List
> Subject: Re: (ngtrans) shipworm and anycast Was: Review of draft-ietf-
> ngtrans-shipworm-03.txt
> 
> Christian,
> 
> Let me jump in on this one issue:
> 
> > 3) A concern about routing stability, and specifically how to
perform
> > ingress filtering in the presence of "anycast" addresses.
> 
> Let me try and explain the concern as I see it.
> 
> There are number of known issues with using anycast addresses. IPv6
> has resisted allowing them to be used as source addresses. But folks
> are using them in IPv4, at relatively controlled settings (e.g.,
> draft-ietf-dnsop-hardie-shared-root-server-07.txt). I think anycast
> can be used safely (even when packets are sourced with anycast
> addresses), but the target environment and possible limitations must
> be carefully understood.
> 
> As I understand it, shipworm relays source all the packets they tunnel
> back to the shipworm client with an IPv4 anycast address. This is a
> single, global, well-known address. Let us assume shipworm is
> successful in the sense that it becomes widely deployed/used.  There
> could be thousands of these shipworm relays sourcing such
> packets. Let's assume further that some significant fraction of the
> traffic carried in IPv4 land is actually shipworm traffic (say 10%).
> 
> Shipworm relays, advertise a single XXXX/16 prefix into the IPv6
> cloud. That means, within the IPv6 cloud, an IPv6 router has
> essentially no knowledge about what the best path is for a particular
> IPv6 packets destined to an IPv6 shipworm destination. In practice, it
> will forward such packets ship to the closest egress point back to the
> IPv4 cloud. Once there, that relay will tunnel it over IPv4. But the
> destination IPv4 address of the tunneled packets could be literally
> anywhere within the IPv4 cloud. They will NOT be topologically close
> to the shipworm relay except by random chance.
> 
> End result, in the IPv4 cloud, you will have a significant amount of
> IPv4 traffic that has an IPv4 anycast source address that is
> essentially meaningless.  When you look at a packet on some arbitrary
> link, you will not be able to tell where it came from. It could come
> from literally any of the shipworm relays.
> 
> This raises alarm bells in my head from an operational side (well, if
> I had one).
> 
> Now throw in ingress filtering. Or rather, suppose somebody somewhere
> decides to enable it for the IPv4 shipworm anycast address. *If* this
> happens, *anywhere*, just *once*, it seems like you now have a
> potential blackhole problem. A packet arriving at some router may be
> coming in on link 1, but because the router thinks the (closer)
> anycast route (to in reality a different relay than the packet that is
> being forwarded) is is through link 2, it will drop the packet. We now
> have a black hole. When this happens depends on how frequently such
> filtering is enabled, where it is enabled, which destinations are
> communicating, how routing changes the flow of packets under varying
> traffic loads, the phase of the moon, etc.
> 
> How will this be detected and tracked down in the real world when it
> happens (recall that the source IPv4 address is meaningless)? Can we
> prevent this from happening, even by "accident", by an ill-advised
> config somewhere? If we can't, it seems like this has the potential
> for creating significant problems in practice, to the point that the
> service will be unreliable enough that it won't work as far as user is
> concerned.
> 
> It is this possible end point that has me very nervous. I hope someone
> can find a flaw in the above analysis or a way to fix it. If we can't,
> I worry that shipworm could actually turn out to be a big step
> backwards in terms of getting IPv6 deployed and used.
> 
> General comment. The use of anycast by shipworm here goes way beyond
> anything that people are using today (or have proposed using) from
> what I can tell. It seems like there are significant architectural
> issues with the notion that the IPv4 source address can just be
> replaced by an anycast address that is shared by thousands of nodes. I
> think we really need to understand what we are getting into before
> recommending such an approach. Will this really not break things as
> they are working today? I'd also like to hear from folks in operations
> about the implications of such a change on their networks.
> 
> Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 30 Jan 2002 16:21:39 -0500

Christian,

I think I understand where you are coming from. I do see the benefits
of keeping servers stateless. 

One of the things I didn't fully appreciate at first is the
requirement that a shipworm packet be able to come back through the
NAT box to the shipworm client, regardless of its IPv4 source
address. Some NATs do this across the board (i.e., let UDP traffic in
from any source so long as the port numbers are recognized as being
part of an active communication), others want the address to match
that used in outgoing communication. So sourcing packets with the IPv4
anycast address is necessary to deal with this aspect of the
problem.

But if relays used their real IPv4 addresses when sourcing packets,
wouldn't you then also have to arrange for return traffic to go
through the same shipworm relay to make sure it then can get through
the NAT device properly (based on the source address) on its way back
ot the shipworm client?  Isn't the problem worse then, in that not
only are relays stateful, but you have to somehow force return traffic
to traverse the same relay? It's not immediately clear how you could
do that.

> one prefix and one IPv4 address world-wide;

This gives us suboptimal routing, which by itself would not be
horrible. But combined with the anycast on the IPv4 side, it seems
like this may cause operational problems.

> separate IPv4 addresses to many groups;

How many would we eventually have?  Who would get to decide who gets
one and who doesn't?  Presumably there are more ISPs than such /16s,
and each may want to provide its own relay service without cooperating
with some other ISP. Seems like same nasty policy issues here, and the
kind of things ICANN feels a need to get involved in. Yikes. Not to
mention somewhat wasteful useage of address space.

Note that this wouldn't solve the anycast issues outright. But it
would seem to limit the scope of where packets from a particular relay
go, reducing the scope of potential problems.

> length prefixes and arbitrary IPv4 addresses.

I'm also not sure we want to go in the direction of leaking more
specific routing information into the IPv6 cloud, which this would
seem to do.  Actually, the same could be said about multiple shipworm
/16s - instead of one prefix for all shipworm routes, you now have
some larger number of routes. How many? Can they be limited? Is it
safe to pollute the IPv6 tables with these routes?

Overall I still don't see a good solution for these issues. Still
thinking though...

Thomas

---

From: Pekka Savola <[email-masked]>
Date: Wed, 30 Jan 2002 23:31:57 +0200 (EET)

On Wed, 30 Jan 2002, Thomas Narten wrote:

Thomas, have you taken a look at RFC3068; it's about about identical
solution for 6to4?  I think the issues are almost the same.  Should we
reconsider RFC3068 too, let go of some "extra requirements" for Shipworm,
or what?

IMO it might make sense to make up our mind about using anycast relays
with 6to4 first, because that's already in standards track, is more widely
understood and has fewer doubtful features.

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords

---

From: Thomas Narten <[email-masked]>
Date: Wed, 30 Jan 2002 17:21:07 -0500

Pekka,

> reconsider RFC3068 too, let go of some "extra requirements" for Shipworm,

I don't understand this comment at all. I just reread 3068. It uses
anycast addresses *only* for destination addresses, i.e, so that one
can easily find a 6to4 relay without special configuration. One can
then just tunnel the packets to the 6to4 anycast address. While
shipworm does do that, that is not where I have an issue.  The issues
I've posted about have to do with using anycast addresses as the
*source* addresses in IPv4 traffic.

Or do I misunderstand?

Thomas

---

From: Pekka Savola <[email-masked]>
Date: Thu, 31 Jan 2002 00:32:52 +0200 (EET)

> *source* addresses in IPv4 traffic.

RFC3068 uses anycast for two things:

 1) 6to4 servers for finding closest relay (configuring default route to 
192.88.99.1)

 2) native clients reaching any 2002::/16 addresses (as it's being 
advertised to the Internet).
   ==> therefore, 192.88.99.1 is used as a source address for IPv6-in-IPv4 
encapsulated packets sent to 6to4 servers (and from there, clients).

You're missing 2), it appears.  I haven't read the draft lately, to see if 
this is sufficiently clear there.

On a 6to4 server, a snippet of a tcpdump packet trail:

00:35:06.506184 212.54.17.42 > 192.88.99.1: 2002:d436:112a::1.ssh > 
2001:670:86:3001::1.3037: P 62992:63296(304) ack 1633 win 13904 (encap)
00:35:06.532328 192.88.99.1 > 212.54.17.42: 2001:670:86:3001::1.3037 > 
2002:d436:112a::1.ssh: . ack 59168 win 34160 (encap)

Here, 212.54.17.42 is the 6to4 server.

---

From: Thomas Narten <[email-masked]>
Date: Wed, 30 Jan 2002 17:50:13 -0500

Pekka,

>  1) 6to4 servers for finding closest relay (configuring default route to 

Agreed.

> encapsulated packets sent to 6to4 servers (and from there, clients).

Please clarify. My understanding/assumption is that packets from the
6to4 server to the relay should have as an IPv4 address the global
IPv4 address of the 6to4 server.

Packets going from a 6to4 relay back to a 6to4 server have as a source
address the proper (non-anycast) IPv4 address of the relay.

Why does it need to be otherwise? (note also that 6to4 relay behavior
is defined in 3056, which predates the anycast document, so it would
seem odd for it to be sourcing packets with such an anycast address).

Is the tcpdump output you are showing from that of a correctly
implemented server?

Thomas

---

From: Pekka Savola <[email-masked]>
Date: Thu, 31 Jan 2002 01:05:17 +0200 (EET)

> IPv4 address of the 6to4 server.

Naturally.
 
> address the proper (non-anycast) IPv4 address of the relay.

Non-anycast address could be used, but this is not the assumed scenario 
by the RFC author *), at least.  With _IPv4_ anycast, I don't see why you 
couldn't (and wouldn't, to identify that the packet came from a relay) use 
the address corresponding to the 6to4 IPv6 anycast address also as source.

*) When the author was suggesting modifying 6to4 security checks such that
any packet coming from IPv4 anycast address would be interpreted to come
from the relay; he automatically seemed made the assumption that existing
relays would do this and would require no modifications.
 
> seem odd for it to be sourcing packets with such an anycast address).

Well, IMO it would be odd to discuss it in 3056: anycast is outside of
scope in that document.  3068 enhances, and one could say updates, 3056
functionality with anycast.

> Is the tcpdump output you are showing from that of a correctly

Yes.  Server and relay are both correct (as far as I can tell, anyway :-).  
Is there something that looks odd to you?  Packets sourced from
192.88.99.1?  It's an address assigned to the relay (actually on loopback 
interface), so why couldn't it be used as a source address?

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 31 Jan 2002 14:22:13 +0100 (CET)

> scaling.

Christian,

I think there are some high-level issues around the philosophy for transition
that I don't think have been discussed.

Stateless relays, as well as the other shipworm aspects like the direct
"short-cuts", makes sense if the goal is to make shipworm be as good as it
possibly can be.

But it isn't clear to me that that is the right thing to do for a transition
mechanism.

I do understand the need to be able to provide IPv6 functionality e.g. to
home users sitting behind NAT boxes. But an alternate approach to
"make it as good as it can be" would be to make the service be useful
enough to present the benefits of global IP addresses and the end-to-end
communication (games or whatever) it enabled with the goal that the home 
users directly or indirectly put pressure on their ISPs to provide IPv6.

I understand that this is rather different than what shipworm tries to
do, but I think it would be very useful to explicitly discuss the tradeoffs
between the two philosophies.

  Erik

---

From: Keith Moore <[email-masked]>
Date: Thu, 31 Jan 2002 10:55:28 -0500

> users directly or indirectly put pressure on their ISPs to provide IPv6.

I'm very much in agreement with this statement.   And while I think it would
be very useful to have a mechanism which allowed folks with NATted networks
or folks behind NATs to have stable IPv6 addresses, I think that shipworm/teredo 
in its current form will do more harm than good.

Keith

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 31 Jan 2002 18:02:42 +0100

> between the two philosophies.

Yes, but not in the shipworm/teredo protocol spec, surely?

   Brian

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 31 Jan 2002 18:35:44 +0100 (CET)

> Yes, but not in the shipworm/teredo protocol spec, surely?

I didn't mean to imply that.
Discuss on the mailing list was what I meant.

Of course, at the end of such a discussion it might be useful to document
the results in the form of some applicability section in a spec.

  Erik

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 31 Jan 2002 14:34:45 +0100 (CET)

>  1) 6to4 servers for finding closest relay (configuring default route to 

The spec is also "missing" 2) as far as I can tell.
Looking for "source" in RFC 3068 doesn't show any place where it
talkes about what the IPv4 source address should be for packets encapsulated
by a relay. It does say in passing something about the IPv4 source address
for packets sent by 6to4 routers but that's it.

So what is the correct thing to do for 6to4?
Are there any benefits to use the 6to4 anycast as a source address for 6to4?

  Erik

---

From: "Tony Hain" <[email-masked]>
Date: Thu, 31 Jan 2002 10:05:35 -0800

> users directly or indirectly put pressure on their ISPs to

The problem with that approach is assuming that the consumer even knows
enough about what protocol is running to pressure the ISP. Consumers
type name-strings, or click on links, then magic happens. The only real
way for them to pressure an ISP is to consume bandwidth (10% tunnel
overhead) that the ISP could have sold to someone else or delayed an
upgrade by 3 months. I would argue the making the presentation of the
IPv6 service to the application 'as good as it can be' is our real goal,
and market dynamics will sort the rest out.

Tony

---

From: Pekka Savola <[email-masked]>
Date: Thu, 31 Jan 2002 23:24:06 +0200 (EET)

> for packets sent by 6to4 routers but that's it.

Ok, it isn't specified, I took it for granted.

(An interesting detail: RFC3056 section 3 second paragraph would require
this for a subset of packets if IPv6 anycast address would not be, well,
IPv6 anycast address that couldn't be used as the source [some relays are
using broken anycast implementation, and they would be required to use
192.88.99.1 as source.)
 
> Are there any benefits to use the 6to4 anycast as a source address for 6to4?

IPv4 anycast could be used to identify the relay routers. (Either from
packets-on-the wire, or in security checks [this is mostly useful only if 
you assume that most IPv4 nodes cannot spoof their sources, or spoofing 
the IPv4 anycast is more difficult])

It appears that there are a few potential drawbacks, pointed out before 
here:
 - RPF checks might fail
   * this happens when RPF is being performed where the path to the 
     closest relay and other relays would be different where RPF is 
     applied.
   * basically this often applies (the list is probably not exhaustive):
     - LAN has a local relay and RPF is applied at first-hop level 
       (note: all LAN's connected to the same router are busted)
     - the organization has its own relay and RPF is at organization-ISP 
       border
     - ISP has a relay and RPF is applied at ISP-ISP or Transit level
     - organization/ISP/.. is multihomed and sees different relays from 
       different connections
 - Tracing where the relay is, is more difficult
   * route aggregator, origin router-id etc. can tell this at some 
     management level
   * ping6 <ipv6-anycast> reveals the native IPv6 address of the relay
     immediately, so I don't think this is significant, at least for IPv6 
     users
 - Others?

Noticeably, RPF does not work well with proper IPv4 "anycast".. I wonder 
whether that's an obstruction we have to be careful about.

(Personally I like static access lists better than RPF, and that will be 
much easier due to aggregation than with IPv4 too.)

---

From: Keith Moore <[email-masked]>
Date: Thu, 31 Jan 2002 21:01:54 -0500

>  I would argue the making the presentation of the

I'd argue the same thing.  But an IPv6 service that doesn't provide stable
addresses isn't much better than NATted IPv4.

Keith

---

From: Randy Bush <[email-masked]>
Date: Thu, 31 Jan 2002 18:07:30 -0800

> addresses isn't much better than NATted IPv4.

if it turns tcp into udp, and has no stable routing, it sure seems a lot
worse to me.

but is sure droll to see extreme anycast fever, far beyond any contamplated
by the most hacker of ops applications, from so many people who argued so
strongly against anycast source addresses so few months ago.

randy