Subject: Re: failure detection
From: Paul Jakma <[email-masked]>
To: Erik Nordmark <[email-masked]>
Cc: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Date: Wed, 31 Aug 2005 12:34:15 +0100 (IST)

Hi Erik,

There's one other question to consider, which I've not really had a 
good answer to (I think I asked Marcelo already ;) ) AFAIR.

How is this going to interact with local site routing policy? The 
ability to express policy in routing typically depends on routing 
being able to, by and large, affect source-selection. Ie that 
applications (or things above the output-IP layer) do not 
unneccesarily go decide what source to use.

To give an example, here's another case of a PA-multihomed site which 
I had (the misfortune[1]) to setup:

A site bought an expensive, and slow, fixed circuit, in order to 
accept inbound SMTP primarily, they also procured a faster DSL 
line[2], but not suitable for hosting inbound services on. The two 
links had different costs, one was very expensive compared to the 
other, in several ways:

- the fixed circuit had much lower bitrate
- the fixed circuit, amazingly, was metered by MB (DSL wasn't)

I had to fix their site when they discovered things didn't "just 
work" (ingress filtering). They wanted all possible traffic to go 
over their cheap/fast DSL obviously.

Solution was fairly obvious:

- source route for the fixed-address range via the slow fixed circuit
- default via the DSL[3]

This worked because:

Other than responses to SMTP connections, all outbound services (SMTP 
outbound, Web proxy, etc.) used unspecified addresses.

Fairly obvious so far :), bear with me.

My question is, how would source-remote n^2 probing shim ever work 
nicely with such multi-cost setups? If shim just happens on the 
source that belongs to the expensive link, it literally could cost 
money, or high-cost for some other definition of cost.

IIRC Marcelo's answer was that shim6 should simply take routing into 
consideration when picking the source(s) to try send probes with.

But that seems very much (to me) either like reinventing the whole 
SAS process which already is implemented elsewhere in the kernel. 
Alternatively, it will require lots of extra layering violations in 
shim6, with shim asking "what sources do you think I could use?" - at 
which point, you could surely just leave the source unspecified and 
let the mechanisms which exist anyway do the work you're going to 
replicate.

Ie, with shim doing probing using different sources:

- Just picking all sources is going to make shim6 really annoying for
   administrators who have inequal-cost site links.

- Manually configuring shim to prefer certain sources would be
   tedious and simply replicating what the kernel already has in its
   routing table (possibly installed dynamically without manual
   intervention)

- Specifying mechanisms for shim to interact with kernel to obtain
   the list of addresses which fit with routing policy and shim's
   desire seems to me to:

    - be a question that can't be answered precisely other than
      "these are the sources I would use for the /best/ route(s)"

    (shim wants to try more than just that if doing n^2 probing, but
     which ones? If just the best, well that can be achieved simply
     by /not/ trying to pick different sources and just not
     specifying an address and hence letting kernel do what it would
     do anyway. If "the best, plus a few more" - which few more, the
     kernel can't tell you from its routing table, do you want the
     lowest cost and the next highest? but what if that's the
     expensive one? there's no way to know.)

   - will require more hooks and 'APIs' between different layers of
     the networking stack. IMHO, that kind of thing can leads to shim
     being hard to, and taking longer to:

     - specify
     - implement

     (particularly when there's talk of other APIs between shim and
      ULP's. I'd strongly suggest following Spencer Dawkin's advice
      and not requiring such, or even investigating this
      much initially.)

--paulj

1. For reasons we won't go into.

2. The site during procurement were told by their provider that fixed 
addresses were only available with "business" class fixed-circuits, 
and not possible with DSL. So they got a quite expensive 128k link 
just for SMTP. sigh.

3. Failover wasn't needed, because they had two VPN links back to 
head office, one using the DSL link, the other the fixed circuit, 
with the former VPN tunnel as the preferred route - keepalives were 
provided either in the tunnel, or dynamic routing protocol ran over 
it - dont remember. If DSL went down they still had their email 
delivered to them, but to the 'expensive' MX for them via the VPN.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Hmmm, look at those eyes.  He's trying to hypnotize me, but not in the
good Las Vegas way.

 		-- Homer Simpson
 		   Mountain of Madness

Delivery-date: Wed, 31 Aug 2005 01:25:48 +0000
Date: Wed, 31 Aug 2005 02:24:58 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Erik Nordmark <[email-masked]>
cc: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Tue, 30 Aug 2005, Erik Nordmark wrote:

> My point is that the correlation is for the chosen interface. If 
> there are two different first-hop routers out the same interface, 
> then which router is used does not effect the next-hop.

affect the /source/ right? :) Yes.

> So I think we are in violent agreement on this one.

Well, that depends.

On Solaris, which I suspect you're accustomed to ;), a seperate 
logical interface structure is kept for each seperate address. So 
resolving a route to a next-hop and hence an interface implies 
automatically you have only one address to use as your source 
(Solaris then caches that in the kernel socket structure iirc).

On other systems, the resolution may go:

- resolve destination to next-hop
- find the local connected subnet for that next-hop

That latter resolution gives you the output interface and your source 
address (for an unspecified-source socket). An interface may have 
multiple addresses, there may be multiple next-hops out that 
interface, on different next-hops and the next-hop used /does/ 
determine the source used (if appropriate).

In both cases though, the step to resolve next-hop to interface 
involves looking at local addresses (the interface follows from 
that). I'd have to go delve further into sources to see for sure, but 
I don't see how you can get from next-hop to interface without taking 
local addresses into account. (be it expressed in terms of 
connected-subnet routes, or by looking through a list of local 
addresses).

> Even if you extend your favorite IGP to carry information that 
> helps the routers do source address based exit selection, it would 
> be nice if you didn't have to fork-lift upgrade all the routers in 
> the routing domain to know about this new feature.

And you wouldn't, only those involved in diverging paths to the 
relevant border routers. If the border routers share link(s), then 
only the border routers need know.

> encapsulate the packets across any routers that are between them 
> means that the non-exit routers don't need to route packets on the 
> source address.

Indeed.

But, ultimately, flooding required source-routing knowledge seems 
easier. Tunnel through unaware parts of your network seems a hack 
till you get your routers upgraded.

> I don't understand how a link-local protocol can be used to carry 
> information from a set of border routers to all the routers in the 
> IGP.

IGPs typically are link-local, yet manage to synchronise routing 
state within an AS. ;) You simply flood the required knowledge onward 
to other links.

> Develop all you want; I have no problem with that.

Fair enough ;)

> My problem is that you want to assume that what you've developed 
> must be deployed in order to deploy shim6. I think shim6 should be 
> deployable with just two hosts on the Internet being able to speak 
> shim6, and not require that the IGP domains those hosts are in have 
> deployed some new IGP and/or router renumbering protocol.

That's a fair consideration, yes.

Note that I don't wish to preclude the more complex host-side probing 
of all addresses. I simply would not want to see that encouraged as 
the norm.

Further, this ISP ingress problem affects more than just shim6. It 
affects all IPv6 communication from such a multihomed site to 
external hosts. Hence why I strongly suspect any IPv6 infrastructure 
that is intended to cope with IPv6 multihoming will ultimately 
acquire 'ISP steering' for routing and RAs regardless of shim6.

Indeed, routers and common Unix OSes have this capability /today/. 
Though, not a protocol to distribute this information to automate 
propogation of such routing information - that'd be nice to have I 
think.

> Still doesn't make the customer's deploy it. If we want a solution 
> that can be easily deployed then ideally it should only require 
> touching one box in the Internet. We can do multi6 solutions with 
> that property as long as it is acceptable that connections fail and 
> need to be restarted; when new communication starts a host can be 
> made to try the different destination and source addresses until it 
> finds a working one.

I think it's complete overkill, but sure, yes. I don't think it 
should be precluded.

However, the site topology you argue should not need to be upgraded 
to work with shim6 likely /already/ has source routing in place to 
allow their v4 and v6 to work /today/. So you're tackling a problem 
which the local site administrator likely has already solved.

> I'm arguing that *requiring* that more than those two boxes are 
> touched (upgraded to some new software, reconfigured, etc) is a bad 
> idea. Making shim6 work better if additional boxes are touched is 
> fine.

Ok.

>> And surely it's more sane to have border-routers be able to affect which 
>> prefixes are posted as 'available' by interior routers within an AS, to the 
>> hosts *before* they ever try pick a source?
>

> I think it is a useful optimization, but we shouldn't make it be a 
> requirement that such technology is deployed before shim6 can be 
> deployed.

Ok, yes: I agree with you.

I'll tell you what my fear is: it's sort of the converse of yours.

Where you fear I want to require IGPs, etc.. to be upgraded for Shim6 
to work - my fear is that shim6 will end up specifying that 
source-address probing will be a specified as a MUST regardless of 
what other (and possibly better) means has a site has to indicate 
prefix availability and steer packets to the right ISP.

> You seem to argue that shim6 should not be allowed to have the 
> capability to explore all n^2 address pairs,

Apologies, I've intended to argue that n^2 should not be /mandated/, 
however that implementations /may/ do so, if needs be.

> which doesn't make sense to me. Having that capability is useful to 
> remove the constraints on the order of the deployment of shim6 and 
> source-based routing, and the end result is still an efficient 
> exploration of the possible paths to which shim6 can failover.

Personally, I think n^2 would be a short-term hack until routing 
catches up (if indeed routing gets there /before/ shim6 - the ISP 
ingress problem will become apparent to /any/ PA-multihomed site, 
regardless of shim6, and you don't need n^2 probing to solve it - 
even today with IPv4).

> As well as the end2end argument; the network can't reliably 
> indicate which address pairs are known to work and which are not 
> known to work. As a result the hosts must be prepared to explore 
> the different address pairs when the current address pair is no 
> longer working.

Sure, but to take this "the hosts can find working paths better than 
internet routing" view to its most extreme you could also mandate 
source-routing to try probe different /remote/ paths in the internet. 
But that's not how we do routing in IP networks - we've always left 
it to the next-hop to know their neighbours and appropriate paths 
better than "we" do. If there's a problem in routing, fix it. :)

So no, I don't accept hosts must be prepared to do this, no more than 
they must today. I don't accept it's required as anything other than 
a hack for networks which are broken anyway.

However, I don't think it should be /precluded/ from shim6, no. I 
just have a fear that it will be specified as a default behaviour.

> Ah - I should have asked "did it work without any manual 
> configuration?" ;-)

I gave the URL to the script ;) - that was run on the relevant 
'routers'. Other than configuring the hosts with addresses (which has 
to be done anyway), nothing.

It's probably worth mentioning that that site eventually moved to BGP 
and PI ;). However the PA addresses and source-routing remained in 
place, and continued to be published in DNS for a while.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
"A computer is a state machine.
  Threads are for people who can't program state machines."

 	- Alan Cox

Delivery-date: Wed, 31 Aug 2005 00:39:25 +0000
Date: Tue, 30 Aug 2005 17:39:03 -0700
From: Erik Nordmark <[email-masked]>
To: Paul Jakma <[email-masked]>
CC: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

Paul Jakma wrote:

>> But here is where the out-of-the-box behavior of the hosts doesn't 
>> match your expectations. The host will pick a source address that is 
>> appropriate for the network interface,
> 
> 
> Right.
> 
>> but apart from that there isn't any correlation between the source 
>> address and the 1st hop router it is using.
> 
> 
> In practice, there likely is a very strong correlation between the 
> output interface and the address(es) on that interface, the latter of 
> which were used to decide which interface is appropriate for the chosen 
> next-hop. So in practice, there's a very strong correlation between 
> next-hop used and source address selected for addr-any sockets sending 
> packets that will go via that next-hop.

My point is that the correlation is for the chosen interface. If there 
are two different first-hop routers out the same interface, then which 
router is used does not effect the next-hop.

So I think we are in violent agreement on this one.

> Tunnels between routers in the /same/ IGP AS does not seem ideal. Just 
> distribute the required source-routing information to those routers 
> which need to be aware of it, eg using Opaque-LSAs. Have each border 
> router originate a "I am a packet-sink for address-range X" LSA. Would 
> require OSPF though - which isn't ideal.

Even if you extend your favorite IGP to carry information that helps the 
routers do source address based exit selection, it would be nice if you 
didn't have to fork-lift upgrade all the routers in the routing domain 
to know about this new feature.

Having the exit routers implement the new IGP thingy and encapsulate the 
packets across any routers that are between them means that the non-exit 
routers don't need to route packets on the source address.

>> This could be combined with some automatic protocol by which the 
>> border routers can inform the interior routers of which prefixes have 
>> failed, and the interior routers can advertise that in their RAs. 
>> (Something like an extension to RFC 2894 "router renumbering protocol" 
>> could be made to work.) However, such a protocol requires some 
>> security configuration, otherwise anybody on the Internet could use 
>> this to reconfigure the RAs your interior routers send.
> 
> 
> One would hope this would this be a link-local protocol at least.. 
> preferably with an HMAC or IPSec AH.

I don't understand how a link-local protocol can be used to carry 
information from a set of border routers to all the routers in the IGP.

>> Thus I don't think we can assume that some such, not yet invented 
>> mechanism for propagating information from border routers to interior 
>> routers, exist.
> 
> 
>> This leads me to conclude that we can't assume that source address 
>> selection can solve the ingress filtering interaction.
> 
> 
> I disagree with this. Just because the required router protocol 
> infrastructure does not exist is not a good reason to try bypass 
> developing such infrastructure.

Develop all you want; I have no problem with that.

My problem is that you want to assume that what you've developed must be 
deployed in order to deploy shim6. I think shim6 should be deployable 
with just two hosts on the Internet being able to speak shim6, and not 
require that the IGP domains those hosts are in have deployed some new 
IGP and/or router renumbering protocol.

> If there's customer demand for IPv6 multihoming (eg cause of shim6 ;) ) 
> and then customers require facilities for steering packets to right ISP 
> according to the source, and require a way to reflect 
> prefix-availability AS-wide, the router vendors will respond.

Still doesn't make the customer's deploy it. If we want a solution that 
can be easily deployed then ideally it should only require touching one 
box in the Internet. We can do multi6 solutions with that property as 
long as it is acceptable that connections fail and need to be restarted; 
when new communication starts a host can be made to try the different 
destination and source addresses until it finds a working one.
But if we want connection survivability, then we must at a minimum touch 
the two hosts at the end.
I'm arguing that *requiring* that more than those two boxes are touched 
(upgraded to some new software, reconfigured, etc) is a bad idea.
Making shim6 work better if additional boxes are touched is fine.

> And surely it's more sane to have border-routers be able to affect which 
> prefixes are posted as 'available' by interior routers within an AS, to 
> the hosts *before* they ever try pick a source?

I think it is a useful optimization, but we shouldn't make it be a 
requirement that such technology is deployed before shim6 can be deployed.

> That prefix-propogation protocol could, at the same time, be used to 
> install the required source-routes in the routers concerned to direct 
> the packets to the border-router appropriate for the prefix concerned.
> 
> That seems achievable and also more proactive and sane than having hosts 
> sending a multitude of packets with every combination of source and 
> destination they have to hand, simply for the reason that "well, the 
> routers don't have that mechanism yet".

When there is a deployed mechanism to convey failed prefixes to the 
hosts, then the effect will be that the hosts will not try the source 
address that are known not to work. Thus as such a mechanism is 
deployed, shim6 would more rapidly find a working address pair.

You seem to argue that shim6 should not be allowed to have the 
capability to explore all n^2 address pairs, which doesn't make sense to 
me. Having that capability is useful to remove the constraints on the 
order of the deployment of shim6 and source-based routing, and the end 
result is still an efficient exploration of the possible paths to which 
shim6 can failover.

> I don't understand why a protocol for /tomorrows/ networks should be 
> trying to hack around a lack of functionality in /todays/ interior 
> routing, when the correct place is obviously to extend capabilities of 
> interior routing (and IPv6 RAs).

Deployment considerations.
As well as the end2end argument; the network can't reliably indicate 
which address pairs are known to work and which are not known to work. 
As a result the hosts must be prepared to explore the different address 
pairs when the current address pair is no longer working.

>> Did it work without any manual configuration of the host?
> 
> 
> *hosts*, and yes it did.
> 
>    ISP1-gw  ISP2-gw
>         |    |
>         |    |
>         R    R
>         |    |
>      -----------
>       | | | | |
>        <hosts>
> 
> The hosts were configured with an address for each PA range, the routers 
> did the magic.

Ah - I should have asked "did it work without any manual configuration?" ;-)

    Erik

Delivery-date: Wed, 31 Aug 2005 00:25:05 +0000
Date: Wed, 31 Aug 2005 01:23:51 +0100 (IST)
From: Paul Jakma <[email-masked]>
Reply-To: [email-masked], [email-masked]
To: Erik Nordmark <[email-masked]>
cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Tue, 30 Aug 2005, Erik Nordmark wrote:

> The idea with *graceful* renumbering is that the admin knows in 
> advance (hours, days, weeks) that the network will be renumbered 
> and that there is some overlap when both prefixes work. The notions 
> of preferred/deprecated/invalid prefixes in RFC 2462 was designed 
> to handle this case.

> In this model communication (including external) continues to work for the 
> deprecated address up until that address becomes invalid.

> The takeaway point is that forcing a deprecated address, by 
> overloading with a notion of "failed", makes it impossible to tell 
> graceful renumbering apart from something which a multihoming 
> solution can repair.

Yes, agreed. A host wouldnt know the difference.

I'm wondering though, given the language in the RFC, why one would 
ever need to distinguish between a prefix no longer preferred due to 
failure and one no longer preferred due to renumbering? Eg, one way 
an administrator of a multihomed site (in lieue of shim6) might deal 
with a failure is to initiate graceful-renumbering ;).

The RFC provides a mechanism to indicate preference. Does it really 
matter for what reason a prefix no longer is preferred? It's a usage 
consistent with the mechanism, regardless of what that mechanism was 
specified for.

The intended use is outside of the mechanism, and not immediately 
important, surely? Nor is the reason for a renumbering that important 
either, surely?

Sorry to harp on about this, just a 'failed' bit has some problems - 
it requires that it wasn't the router<->host link which failed for 
one, and it seems to me that the high-level difference as to why a 
prefix should no longer be used is not that important.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Truth never comes into the world but like a bastard, to the ignominy
of him that brought her birth.
 		-- Milton

Delivery-date: Wed, 31 Aug 2005 00:05:30 +0000
Date: Tue, 30 Aug 2005 17:05:22 -0700
From: Erik Nordmark <[email-masked]>
To: Paul Jakma <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

Paul Jakma wrote:

>>     There is some graceful renumbering going on.
> 
> 
>>       Make sure that new
>>     communication avoid the deprecated address. But existing
>>     communication can continue to use the deprecated address until
>>     the valid lifetime expires.
> 
> 
> Other than that first line, those are the semantics we want surely? 
> Transport whose state is inexorably entwined with that prefix is hosed 
> (eg TCP). Shim6 shouldn't be ;). (Nor transports using shim6, eg TCP on 
> top of shim).
> 
> Note that for internal communication, TCP continues to work, as long as 
> the prefix stays valid.

The idea with *graceful* renumbering is that the admin knows in advance 
(hours, days, weeks) that the network will be renumbered and that there 
is some overlap when both prefixes work.
The notions of preferred/deprecated/invalid prefixes in RFC 2462 was 
designed to handle this case.

In this model communication (including external) continues to work for 
the deprecated address up until that address becomes invalid.

>>       But for existing
>>     communication when there isn't an easy way to switch to another 
>> address, it isn't clear what to do. (In some cases it might make
>>     sense to reset the ULP connection and recreating it, which will
>>     make it get a non-failed source address, but in other cases it
>>     would be better to wait for the failed address to start working
>>     again.)
> 
> 
> Ah, you're working on a "shim6 has intimate knowledge of ULP" model in 
> mind. Yes, things will get tricky that way - which seems a good reason 
> to /not/ do that.

No, I wasn't making any such assumptions. I was commenting on the 
general aspects of what be desirable and not; I was not commenting on 
what would be implemented in any particular software layer.
The takeaway point is that forcing a deprecated address, by overloading 
with a notion of "failed", makes it impossible to tell graceful 
renumbering apart from something which a multihoming solution can repair.

    Erik

Delivery-date: Wed, 31 Aug 2005 00:04:10 +0000
Date: Wed, 31 Aug 2005 01:03:22 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Erik Nordmark <[email-masked]>
cc: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Tue, 30 Aug 2005, Erik Nordmark wrote:

> Paul Jakma wrote:
>
>> But SAS is inherently tied in with routing. Let it do its magic.
>
> According to which RFC? ;-)

Don't think there are any standards-track for IPv4, not even sure if 
there are informational ones. There is something for IPv6 isn't 
there? However, in implementation, that tends to be how things are 
done in order to be able to get packets to the right place.

> The source address selection is influenced by which interface is 
> used (I think RFC 3484 says that), but I know of no text in an RFC 
> that ties the source address selection to the first hop router 
> being used.

Indeed, but it makes sense, and that's generally how source is 
selected for unspecified packets. Pick the nexthop, then pick an 
appropriate source according to connected network being output to 
for that next-hop.

> Yes, draft-huitema already points this out.

Excellent ;)

> But here is where the out-of-the-box behavior of the hosts doesn't 
> match your expectations. The host will pick a source address that 
> is appropriate for the network interface,

Right.

> but apart from that there isn't any correlation between the source 
> address and the 1st hop router it is using.

In practice, there likely is a very strong correlation between the 
output interface and the address(es) on that interface, the latter of 
which were used to decide which interface is appropriate for the 
chosen next-hop. So in practice, there's a very strong correlation 
between next-hop used and source address selected for addr-any 
sockets sending packets that will go via that next-hop.

You *can* setup really strange scenarios where you use a next-hop for 
a route (eg your default) for which none of your interfaces have 
addresses which they share with that nexthop. But then you must 
specify the output interface manually anyway, and hence you can 
easily specify the appropriate source too at that stage.

> But we also need to capture the slightly larger case where there 
> might be a handful of routers in the site, thus the host might not 
> be directly attached to the border routers.

Right, but there still must be a router attached to same network as 
one of its interfaces.

> draft-huitema suggests different approaches for doing this, and the 
> simplest one for multiple border routers is to have the border 
> routers setup tunnels between themselves so that they can pass 
> packets to an exit that is appropriate for the source address. In 
> the case of the border routers being on the same link, they can 
> discover each other automatically, but manual configuration might 
> be needed when such discovery isn't possible.

Tunnels between routers in the /same/ IGP AS does not seem ideal. 
Just distribute the required source-routing information to those 
routers which need to be aware of it, eg using Opaque-LSAs. Have each 
border router originate a "I am a packet-sink for address-range X" 
LSA. Would require OSPF though - which isn't ideal.

A simple standalone flooding protocol, to distribute knowledge of 
available prefixes to interior and work alongside RAs, might be an 
idea.

> This could be combined with some automatic protocol by which the 
> border routers can inform the interior routers of which prefixes 
> have failed, and the interior routers can advertise that in their 
> RAs. (Something like an extension to RFC 2894 "router renumbering 
> protocol" could be made to work.) However, such a protocol requires 
> some security configuration, otherwise anybody on the Internet 
> could use this to reconfigure the RAs your interior routers send.

One would hope this would this be a link-local protocol at least.. 
preferably with an HMAC or IPSec AH.

> Thus I don't think we can assume that some such, not yet invented 
> mechanism for propagating information from border routers to 
> interior routers, exist.

> This leads me to conclude that we can't assume that source address 
> selection can solve the ingress filtering interaction.

I disagree with this. Just because the required router protocol 
infrastructure does not exist is not a good reason to try bypass 
developing such infrastructure.

If there's customer demand for IPv6 multihoming (eg cause of shim6 ;) 
) and then customers require facilities for steering packets to right 
ISP according to the source, and require a way to reflect 
prefix-availability AS-wide, the router vendors will respond.

> In both the single and multiple border router cases, the host will 
> need to pick a source address before the packet leaves the host. 
> The source-based routing in the border router(s) ensure that when 
> both paths are working, the packet will leave on an exit where it 
> doesn't get filtered out.

Right.

And surely it's more sane to have border-routers be able to affect 
which prefixes are posted as 'available' by interior routers within 
an AS, to the hosts *before* they ever try pick a source?

That prefix-propogation protocol could, at the same time, be used to 
install the required source-routes in the routers concerned to direct 
the packets to the border-router appropriate for the prefix 
concerned.

That seems achievable and also more proactive and sane than having 
hosts sending a multitude of packets with every combination of source 
and destination they have to hand, simply for the reason that "well, 
the routers don't have that mechanism yet".

I don't understand why a protocol for /tomorrows/ networks should be 
trying to hack around a lack of functionality in /todays/ interior 
routing, when the correct place is obviously to extend capabilities 
of interior routing (and IPv6 RAs).

> However, when there is a failure then presumably the border routers 
> will send packets up the exits they know are working; in any case, 
> after a failure towards ISP1 packets with the source address from 
> ISP1's prefix will not make it through (either they are sent out 
> the exit to ISP1 and experience the failure, or they are sent out 
> the exit to ISP2 and are ingress filtered away).

> As a result, the host must be prepared to try sending packets with 
> different source addresses.

How do you reach that conclusion? The conclusion I would draw:

You need your routers to do what they're supposed to do: Direct 
packets to the appropriate next-hop. If that will require 
source-prefix routing and means to distribute source-specific routes, 
then let the routing area figure that out.

It could be done as a vendor specific hack with opaque LSAs pretty 
easily.

> Did it work without any manual configuration of the host?

*hosts*, and yes it did.

    ISP1-gw  ISP2-gw
         |    |
         |    |
         R    R
         |    |
      -----------
       | | | | |
        <hosts>

The hosts were configured with an address for each PA range, the 
routers did the magic.

> I think that is the crux of the matter; we can automate this for the single 
> SOHO router case by adding a "failed" notion to the prefixes in the RAs.

Positive signalling of failure is problematic though.

> But slightly larger networks would require a secure way for border 
> routers to inform interior routers, which I think requires 
> configuring some security somewhere.

Sure.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Integrity has no need for rules.

Delivery-date: Tue, 30 Aug 2005 23:19:35 +0000
Date: Wed, 31 Aug 2005 00:18:20 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Erik Nordmark <[email-masked]>
cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Tue, 30 Aug 2005, Erik Nordmark wrote:

> The problem is, as preferred/valid is defined and used in RFC 2462, a host 
> would interpret the prefix becoming deprecated as

> 	There is some graceful renumbering going on.

>       Make sure that new
> 	communication avoid the deprecated address. But existing
> 	communication can continue to use the deprecated address until
> 	the valid lifetime expires.

Other than that first line, those are the semantics we want surely? 
Transport whose state is inexorably entwined with that prefix is 
hosed (eg TCP). Shim6 shouldn't be ;). (Nor transports using shim6, 
eg TCP on top of shim).

Note that for internal communication, TCP continues to work, as long 
as the prefix stays valid.

> The behavior we'd want in multihoming is instead

> 	The address prefix has failed (at least for external
> 	communication). If there is a mechanism (like shim6) which can
> 	be used to quickly failover to some other address, then it makes
> 	sense to invoke it. Also, new communication which picks a source
> 	address should avoid using the failed prefix.

Well, as only preferred prefixes may be considered for 'new' 
connections, that happens automatically, right - at least for 
datagramme-like transports which dont specify their source..

>       But for existing
> 	communication when there isn't an easy way to switch to another 
> address, it isn't clear what to do. (In some cases it might make
> 	sense to reset the ULP connection and recreating it, which will
> 	make it get a non-failed source address, but in other cases it
> 	would be better to wait for the failed address to start working
> 	again.)

Ah, you're working on a "shim6 has intimate knowledge of ULP" model 
in mind. Yes, things will get tricky that way - which seems a good 
reason to /not/ do that.

> If we overload the "failed" semantics on the "deprecated" notion, 
> then the effect is that during graceful renumbering there will be a 
> storm of shim6 traffic when everybody tries to immediately switch 
> over, even though they might have a week during which the 
> deprecated address will continue to work.

Ah, yes. Good point.

I'll have to mull that one over. :)

>   Erik

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Television is now so desperately hungry for material that it is scraping
the top of the barrel.
 		-- Gore Vidal

Delivery-date: Tue, 30 Aug 2005 22:37:04 +0000
Date: Tue, 30 Aug 2005 15:36:49 -0700
From: Erik Nordmark <[email-masked]>
To: Paul Jakma <[email-masked]>
CC: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

Paul Jakma wrote:

> But SAS is inherently tied in with routing. Let it do its magic.

According to which RFC? ;-)

The source address selection is influenced by which interface is used (I 
think RFC 3484 says that), but I know of no text in an RFC that ties the 
source address selection to the first hop router being used.

> Ie, you mean the case of:
> 
>                      ____ISP1
>                     /
> host---<SOHO router>
>                     \____ISP2
> 
> 
> Where the SOHO router is just a normal SOHO "router" (ie a wee DSL 
> router) doing ye normal forward-by-destination, the host is shim6'ing? 
> You want to cover the case where the router is 'dumb' and still have 
> things work, regardless of which ISP it forwards to?
> 
> Things that come to mind:
> 
> 1. A dual connection SOHO DSL router likely would have to support
>   source-prefix routing anyway, to allow non-shim6 IP to work

Yes, draft-huitema already points this out.

> The other possibility for SOHO:
> 
> 
>      host
>       |
>  -----------
>    |     |
>  SOHO1  SOHO2
>    |     |
>  ISP1   ISP2
> 
> In which case the /host/ must pick which router to send to. In which 
> case everything is under its control - shim6 can easily leave source 
> unspecified. IP output then does SAS using routing information as 
> normal, picks a prefix and uses a source route to send it to the right 
> SOHO router.

But here is where the out-of-the-box behavior of the hosts doesn't match 
your expectations. The host will pick a source address that is 
appropriate for the network interface, but apart from that there isn't 
any correlation between the source address and the 1st hop router it is 
using.

But we also need to capture the slightly larger case where there might 
be a handful of routers in the site, thus the host might not be directly 
attached to the border routers.

draft-huitema suggests different approaches for doing this, and the 
simplest one for multiple border routers is to have the border routers 
setup tunnels between themselves so that they can pass packets to an 
exit that is appropriate for the source address. In the case of the 
border routers being on the same link, they can discover each other 
automatically, but manual configuration might be needed when such 
discovery isn't possible.

This could be combined with some automatic protocol by which the border 
routers can inform the interior routers of which prefixes have failed, 
and the interior routers can advertise that in their RAs. (Something 
like an extension to RFC 2894 "router renumbering protocol" could be 
made to work.)
However, such a protocol requires some security configuration, otherwise 
anybody on the Internet could use this to reconfigure the RAs your 
interior routers send.
Thus I don't think we can assume that some such, not yet invented 
mechanism for propagating information from border routers to interior 
routers, exist. This leads me to conclude that we can't assume that 
source address selection can solve the ingress filtering interaction.

In both the single and multiple border router cases, the host will need 
to pick a source address before the packet leaves the host. The 
source-based routing in the border router(s) ensure that when both paths 
are working, the packet will leave on an exit where it doesn't get 
filtered out.
However, when there is a failure then presumably the border routers will 
send packets up the exits they know are working; in any case, after a 
failure towards ISP1 packets with the source address from ISP1's prefix 
will not make it through (either they are sent out the exit to ISP1 and 
experience the failure, or they are sent out the exit to ISP2 and are 
ingress filtered away).

As a result, the host must be prepared to try sending packets with 
different source addresses.

> Well, I've run an IPv4 site with PA prefixes and ISPs which did ingress 
> filtering, unspecified addresses + source routing works nicely..

Did it work without any manual configuration of the host?

I think that is the crux of the matter; we can automate this for the 
single SOHO router case by adding a "failed" notion to the prefixes in 
the RAs. But slightly larger networks would require a secure way for 
border routers to inform interior routers, which I think requires 
configuring some security somewhere.

   Erik

Delivery-date: Tue, 30 Aug 2005 22:02:47 +0000
Date: Tue, 30 Aug 2005 15:01:50 -0700
From: Erik Nordmark <[email-masked]>
To: Paul Jakma <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

Paul Jakma wrote:

> Just not sending RAs (or not including the affected prefix in RAs) would 
> achieve same effect though wouldn't it?
> 
> I'm just curious what would be wrong with a setup like:
> 
> - valid lifetime: very high (say weeks)
> - preferred: low, eg 10s or 15s
> - RA interval: very low, 5s or so.
> 
> If link goes down, just stop sending RAs/including prefix in RAs. In 15s 
> hosts start preferring other prefixes. For datagramme protocols with no 
> kernel flow state (eg, potentially, shim6), the next packet after 
> preferred times out would use a new source address.

The problem is, as preferred/valid is defined and used in RFC 2462, a 
host would interpret the prefix becoming deprecated as
	There is some graceful renumbering going on. Make sure that new
	communication avoid the deprecated address. But existing
	communication can continue to use the deprecated address until
	the valid lifetime expires.

The behavior we'd want in multihoming is instead
	The address prefix has failed (at least for external
	communication). If there is a mechanism (like shim6) which can
	be used to quickly failover to some other address, then it makes
	sense to invoke it. Also, new communication which picks a source
	address should avoid using the failed prefix. But for existing
	communication when there isn't an easy way to switch to another 	
	address, it isn't clear what to do. (In some cases it might make
	sense to reset the ULP connection and recreating it, which will
	make it get a non-failed source address, but in other cases it
	would be better to wait for the failed address to start working
	again.)

If we overload the "failed" semantics on the "deprecated" notion, then 
the effect is that during graceful renumbering there will be a storm of 
shim6 traffic when everybody tries to immediately switch over, even 
though they might have a week during which the deprecated address will 
continue to work.

    Erik

Delivery-date: Tue, 30 Aug 2005 14:10:52 +0000
Date: Tue, 30 Aug 2005 15:10:14 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: Erik Nordmark <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Tue, 30 Aug 2005, marcelo bagnulo braun wrote:

> In this case, i guess you are considering that the host will choose 
> the source address containing prefix Pref1 when sending packets 
> with next hop router Soho1, right? would this be the case of 
> available implementations?

It is definitely the case for Linux and Solaris for IPv4. The source 
(if unspecified) is chosen according to which subnet the gateway is 
on. I'm hopeful its the same for IPv6.

> How would you deal with the following situation:
>
>     host
>      |
>  ----------
>      |
>    router3
>      |
> -----------
>   |     |
> SOHO1  SOHO2
>   |     |
> ISP1   ISP2
>
> In this case, router3 is internal to the SOHO site and it is announcing both 
> prefixes.

Router3 would have to forward according to source-address.

The question here though is: Who is doing the shimming? If the host, 
then the host needs to be informed somehow of local failures. Eg, 
router3's RAs to the host subnet would need to somehow be influenced 
by SOHO1 and SOHO2. (Some discussions on possibilities for this in 
other mails).

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
FORTUNE PROVIDES QUESTIONS FOR THE GREAT ANSWERS: #15
A:	The Royal Canadian Mounted Police.
Q:	What was the greatest achievement in taxidermy?

Delivery-date: Tue, 30 Aug 2005 13:44:01 +0000
Date: Tue, 30 Aug 2005 16:43:46 +0300
From: Jari Arkko <[email-masked]>
To: Pekka Nikander <[email-masked]>
CC: shim6 <[email-masked]>, marcelo bagnulo braun <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing

Pekka Nikander wrote:

>> On the other hand I'm not sure if you classification above
>> is exact. Primary locators are used in SCTP, but do
>> we need them in shim? Or did you mean "one current locator"?
>
>
> I don't understand the difference.

Looked at RFC 2960 again... "primary" appears to be current
selection in SCTP. Sorry, I mixed it with the application level
defaults that are provided in SCTP. Yes, a single current path
is a good idea.

> I am not so sure about the usefulness of a generic bypass mechanism.   
> Diagnostic applications will require such bypass, for sure, and it  
> will probably be a useful mechanism during a transition period, e.g.,  
> for SCTP.
>
> However, from an architectural point of view I think that the shim  
> layer is exactly the right layer to share information about the  
> available IP addresses.  The set of addresses is a property of the  
> host, and therefore seems to belong to the upper part of the IP  
> layer.  The more specific properties of a path (an address pair),  
> like QoS and what protocols/ports are available, seem to belong to a  
> upper layer, though.  Hence, from this point of view IMHO the shim  
> should take care of distributing information about the addresses that  
> belong to the same host while, in general, path discovery functions  
> belong to the upper layers.
>
> This was reflected also in the other mail with the "stepped" protocol  
> model, where there is an additional layer between the "basic shim"  
> and the unmodified ULs.

"transition" (not sure how temporary its going to be...) and SCTP
were what I had in mind. So I agree. But in the end that means
that we need the bypass...

--Jari

Delivery-date: Tue, 30 Aug 2005 12:28:04 +0000
Cc: shim6 <[email-masked]>, Erik Nordmark <[email-masked]>, Iljitsch van Beijnum <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Tue, 30 Aug 2005 14:27:22 +0200
To: Paul Jakma <[email-masked]>

El 29/08/2005, a las 23:06, Paul Jakma escribi=F3:
>
> A dedicated protocol for carrying prefix-availability information=20
> within an AS[1], not just per-link, would be better, likely. Did you=20=

> say people were looking at that?
>

what about Router Renumbering protocol?

regards, marcelo

> 1. AS in the IGP sense.
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Drop that pickle!
>

Delivery-date: Tue, 30 Aug 2005 12:28:03 +0000
Cc: Erik Nordmark <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Tue, 30 Aug 2005 14:27:05 +0200
To: Paul Jakma <[email-masked]>

El 29/08/2005, a las 23:54, Paul Jakma escribi=F3:

>
> The other possibility for SOHO:
>
>
>      host
>       |
>  -----------
>    |     |
>  SOHO1  SOHO2
>    |     |
>  ISP1   ISP2
>
> In which case the /host/ must pick which router to send to. In which=20=

> case everything is under its control - shim6 can easily leave source=20=

> unspecified. IP output then does SAS using routing information as=20
> normal, picks a prefix and uses a source route to send it to the right=20=

> SOHO router.
>

In this case, i guess you are considering that the host will choose the=20=

source address containing prefix Pref1 when sending packets with next=20
hop router Soho1, right? would this be the case of available=20
implementations?

> So, I think for the former case (one SOHO router), the router almost=20=

> certainly will have to support source-prefix routing to be a viable=20
> consumer product. The shim6 layer can use unspecified address,=20
> regardless which prefix the underlying IP layer chooses (hopefully=20
> using RA information, which hopefully the SOHO router is good enough=20=

> to update according to state of links to ISPs), the SOHO router will=20=

> send it to right ISP.
>
> In the latter case, its under the hosts control, and again the=20
> underlying IP SAS will do the right thing if shim6 leaves the address=20=

> unspecified.
>

How would you deal with the following situation:

      host
       |
   ----------
       |
     router3
       |
  -----------
    |     |
  SOHO1  SOHO2
    |     |
  ISP1   ISP2

In this case, router3 is internal to the SOHO site and it is announcing=20=

both prefixes.

Delivery-date: Tue, 30 Aug 2005 09:55:55 +0000
Cc: shim6 <[email-masked]>, marcelo bagnulo braun <[email-masked]>
From: Pekka Nikander <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing
Date: Tue, 30 Aug 2005 11:55:44 +0200
To: Jari Arkko <[email-masked]>

> On the other hand I'm not sure if you classification above
> is exact. Primary locators are used in SCTP, but do
> we need them in shim? Or did you mean "one current locator"?

I don't understand the difference.

> Also, some extensions in the interface appear unavoidable,
> like telling transport that it needs a slow start, or the ability
> of multihoming aware transports and applications to bypass
> the shim if they want to.

Agreed wrt sending events to the transports.

I am not so sure about the usefulness of a generic bypass mechanism.   
Diagnostic applications will require such bypass, for sure, and it  
will probably be a useful mechanism during a transition period, e.g.,  
for SCTP.

However, from an architectural point of view I think that the shim  
layer is exactly the right layer to share information about the  
available IP addresses.  The set of addresses is a property of the  
host, and therefore seems to belong to the upper part of the IP  
layer.  The more specific properties of a path (an address pair),  
like QoS and what protocols/ports are available, seem to belong to a  
upper layer, though.  Hence, from this point of view IMHO the shim  
should take care of distributing information about the addresses that  
belong to the same host while, in general, path discovery functions  
belong to the upper layers.

This was reflected also in the other mail with the "stepped" protocol  
model, where there is an additional layer between the "basic shim"  
and the unmodified ULs.

--Pekka

Delivery-date: Tue, 30 Aug 2005 09:21:51 +0000
Date: Tue, 30 Aug 2005 12:21:25 +0300
From: Jari Arkko <[email-masked]>
To: Pekka Nikander <[email-masked]>
CC: shim6 <[email-masked]>, marcelo bagnulo braun <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing

This seems like a very useful thread. Some comments:

> So, it looks to me that we are facing something like the following steps:
>
> 1. Minimally intelligent shim, one primary locator, unchanged
> transports
>
> 2. Shim as in #1, richer interface between shim and ULPs,
> shim-aware transports being able to use multiple locators
> at the same time
>
> 3. Further research on the proper new location(s) of traditional
> transport functions in the stack
>
> #2 and #3 are clearly research at this point of time, and outside the 
> scope of the WG charter. As I wrote in my first message in this 
> thread, I would very much see running code or simulations in that 
> space; without that we'd be driving in the dark.

I am very much in favor of the minimal shim. I realize finer grain
control is possible, but at least for now I remain unconvinced
of the real need -- particularly when we will still have to allow
for transports and applications that can do some of these things
themselves. A number of examples about various interfaces
and applications were listed earlier on this thread, but I
fear that its hard to find out the information on which you
can base the decision on, and automating everything would
require a TCP-like measurement process, information about
applications' needs, and perhaps even pricing information.

On the other hand I'm not sure if you classification above
is exact. Primary locators are used in SCTP, but do
we need them in shim? Or did you mean "one current locator"?
Also, some extensions in the interface appear unavoidable,
like telling transport that it needs a slow start, or the ability
of multihoming aware transports and applications to bypass
the shim if they want to.

--Jari

Delivery-date: Mon, 29 Aug 2005 21:54:50 +0000
Date: Mon, 29 Aug 2005 22:54:01 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Erik Nordmark <[email-masked]>
cc: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Mon, 29 Aug 2005, Erik Nordmark wrote:

> When talking about actual uni-directional failures due to links or 
> routers going bad, I'd agree.

> But one assumption we've been making in the multi6/shim6 work is 
> that we want to cover the SOHO multihoming case.

Indeed.

> Using basically retain ISP service from more than one ISP, where 
> the ISPs might apply ingress filtering, and the site can't get the 
> ISP to relax their ingress filtering.

Right.

I think ISP ingress filtering should be assumed, it's unlikely to 
/not/ be in place these days.

> If we don't do something in this case, then we'll have the routing 
> (and selecting of exit ISP) be completely independent of the 
> (hosts') source address selection,

But SAS is inherently tied in with routing. Let it do its magic.

> and we will have a very high 
> incidence of packets being dropped by the ingress filters.

Ie, you mean the case of:

                      ____ISP1
                     /
host---<SOHO router>
                     \____ISP2

Where the SOHO router is just a normal SOHO "router" (ie a wee DSL 
router) doing ye normal forward-by-destination, the host is 
shim6'ing? You want to cover the case where the router is 'dumb' and 
still have things work, regardless of which ISP it forwards to?

Things that come to mind:

1. A dual connection SOHO DSL router likely would have to support
   source-prefix routing anyway, to allow non-shim6 IP to work

   - in which case, the shim6 host could use either prefix, and
     relying on some other prefix-availability information would be
     much better (eg, SOHO DSL often uses PPP, which includes its own
     keepalive information. Which can be propogated via RAs, as
     per our other emails)

2. If the SOHO router doesn't support source-routing to deal with
   ingress filtering, we don't know what, if any, balancing it does.

   - best case: forwarding is pinned to one ISP while links dont
     change
     - one source always works (while links dont change)
     - other does not

   - worst case: it tries to load balance amongst links:
     - both sources the shim6 host could use will experience regular
       drops, lots of 'retraining' of the 'shim6' protocol and pretty
       pathological performance one suspects
     - such a router would suck for multihoming, regardless of shim6,
       hence why I suspect 1 would be more likely

The other possibility for SOHO:

      host
       |
  -----------
    |     |
  SOHO1  SOHO2
    |     |
  ISP1   ISP2

In which case the /host/ must pick which router to send to. In which 
case everything is under its control - shim6 can easily leave source 
unspecified. IP output then does SAS using routing information as 
normal, picks a prefix and uses a source route to send it to the 
right SOHO router.

So, I think for the former case (one SOHO router), the router almost 
certainly will have to support source-prefix routing to be a viable 
consumer product. The shim6 layer can use unspecified address, 
regardless which prefix the underlying IP layer chooses (hopefully 
using RA information, which hopefully the SOHO router is good enough 
to update according to state of links to ISPs), the SOHO router will 
send it to right ISP.

In the latter case, its under the hosts control, and again the 
underlying IP SAS will do the right thing if shim6 leaves the address 
unspecified.

> Some options for what to do is in 
> draft-huitema-multi6-ingress-filtering-00 (now expired). Even if 
> such a scheme is standardized and deployed, it might not be 
> deployed everywhere. Which is why some of see utility in having the 
> capability to try all <source,dest> combinations in both directions 
> of the communication. But I sure do hope that in normal deployment 
> we don't have link/router failures that cause shim6 to have to try 
> close to n^2 locator pairs before finding a working one.

Well, I've run an IPv4 site with PA prefixes and ISPs which did 
ingress filtering, unspecified addresses + source routing works 
nicely..

In the face of ingress filtering, the /only/ correct outward path is 
easily determined by the source. Given the source can easily be 
influenced by adding/removing addresses (or depreferring in IPv6), 
*IF* the upper layer leaves the address unspecified, seems like that 
is best way to tackle problem to me.. See:

 	http://hibernia.jakma.org/~paul/rc.iprules

That was in production for several years for an *IPv4* multi-PA 
address SMTP and HTTP service site. It worked for inbound, and it 
worked for outbound (SMTP) because those services did not try to get 
clever about source-address selection.

It didn't try to do failover, cause our links were reliable enough. 
If they had not been, that could have done by adding/removing the 
relevant PA addresses based on whatever link/path availability 
information you had to hand (even a 'ping' script - there are /lots/ 
of possibilities).

Eg, Solaris has the same kind of thing, but more sophisticated: IPMP.

No n^2 needed.

> I think one can come up with strategies for the order in which 
> locator pairs are tried that will work efficiently for the normal 
> case of some source address dependent exit router selection (in the 
> cases this is necessary to avoid ingress filtering dropping the 
> packets), and link/router failures. For instance, if A has locators 
> A1, A2 and B has B1, B2, B3, and A is communicating with B using 
> <A1, B1> then when A suspects that the locator pair has stopped 
> working it can try to maximize the probability of using a different 
> path by trying to combinations which change both the source and 
> destination first, and then the pairs that only change one of them. 
> For example, try in the order of

> 	<A2, B2>
> 	<A2, B3>
> followed by
> 	<A1, B2>
> 	<A1, B3>
> 	<A2, B1>

I contend you just need, worst case, B1...Bn, probe packets.

Just let the existing mechanisms for source selection decide the 
output path.

I will try write up my view of things sometime, ;) just busy with 
other stuff at the moment.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
"`The best way to get a drink out of a Vogon is to stick
your finger down his throat...'"

- The Book, on one of the Vogon's social inadequacies.

Delivery-date: Mon, 29 Aug 2005 21:18:51 +0000
Date: Mon, 29 Aug 2005 14:18:33 -0700
From: Erik Nordmark <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: Pekka Nikander <[email-masked]>, shim6 <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing

Iljitsch van Beijnum wrote:
> On 24-aug-2005, at 1:00, Erik Nordmark wrote:
> 
>>> Another approach would be for the application on our side to ask  
>>> the  transport and for the transport to ask the shim for a  certain  
>>> behavior, such as "make incoming traffic for this FTP app  come in 
>>> at  address X if possible", and the shim communicates this  request 
>>> to the  shim layer at the correspondent, which then takes  the 
>>> appropriate  action.
> 
> 
>> It isn't clear for me what the motivation is for such an approach.
> 
> 
> Simple: if I'm receiving a large file I would really like it to come  in 
> over my cheap high bandwidth link rather than by expesive, low  
> bandwidth bounded delay link.

That's a motivation for traffic engineering type activities where some
network or system admin controls things. But it was your use of
"application" above that made me ask the question. Is there a case
where it makes sense to have the *application* do this?

If we go that path it would seem to entail a lot of work - adding
options to control this to scp, wget, firefox, ...

I think we need to understand the larger "TE-like" requirements that 
shim6 can reasonably satisfy, and see if the administrators control can 
be used to address this example, and not assume that the application 
will be involved.

>> It seems quite odd for the application on A to say something about  
>> which path B is using to send packets. If this was really a need,  
>> wouldn't we already see requests for the ability to the application  
>> on A to tell B whether to send over the "eth0" vs. "eth1" network  
>> interface?
> 
> 
> I'm not sure I understand what you mean... Do you mean A's eth0 or  B's 
> eth0?

A telling B to send its packets out over B's eth0.

>> And *if* it turns out that we need the ability for the app on A to  
>> affect how B sends it packets, then this can be viewed as a locator  
>> pair preference mechanism which is layered on top of the shim  proper. 
>> There isn't a need to have different contexts to do this,  and it 
>> doesn't help because the granularity would be finer than the  ULID pair.
> 
> 
> This also means we don't have to solve it now, as long as we make the  
> protocol extensible enough to allow adding features like this in the  
> future.

To some degree, but we either need to make the shim6 protocol capable of 
carrying a way to specify the preferences when sending packets in the 
return direction, or decide that it is the application protocol which 
will be used to communicate these preferences.

    Erik

Delivery-date: Mon, 29 Aug 2005 21:11:16 +0000
Date: Mon, 29 Aug 2005 14:11:07 -0700
From: Erik Nordmark <[email-masked]>
To: Paul Jakma <[email-masked]>
CC: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

[I'm still catching up on some mail]

Paul Jakma wrote:

> My gut feelings though are:
> 
> - Failures typically are near the edges
> - Failures are typically bi-directional for a given path
> - Uni-directional failures tend to be due to /congestion/, not
>   actual failures - again, typically at the edges. Congestion related
>   "failures" tend to be very transient/sporadic.

When talking about actual uni-directional failures due to links or 
routers going bad, I'd agree.

But one assumption we've been making in the multi6/shim6 work is that we 
want to cover the SOHO multihoming case. Using basically retain ISP 
service from more than one ISP, where the ISPs might apply ingress 
filtering, and the site can't get the ISP to relax their ingress filtering.

If we don't do something in this case, then we'll have the routing (and 
selecting of exit ISP) be completely independent of the (hosts') source 
address selection, and we will have a very high incidence of packets 
being dropped by the ingress filters.

Some options for what to do is in 
draft-huitema-multi6-ingress-filtering-00 (now expired).
Even if such a scheme is standardized and deployed, it might not be 
deployed everywhere.
Which is why some of see utility in having the capability to try all 
<source,dest> combinations in both directions of the communication. But 
I sure do hope that in normal deployment we don't have link/router 
failures that cause shim6 to have to try close to n^2 locator pairs 
before finding a working one.

I think one can come up with strategies for the order in which locator 
pairs are tried that will work efficiently for the normal case of some 
source address dependent exit router selection (in the cases this is 
necessary to avoid ingress filtering dropping the packets), and 
link/router failures. For instance, if A has locators A1, A2 and B has 
B1, B2, B3, and A is communicating with B using <A1, B1> then when A 
suspects that the locator pair has stopped working it can try to 
maximize the probability of using a different path by trying to 
combinations which change both the source and destination first, and 
then the pairs that only change one of them. For example, try in the 
order of
	<A2, B2>
	<A2, B3>
followed by
	<A1, B2>
	<A1, B3>
	<A2, B1>

    Erik

Delivery-date: Mon, 29 Aug 2005 21:07:15 +0000
Date: Mon, 29 Aug 2005 22:06:01 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Erik Nordmark <[email-masked]>
cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Mon, 29 Aug 2005, Erik Nordmark wrote:

> "failed" is about right. The semantics is basically a 
> recommendation to hosts to not use that prefix, and if possible 
> switch to using other prefixes.

Right.

Just not sending RAs (or not including the affected prefix in RAs) 
would achieve same effect though wouldn't it?

I'm just curious what would be wrong with a setup like:

- valid lifetime: very high (say weeks)
- preferred: low, eg 10s or 15s
- RA interval: very low, 5s or so.

If link goes down, just stop sending RAs/including prefix in RAs. In 
15s hosts start preferring other prefixes. For datagramme protocols 
with no kernel flow state (eg, potentially, shim6), the next packet 
after preferred times out would use a new source address.

You would prefer failure to be a positive thing? I think possibly 
that might be hard to ensure. Eg, what if it was not the router<->ISP 
link which failed, but the router<->local LAN link? A positive 
"failed" bit would be tricky..

I'm trying to puzzle out why "lost our link to ISP supplying external 
connectivity for that prefix" would /ever/ be at odds with "Preferred 
lifetime expired".

> But one of the tricky aspects is that the prefix might work 
> perfectly for traffic internal to the site, and it is only external 
> traffic that is effected. I don't know how easy it is to convey 
> that notion to the hosts, since the hosts don't know how large 
> prefix is assigned to the site (assuming it is and always will be a 
> /48 is probably a bad idea).

Hmm, it's probably not useful for hosts to know how far a network 
reaches. Only the routers can really have a notion of that, and 
signal which networks are available via RAs.

> But even if that is the case, we still need to handle the case when 
> the peer host doesn't support shim6.

Yep.

> You mean pinning it to a route for some external prefix? Presumably 
> we want things to work when a site has multiple exits and each ISP 
> just advertises a default route into the site.

In lieu of a dedicated multi-hop prefix-availability protocol for 
routers, there likely are implementation-local hacks which could be 
done. Eg:

- on the borders routers, make advertisement of a high-metric prefix
   covering ISPx assigned address range, conditional on some other
   route (eg, in your example, a default route with protocol X and
   nexthop corresponding to the appropriate ISP's router)

- on all routers, make sending/including RA for that prefix
   conditional on the high-metric prefix being present in IGP.

Bit hacky though.

A dedicated protocol for carrying prefix-availability information 
within an AS[1], not just per-link, would be better, likely. Did you 
say people were looking at that?

1. AS in the IGP sense.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Drop that pickle!

Delivery-date: Mon, 29 Aug 2005 20:47:05 +0000
Date: Mon, 29 Aug 2005 13:45:08 -0700
From: Erik Nordmark <[email-masked]>
To: Paul Jakma <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

Paul Jakma wrote:

> One thing though: from the POV of the router which wants to change RA's 
> due to link-failure, it has no idea of how temporary the failure is. So 
> maybe call it 'failed' instead? But that implies continuing to send RAs 
> for the prefix.

"failed" is about right. The semantics is basically a recommendation to 
hosts to not use that prefix, and if possible switch to using other 
prefixes.

But one of the tricky aspects is that the prefix might work perfectly 
for traffic internal to the site, and it is only external traffic that 
is effected. I don't know how easy it is to convey that notion to the 
hosts, since the hosts don't know how large prefix is assigned to the 
site (assuming it is and always will be a /48 is probably a bad idea).

> Maybe because no one is interested in multi-prefix IPv6 sites until 
> solutions like shim6 are available?

But even if that is the case, we still need to handle the case when the 
peer host doesn't support shim6.

>> So it's not clear to me whether such a mechanism would be limited to 
>> the single link site where the RAs come directly from the border router.
> 
> 
> Hmm, I guess yes. The multi-router case and protocol for distributing 
> external link-availibility seems a general one.
> 
> I guess an implementation could offer a facility to 'pin' RAs to a 
> specific prefix announced in IGP. IGP withdraws route, the routers RA 
> code either stops sending RAs for that prefix or uses the 'failed' bit?

You mean pinning it to a route for some external prefix?
Presumably we want things to work when a site has multiple exits and 
each ISP just advertises a default route into the site.

    Erik

Delivery-date: Mon, 29 Aug 2005 15:08:06 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: failure detection
Date: Mon, 29 Aug 2005 17:07:29 +0200
To: marcelo bagnulo braun <[email-masked]>

On 29-aug-2005, at 15:52, marcelo bagnulo braun wrote:

>> So I think that if we are going to add an indication that a prefix  
>> isn't working, so that shim6 hosts can change the locator they use  
>> more quickly,

> agree, but i was thinking also about non-shim hosts  (sorry for not  
> stating) this before) (i.e. unmodified hosts) and whether it would  
> be possible to enable routers to detect that the link with a given  
> ISP is down and inform hosts within the multihomed host not to use  
> it anymore (especially in order to avoid trying with prefixes that  
> we know are down for establishing new communications and reducing  
> latency at establishment)

I think it would be entirely appropriate to deprecate prefixes tied  
to the impaired ISP.

This is also something that can actually happen quite fast: routers  
can just send out an RA with the prefix' preferred lifetime set to 0  
and the hosts will deprecate their addresses immediately. (Doing the  
same for the valid lifetime won't work as there is a minimum of two  
hours for security reasons.)

Delivery-date: Mon, 29 Aug 2005 13:52:27 +0000
Cc: Iljitsch van Beijnum <[email-masked]>, Paul Jakma <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Mon, 29 Aug 2005 15:52:45 +0200
To: Erik Nordmark <[email-masked]>

El 29/08/2005, a las 8:24, Erik Nordmark escribi=F3:

> marcelo bagnulo braun wrote:
>
>> I agree that using the "preferred lifetime" for signaling that a=20
>> prefix is no longer working would imply to change the meaning of the=20=

>> field, but wouldn't it be possible to use the "valid lifetime" (set=20=

>> to zero) to inform the hosts that a prefix is no longer available?
>
> If you have non-shim6 hosts, then a valid lifetime set to zero might=20=

> mean that they'll immediately terminate any tcp connections that use=20=

> that prefix. I don't think we want to do that, because packets might=20=

> start flowing again before e.g., TCP gives up.
>
> So I think that if we are going to add an indication that a prefix=20
> isn't working, so that shim6 hosts can change the locator they use=20
> more quickly,

agree, but i was thinking also about non-shim hosts  (sorry for not=20
stating) this before) (i.e. unmodified hosts) and whether it would be=20
possible to enable routers to detect that the link with a given ISP is=20=

down and inform hosts within the multihomed host not to use it anymore=20=

(especially in order to avoid trying with prefixes that we know are=20
down for establishing new communications and reducing latency at=20
establishment)

(in this case, because we are dealing with unmodified hosts the option=20=

would be to use an existent tool or not to provide the functionality, i=20=

guess)

>  we should add a new indication and not overload an existing one to=20
> have these semantics.
>
>    Erik
>

Delivery-date: Mon, 29 Aug 2005 10:17:25 +0000
Date: Mon, 29 Aug 2005 11:16:14 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Erik Nordmark <[email-masked]>
cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

Hi Erik,

On Fri, 26 Aug 2005, Erik Nordmark wrote:

> I had one minor comment on the above.
>
> It might very well be a useful optimization for routers to be able 
> to express to hosts which prefixes are currently not working for 
> going outside the site.

Yeah, probably. This wouldn't even be shim6 specific.

> But I think it would be a mistake to overload the 
> preferred/deprecated notion for this purpose; if we want to do this 
> we can just as well add a "temporarily unavailable" bit to the 
> prefix information options, and that way avoid confusing any hosts 
> that are not shim6 aware.

You're the expert. :)

One thing though: from the POV of the router which wants to change 
RA's due to link-failure, it has no idea of how temporary the failure 
is. So maybe call it 'failed' instead? But that implies continuing to 
send RAs for the prefix.

I was envisaging that the router would simply stop sending RAs, and 
once the preferred time ran out, hosts would use other prefixes (if 
any were available).

> It remains to be seen how hard it would be to provide this for a 
> site with multiple routers. We have the router renumbering RFC, 
> which in theory could be used to carry this from a border router to 
> other routers in the site, but nobody is implementing this protocol 
> AFAIK.

Maybe because no one is interested in multi-prefix IPv6 sites until 
solutions like shim6 are available?

> So it's not clear to me whether such a mechanism would be limited 
> to the single link site where the RAs come directly from the border 
> router.

Hmm, I guess yes. The multi-router case and protocol for distributing 
external link-availibility seems a general one.

I guess an implementation could offer a facility to 'pin' RAs to a 
specific prefix announced in IGP. IGP withdraws route, the routers RA 
code either stops sending RAs for that prefix or uses the 'failed' 
bit?

>   Erik

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Why are you so hard to ignore?

Delivery-date: Mon, 29 Aug 2005 06:26:02 +0000
Date: Sun, 28 Aug 2005 23:24:58 -0700
From: Erik Nordmark <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, Paul Jakma <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

marcelo bagnulo braun wrote:

> I agree that using the "preferred lifetime" for signaling that a prefix 
> is no longer working would imply to change the meaning of the field, but 
> wouldn't it be possible to use the "valid lifetime" (set to zero) to 
> inform the hosts that a prefix is no longer available?

If you have non-shim6 hosts, then a valid lifetime set to zero might 
mean that they'll immediately terminate any tcp connections that use 
that prefix. I don't think we want to do that, because packets might 
start flowing again before e.g., TCP gives up.

So I think that if we are going to add an indication that a prefix isn't 
working, so that shim6 hosts can change the locator they use more 
quickly, we should add a new indication and not overload an existing one 
to have these semantics.

    Erik

Delivery-date: Sun, 28 Aug 2005 20:55:37 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: failure detection
Date: Sun, 28 Aug 2005 22:55:24 +0200
To: marcelo bagnulo braun <[email-masked]>

On 28-aug-2005, at 22:44, marcelo bagnulo braun wrote:

>> It might very well be a useful optimization for routers to be able  
>> to express to hosts which prefixes are currently not working for  
>> going outside the site. But I think it would be a mistake to  
>> overload the preferred/deprecated notion for this purpose;

> I agree that using the "preferred lifetime" for signaling that a  
> prefix is no longer working would imply to change the meaning of  
> the field, but wouldn't it be possible to use the "valid  
> lifetime" (set to zero) to inform the hosts that a prefix is no  
> longer available?

The behavior when the preferred and valid lifetimes reach 0 is  
already well-defined, I don't think we can (or should want to) change  
this.

We have to be careful to stay away from premature optimization:  
either and address works, or it doesn't. The reachability detection  
will figure it out, that's good enough for now...

Delivery-date: Sun, 28 Aug 2005 20:44:35 +0000
Cc: Iljitsch van Beijnum <[email-masked]>, Paul Jakma <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Sun, 28 Aug 2005 22:44:45 +0200
To: Erik Nordmark <[email-masked]>

El 27/08/2005, a las 1:15, Erik Nordmark escribi=F3:

> Paul Jakma wrote:
>
>> We already have protocols for this, there is at least one=20
>> possibility, IPv6 RAs, so:
>> - Use IPv6 RAs to advertise both prefixes, with a preferred lifetime
>>   set according to how quickly you want to switch, eg set it equal
>>   to twice the RA interval, or even equal to it.
>>   When the DSL router detects a link has gone down, it simply stops
>>   advertising the relevant prefix.
>> - We may need to invent protocols to carry valid-source prefix
>>   information across multiple routers in a site. (to update
>>   prefix-advertisement information). I think that's a general IPv6 RA
>>   problem though.
>> - Other stateful configuration protocols may be used, eg DHCPv6
>>   the server would need to become multi-address aware, an
>>   implementation detail.
>
> Paul,
>
> I had one minor comment on the above.
>
> It might very well be a useful optimization for routers to be able to=20=

> express to hosts which prefixes are currently not working for going=20
> outside the site. But I think it would be a mistake to overload the=20
> preferred/deprecated notion for this purpose; if we want to do this we=20=

> can just as well add a "temporarily unavailable" bit to the prefix=20
> information options, and that way avoid confusing any hosts that are=20=

> not shim6 aware.
>

I agree that using the "preferred lifetime" for signaling that a prefix=20=

is no longer working would imply to change the meaning of the field,=20
but wouldn't it be possible to use the "valid lifetime" (set to zero)=20
to inform the hosts that a prefix is no longer available?

Regards, marcelo

> It remains to be seen how hard it would be to provide this for a site=20=

> with multiple routers. We have the router renumbering RFC, which in=20
> theory could be used to carry this from a border router to other=20
> routers in the site, but nobody is implementing this protocol AFAIK.
> So it's not clear to me whether such a mechanism would be limited to=20=

> the single link site where the RAs come directly from the border=20
> router.
>
>    Erik
>

Delivery-date: Fri, 26 Aug 2005 23:16:24 +0000
Date: Fri, 26 Aug 2005 16:15:47 -0700
From: Erik Nordmark <[email-masked]>
To: Paul Jakma <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection

Paul Jakma wrote:

> We already have protocols for this, there is at least one possibility, 
> IPv6 RAs, so:
> 
> - Use IPv6 RAs to advertise both prefixes, with a preferred lifetime
>   set according to how quickly you want to switch, eg set it equal
>   to twice the RA interval, or even equal to it.
> 
>   When the DSL router detects a link has gone down, it simply stops
>   advertising the relevant prefix.
> 
> - We may need to invent protocols to carry valid-source prefix
>   information across multiple routers in a site. (to update
>   prefix-advertisement information). I think that's a general IPv6 RA
>   problem though.
> 
> - Other stateful configuration protocols may be used, eg DHCPv6
>   the server would need to become multi-address aware, an
>   implementation detail.

Paul,

I had one minor comment on the above.

It might very well be a useful optimization for routers to be able to 
express to hosts which prefixes are currently not working for going 
outside the site. But I think it would be a mistake to overload the 
preferred/deprecated notion for this purpose; if we want to do this we 
can just as well add a "temporarily unavailable" bit to the prefix 
information options, and that way avoid confusing any hosts that are not 
shim6 aware.

It remains to be seen how hard it would be to provide this for a site 
with multiple routers. We have the router renumbering RFC, which in 
theory could be used to carry this from a border router to other routers 
in the site, but nobody is implementing this protocol AFAIK.
So it's not clear to me whether such a mechanism would be limited to the 
single link site where the RAs come directly from the border router.

    Erik

Delivery-date: Thu, 25 Aug 2005 00:46:20 +0000
Date: Wed, 24 Aug 2005 17:41:07 -0700
From: Erik Nordmark <[email-masked]>
To: Pekka Nikander <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing

Pekka Nikander wrote:

> Something like the following?
> 
>                  +----------------+  +--------------+
>                  | SHIM unaware UL|  | SHIM aware UL|
>                  +----------------+  +--------------+
>                          ^                   ^
>                          | ULIDs only        |
>                          v                   |
>      Admin       +----------------+          |
>      interface-->| "Policy" subL  |          | ULIDs+locators
>                  +----------------+          |
>                          ^                   |
>                          | ULIDs+locators    |
>                          v                   v
>                  +----------------------------------+
>                  |               SHIM               |
>                  +----------------------------------+
>                                   ^
>                                   | locators only
> 

Well, it could also be that the same ULP can choose whether to annotate 
the ULIDs with locators, and not have different types of ULPs always 
doing the same thing.

But other than this detail it looks right.

    Erik

Envelope-to: [email-masked]
Delivery-date: Wed, 24 Aug 2005 12:07:25 +0000
From: Tim Shepard <[email-masked]>
To: Pekka Nikander <[email-masked]>
Cc: shim6 <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Wed, 24 Aug 2005 08:06:54 -0400
Message-Id: <[email-masked]>

Pekka Nikander said:

> > Could this be something passed down from the ULP (prefer a given  
> > locator pair) for ULPs that are aware of multiple locators? Or a  
> > "locator pair preference sub-layer" for other ULPs?
> 
> Something like the following?
> 
>                   +----------------+  +--------------+
>                   | SHIM unaware UL|  | SHIM aware UL|
>                   +----------------+  +--------------+
>                           ^                   ^
>                           | ULIDs only        |
>                           v                   |
>       Admin       +----------------+          |
>       interface-->| "Policy" subL  |          | ULIDs+locators
>                   +----------------+          |
>                           ^                   |
>                           | ULIDs+locators    |
>                           v                   v
>                   +----------------------------------+
>                   |               SHIM               |
>                   +----------------------------------+
>                                    ^
>                                    | locators only
> 
> 
> The admin interface could then used to implement policy negotiation
> on the top of the basic SHIM, at some later point, if needed.  SHIM
> aware ULPs (like SCTP++?) could implement similar functionality
> themselves.

Nice picture!  It reminds me of a thought that's been bouncing around
in my head for the last 36 hours.....  At the ACM Turing Award Lecture
on Monday night (streamed live on the net, but video on line somewhere
I think), Bob Kahn mentioned that while they were thinking of a
layered system while designing the Internet protocols, the picture
they had in mind was not the usual vertical stack of protocols, but
rather stair steps.  That way, the interface of each layer is exposed
allowing others to build on top using whatever interfaces they need.
Pluggability and extensibility were more important to them than
layering.

Your picture makes me think you may have seen the lecture.

BTW, some of the discussion between Cerf and Kahn was very relevant to
the issues facing the net today regarding the mobile, multihomed, and
less-than-always-connected natures of the machines connected to the
net today.  (Cerf said that when they were designing the Internet
protocols in the 1970s, the machines connected to the net were rather
large and tended to stay put.)

			-Tim Shepard
			 [email-masked]

Delivery-date: Wed, 24 Aug 2005 10:18:05 +0000
Cc: Iljitsch van Beijnum <[email-masked]>, marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
From: Pekka Nikander <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing
Date: Wed, 24 Aug 2005 12:17:29 +0200
To: Erik Nordmark <[email-masked]>

> And *if* it turns out that we need the ability for the app on A to  
> affect how B sends it packets, then this can be viewed as a locator  
> pair preference mechanism which is layered on top of the shim  
> proper. There isn't a need to have different contexts to do this,  
> and it doesn't help because the granularity would be finer than the  
> ULID pair.

Agreed.

> Could this be something passed down from the ULP (prefer a given  
> locator pair) for ULPs that are aware of multiple locators? Or a  
> "locator pair preference sub-layer" for other ULPs?

Something like the following?

                  +----------------+  +--------------+
                  | SHIM unaware UL|  | SHIM aware UL|
                  +----------------+  +--------------+
                          ^                   ^
                          | ULIDs only        |
                          v                   |
      Admin       +----------------+          |
      interface-->| "Policy" subL  |          | ULIDs+locators
                  +----------------+          |
                          ^                   |
                          | ULIDs+locators    |
                          v                   v
                  +----------------------------------+
                  |               SHIM               |
                  +----------------------------------+
                                   ^
                                   | locators only

The admin interface could then used to implement policy negotiation
on the top of the basic SHIM, at some later point, if needed.  SHIM
aware ULPs (like SCTP++?) could implement similar functionality
themselves.

--Pekka Nikander

Delivery-date: Wed, 24 Aug 2005 07:12:02 +0000
Cc: Pekka Nikander <[email-masked]>, shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing
Date: Wed, 24 Aug 2005 09:11:05 +0200
To: Erik Nordmark <[email-masked]>

On 24-aug-2005, at 1:00, Erik Nordmark wrote:

>> Another approach would be for the application on our side to ask  
>> the  transport and for the transport to ask the shim for a  
>> certain  behavior, such as "make incoming traffic for this FTP app  
>> come in at  address X if possible", and the shim communicates this  
>> request to the  shim layer at the correspondent, which then takes  
>> the appropriate  action.

> It isn't clear for me what the motivation is for such an approach.

Simple: if I'm receiving a large file I would really like it to come  
in over my cheap high bandwidth link rather than by expesive, low  
bandwidth bounded delay link.

> It seems quite odd for the application on A to say something about  
> which path B is using to send packets. If this was really a need,  
> wouldn't we already see requests for the ability to the application  
> on A to tell B whether to send over the "eth0" vs. "eth1" network  
> interface?

I'm not sure I understand what you mean... Do you mean A's eth0 or  
B's eth0?

> And *if* it turns out that we need the ability for the app on A to  
> affect how B sends it packets, then this can be viewed as a locator  
> pair preference mechanism which is layered on top of the shim  
> proper. There isn't a need to have different contexts to do this,  
> and it doesn't help because the granularity would be finer than the  
> ULID pair.

This also means we don't have to solve it now, as long as we make the  
protocol extensible enough to allow adding features like this in the  
future.

>> - rewriting, which happens based on ULID (or more than just ULID  
>> if  we for the ULID state??? what would be the secondary  
>> differentiator?)

> Could this be something passed down from the ULP (prefer a given  
> locator pair) for ULPs that are aware of multiple locators? Or a  
> "locator pair preference sub-layer" for other ULPs?

Of course. Then obviously some kind of info about which locators to  
use would trickle down from upper layers. (We'll have to look at how  
that should happen at some point, because we really don't want to do  
complex stuff for each individual packet.)

But I guess I was thinking about the situation where such ULP info on  
what should happen isn't available...

> One can envision simple examples of such a sublayer in the  
> multihomed host case - e.g. a table which indicates that the VoIP  
> port numbers should prefer the GPRS interface over the WLAN  
> interface i.e. preferences on the local locators.

Yes, but this immediately begs the question: how is this table of  
port numbers maintained? (BTW I wouldn't want to do VoIP over GRPS:  
you get the worst of two worlds.)

Delivery-date: Tue, 23 Aug 2005 23:00:11 +0000
Date: Tue, 23 Aug 2005 16:00:47 -0700
From: Erik Nordmark <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: marcelo bagnulo braun <[email-masked]>, Pekka Nikander <[email-masked]>, shim6 <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing

Iljitsch van Beijnum wrote:

> Well, for outgoing packets that shouldn't be a problem. If the FTP  
> application wants to use the cheap high bandwidth link while the VoIP  
> application wants to use the lower bandwidth, more expensive but  
> bounded-delay link, the shim can just insert the right addresses and  
> send the packets toward the right next hop and everything will work  fine.

Yep.

> Another approach would be for the application on our side to ask the  
> transport and for the transport to ask the shim for a certain  behavior, 
> such as "make incoming traffic for this FTP app come in at  address X if 
> possible", and the shim communicates this request to the  shim layer at 
> the correspondent, which then takes the appropriate  action.

It isn't clear for me what the motivation is for such an approach. It 
seems quite odd for the application on A to say something about which 
path B is using to send packets. If this was really a need, wouldn't we 
already see requests for the ability to the application on A to tell B 
whether to send over the "eth0" vs. "eth1" network interface?

So I'm questioning whether we need this complexity.

> Note that there is no real reason to do a full "fork" of the shim  
> state: the most important thing the shim has to do is allow for the  
> rewriting and demux of packets. The rewriting is impacted, but not  the 
> demux. In fork terminology: rather than fork off a new process,  we're 
> starting a new thread within the same process, which isn't  nearly as 
> heavy an operation. I think all of this can happen based on  transport 
> demux info (i.e., port numbers).

I think I agree. The sender can use any locator pair - and use different 
pairs for different traffic and the receiver just rewrites it to use the 
ULID. So we don't need different shim6 contexts.

And *if* it turns out that we need the ability for the app on A to 
affect how B sends it packets, then this can be viewed as a locator pair 
preference mechanism which is layered on top of the shim proper. There 
isn't a need to have different contexts to do this, and it doesn't help 
because the granularity would be finer than the ULID pair.

> It's interesting to note that by now we have several types of  
> communication happening at several levels of granularity:
> 
> - service differentiation information that is ULP session specific
> - rewriting, which happens based on ULID (or more than just ULID if  we 
> for the ULID state??? what would be the secondary differentiator?)

Could this be something passed down from the ULP (prefer a given locator 
pair) for ULPs that are aware of multiple locators? Or a "locator pair 
preference sub-layer" for other ULPs?
One can envision simple examples of such a sublayer in the multihomed 
host case - e.g. a table which indicates that the VoIP port numbers 
should prefer the GPRS interface over the WLAN interface i.e. 
preferences on the local locators.

> - demuxing, which is mostly specific to ULIDs but needs help from  
> something else (flow label, tag, locator set)
> - reachability detection, which happens address pair to address pair
> - reachability acks, which happen from host to host
> 
> 

    Erik

Delivery-date: Tue, 23 Aug 2005 22:13:36 +0000
Date: Tue, 23 Aug 2005 15:13:51 -0700
From: Erik Nordmark <[email-masked]>
To: [email-masked]
CC: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: shim-aware transports

[email-masked] wrote:

> I might be missing something here, but if a ULP forks-off of a shim
> session toand uses a different locator pair than the default, shouldn't
> this be signaled to the other end point, otherwise we run the risk
> that each endpoint might get out of 'sync' from each other.  

I don't think there is any out of sync problem with the uncoordinated 
"forking". A has a set of locator pairs it is using when sending to B, 
and B accept all those locator pairs (and context tag) as mapping to the 
same ULID pair.

A chooses which locator pair to use for any given packet, based on its 
probing and path exploration (after a suspected failure), and it doesn't 
need to coordinate this with B.

If we add the uncoordinated forking, this just means that A might, more 
or less persistently, use different locator pairs for different classes 
of traffic (e.g., {A1, B3} for "time sensitive" and {A2, B2} for 
"bulk"), which doesn't have any impact on B, since B will map those to 
the same ULID pair.

> If endpoint a decides to fork-off and start using locater a1' rather
> than
> a1, the ULP at the opposite end might still be using a1 to send packets
> to endpoint a.  

Yes, but that "uncoordinated" property is what we need to handle 
unidirectional locator pairs in any case.

FWIW it isn't clear to me that we need a "coordinated forking" 
mechanism; it seems to be counter to the idea that each end can decide 
which locator pair to use when sending.

    Erik

Delivery-date: Tue, 23 Aug 2005 17:15:28 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: shim proxy (was Re: failure detection)
Date: Tue, 23 Aug 2005 19:15:54 +0200
To: Paul Jakma <[email-masked]>

El 23/08/2005, a las 17:55, Paul Jakma escribi=F3:

> On Tue, 23 Aug 2005, marcelo bagnulo braun wrote:
>
>> Some questions about the scheme that you are considering:
>
>> - What upper layer identifiers are used in the endpoints? in=20
>> particular which prefixes do they contain? global unicast or a=20
>> special purpose prefix (as in GSE)?
>

> To Be Assigned, I guess. I don't know.
>

i guess that my point is that when you start to actually define how=20
this things are done, isseus start to pop up, in particular, issues=20
with the security mechanisms (basically because when you want to=20
provide proxied security, the trust model becomes more complex)

>> - Are the endpoints of the communication aware of the prefix sets=20
>> (their own and the peer)? or just the proxy is aware of them?
>
> The proxied hosts, no. They'd happily think they're using normal IPv6.=20=

> Just the proxies with intermediate their access to rest of world would=20=

> recognise their IPv6 network prefix and host identifier as being a=20
> shim6 ULID.
>
>> - How do they (endpoint and/or proxy) learn the prefix set of the=20
>> peer? how are they secured?
>
> The remote shim6 peer? Via the TBD shim6 protocol.
>
>> - How does the security mechanism for securing the prefix set and the=20=

>> identifier interact with the proxy and endpoint?
>
> I'm not sure I understand, could you elaborate?
>

see the threat that Iljitsch started for additional insights of the=20
complexity of not having the endpoints aware of the locator set and the=20=

ways to secure this

regards, marcelo

>> i was referring to the threats described in=20
>> draft-ietf-multi6-multihoming-threats-03.txt which need to be dealt=20=

>> with
>
> Thanks, I'll have a look.
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Practice is the best of all instructors.
> 		-- Publilius
>

Delivery-date: Tue, 23 Aug 2005 16:25:36 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: Rewriting and proxies
Date: Tue, 23 Aug 2005 18:25:14 +0200
To: marcelo bagnulo braun <[email-masked]>

On 23-aug-2005, at 18:17, marcelo bagnulo braun wrote:

>> One way around this would be for the shim proxy to intercept  
>> packets from the hosts that don't implement the shim, and  
>> immediately try to set up a shim-enabled session using an HBA/CGA  
>> compatible address. So host A sends out a packet to host B with  
>> address A1(eui64), and the proxy immediately rewrites this into A1 
>> (hba). If the shim negotiation then completes, the proxy can  
>> inform the other side that the ULID is actually A1(eui64) and  
>> everything works completely transparently. If the shim  
>> negotiations fail, the proxy can either continue the session and  
>> keep rewriting, which is faster but not very clean,

> But in this case, the ULID used by the proxied host and the ULID  
> perceived the peer would be different wouldn't they?

Right. However, using the term "ULID" makes it seem like this still  
uses shim6, but it doesn't: this is the fallback mode to backward  
compatible non-shim communication.

> If so, this would imply that the rewriting is no longer transparent  
> to the upper layers, which i think we agreed that was not such a  
> good idea (all the nat implications an so on)

Right, that's why I also included:

>> or send the original A1(eui64) packet.

> An alternative that we discussed a while ago was to let the proxy  
> assign the HBA/CGA addresses to the proxied host (through dhcp for  
> instance) so that the proxied host has the correct addresses  
> configured, and the proxy can use the security features of such  
> addresses to secure the shim protocol locator exchange even if the  
> proxied host ois not aware of it.

That's also possible, but it wouldn't be without deployment issues.

Iljitsch

Delivery-date: Tue, 23 Aug 2005 16:17:00 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Rewriting and proxies
Date: Tue, 23 Aug 2005 18:17:46 +0200
To: Iljitsch van Beijnum <[email-masked]>

El 23/08/2005, a las 17:42, Iljitsch van Beijnum escribi=F3:

> Hi,
>
> As I wrote a response to one of Paul's messages a few days ago it=20
> occurred to me that if we add a demultiplexing header ("if" in the=20
> sense that sometimes we do, but when we don't want/need it we don't)=20=

> we don't really need the source address in the packet for=20
> demultiplexing anymore, so source address rewriting by routers would=20=

> be easy to do.
>
> Since the destination address is of very little use for multiplexing,=20=

> this means that pretty much all demultiplexing must be done based on=20=

> the demux tag, and also that these tags can't overlap between=20
> different correspondents, which means that they must be allocated by=20=

> the receiver and not the sender. But I think 24 to 32 bits is enough=20=

> for this. We can add another two 4 bit or so fields that point to the=20=

> source and destination ULIDs so it becomes possible to have sessions=20=

> to/from different ULIDs share a demux tag.
>
> Are there any security problems with ignoring the source address on=20
> demux? I guess it's slightly less secure than also considering the=20
> source address, as someone who doesn't have the capability to spoof=20
> the source address but does manage to find the source/dest-demux field=20=

> mapping can now spoof, while that person previously could not.
>
> With HBA/CGA we must depend on the lower 64 bits of the source address=20=

> for security, which makes implementation of the shim in a proxy or=20
> middlebox harder, as we still need to get the real host (which doesn't=20=

> know about the shim) to use the HBA/CGA compatible address.
>
> One way around this would be for the shim proxy to intercept packets=20=

> from the hosts that don't implement the shim, and immediately try to=20=

> set up a shim-enabled session using an HBA/CGA compatible address. So=20=

> host A sends out a packet to host B with address A1(eui64), and the=20
> proxy immediately rewrites this into A1(hba). If the shim negotiation=20=

> then completes, the proxy can inform the other side that the ULID is=20=

> actually A1(eui64) and everything works completely transparently. If=20=

> the shim negotiations fail, the proxy can either continue the session=20=

> and keep rewriting, which is faster but not very clean, or send the=20
> original A1(eui64) packet.
>

But in this case, the ULID used by the proxied host and the ULID=20
perceived the peer would be different wouldn't they?

If so, this would imply that the rewriting is no longer transparent to=20=

the upper layers, which i think we agreed that was not such a good idea=20=

(all the nat implications an so on)

An alternative that we discussed a while ago was to let the proxy=20
assign the HBA/CGA addresses to the proxied host (through dhcp for=20
instance) so that the proxied host has the correct addresses=20
configured, and the proxy can use the security features of such=20
addresses to secure the shim protocol locator exchange even if the=20
proxied host ois not aware of it.

regards, marcelo

> In any case, there will be a slight delay. Of course, the proxy may=20
> cache remote capabilities and not try to do any shimming towards=20
> correspondents that didn't support the shim earlier. The main thing=20
> that we lose here is the ability to upgrade a non-shim session to a=20
> shim session.
>
> Alternatively, we can add an additional security mechanism that allows=20=

> correspondents to authenticate proxies without relying on HBA/CGA.
> Since presumably, having a shim proxy is something that will appeal to=20=

> non-tiny organizations, it could be ok to use a standard SSL (like)=20
> certificate for this.
>
> Thoughts?
>
> Iljitsch
>

Delivery-date: Tue, 23 Aug 2005 15:56:21 +0000
Date: Tue, 23 Aug 2005 16:55:56 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: shim proxy (was Re: failure detection)
Mail-Copies-To: [email-masked]

On Tue, 23 Aug 2005, marcelo bagnulo braun wrote:

> Some questions about the scheme that you are considering:

> - What upper layer identifiers are used in the endpoints? in particular which 
> prefixes do they contain? global unicast or a special purpose prefix (as in 
> GSE)?

To Be Assigned, I guess. I don't know.

> - Are the endpoints of the communication aware of the prefix sets (their own 
> and the peer)? or just the proxy is aware of them?

The proxied hosts, no. They'd happily think they're using normal 
IPv6. Just the proxies with intermediate their access to rest of 
world would recognise their IPv6 network prefix and host identifier 
as being a shim6 ULID.

> - How do they (endpoint and/or proxy) learn the prefix set of the peer? how 
> are they secured?

The remote shim6 peer? Via the TBD shim6 protocol.

> - How does the security mechanism for securing the prefix set and the 
> identifier interact with the proxy and endpoint?

I'm not sure I understand, could you elaborate?

> i was referring to the threats described in 
> draft-ietf-multi6-multihoming-threats-03.txt which need to be dealt with

Thanks, I'll have a look.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Practice is the best of all instructors.
 		-- Publilius

Delivery-date: Tue, 23 Aug 2005 15:43:25 +0000
To: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Rewriting and proxies
Date: Tue, 23 Aug 2005 17:42:31 +0200

Hi,

As I wrote a response to one of Paul's messages a few days ago it  
occurred to me that if we add a demultiplexing header ("if" in the  
sense that sometimes we do, but when we don't want/need it we don't)  
we don't really need the source address in the packet for  
demultiplexing anymore, so source address rewriting by routers would  
be easy to do.

Since the destination address is of very little use for multiplexing,  
this means that pretty much all demultiplexing must be done based on  
the demux tag, and also that these tags can't overlap between  
different correspondents, which means that they must be allocated by  
the receiver and not the sender. But I think 24 to 32 bits is enough  
for this. We can add another two 4 bit or so fields that point to the  
source and destination ULIDs so it becomes possible to have sessions  
to/from different ULIDs share a demux tag.

Are there any security problems with ignoring the source address on  
demux? I guess it's slightly less secure than also considering the  
source address, as someone who doesn't have the capability to spoof  
the source address but does manage to find the source/dest-demux  
field mapping can now spoof, while that person previously could not.

With HBA/CGA we must depend on the lower 64 bits of the source  
address for security, which makes implementation of the shim in a  
proxy or middlebox harder, as we still need to get the real host  
(which doesn't know about the shim) to use the HBA/CGA compatible  
address.

One way around this would be for the shim proxy to intercept packets  
from the hosts that don't implement the shim, and immediately try to  
set up a shim-enabled session using an HBA/CGA compatible address. So  
host A sends out a packet to host B with address A1(eui64), and the  
proxy immediately rewrites this into A1(hba). If the shim negotiation  
then completes, the proxy can inform the other side that the ULID is  
actually A1(eui64) and everything works completely transparently. If  
the shim negotiations fail, the proxy can either continue the session  
and keep rewriting, which is faster but not very clean, or send the  
original A1(eui64) packet.

In any case, there will be a slight delay. Of course, the proxy may  
cache remote capabilities and not try to do any shimming towards  
correspondents that didn't support the shim earlier. The main thing  
that we lose here is the ability to upgrade a non-shim session to a  
shim session.

Alternatively, we can add an additional security mechanism that  
allows correspondents to authenticate proxies without relying on HBA/ 
CGA. Since presumably, having a shim proxy is something that will  
appeal to non-tiny organizations, it could be ok to use a standard  
SSL (like) certificate for this.

Thoughts?

Iljitsch

Delivery-date: Tue, 23 Aug 2005 14:05:52 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: shim proxy (was Re: failure detection)
Date: Tue, 23 Aug 2005 16:06:39 +0200
To: Paul Jakma <[email-masked]>

El 22/08/2005, a las 16:40, Paul Jakma escribi=F3:

> On Mon, 22 Aug 2005, marcelo bagnulo braun wrote:
>
>> the problem is that the is no way to prove the binding between the=20
>> identifier and their locator sets... i.e. any prefix could be used=20
>> with any identifier and it would be ok, so any rewriting would be ok,=20=

>> hence the potential attacks...
>
> If, as a subset of all ULIDs, we allow a set of ULIDs to be composed=20=

> of a network identifier (ie the first 64 bits) and a host identifier=20=

> (last / least significant 64 bits), ie that the ULID essentially be a=20=

> valid IPv6 address (which the shim6 drafts anticipate being possible),=20=

> then the 'proxy' can have a static mapping which need only map the=20
> /network/ portion of the ULID to the network portion of a locator. Ie=20=

> leaving the host portion unchanged.
>
> The security implications are no different from normal static=20
> forwarding, as far as I can tell.
>

Not sure...

Some questions about the scheme that you are considering:
- What upper layer identifiers are used in the endpoints? in particular=20=

which prefixes do they contain? global unicast or a special purpose=20
prefix (as in GSE)?
- Are the endpoints of the communication aware of the prefix sets=20
(their own and the peer)? or just the proxy is aware of them?
- How do they (endpoint and/or proxy) learn the prefix set of the peer?=20=

how are they secured?
- How does the security mechanism for securing the prefix set and the=20
identifier interact with the proxy and endpoint?

>> Perhaps you could try to evaluate how would such solution cope with=20=

>> the threats described in the threat analysis...
>
> I don't see the threat.

i was referring to the threats described in=20
draft-ietf-multi6-multihoming-threats-03.txt which need to be dealt=20
with

regards, marcelo

>
>> as i said, i consider this proxy capability to be really interesting,=20=

>> but i am afraid you are underestimating the security issues here.
>
> Possible :).
>
>> regards, marcelo
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Don't put off for tomorrow what you can do today because if you enjoy=20=

> it today,
> you can do it again tomorrow.
>

Delivery-date: Tue, 23 Aug 2005 06:04:47 +0000
Content-class: urn:content-classes:message
Subject: RE: shim-aware transports
Date: Tue, 23 Aug 2005 09:04:27 +0300
From: <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>, <[email-masked]>

Pekka,

>I have a feeling we have some sort of a communication problem=20
>here but that we fundamentally agree; this message is an=20
>attempt to clarify what I am trying to say.
>
>I am all for the initial goal of keeping the ULPs unchanged. =20

To finese this a bit, I would hope our goal would be that shim6
would work if the ULPs are unchanged, but ULPs might benefit
from some extra richness that the shim would provide.  I guess
what I am wondering is, do we create some extra richness in=20
socket options (for example) or whatever is the OS API for the
shim layer, what ULPs could take advantage of.

>That's the whole point.  However, when doing so I also think that we
>*probably* should adopt the conservative approach adopted by e.g. =20
>SCTP, i.e., to have one preferred locator and to use the rest=20
>as equal backups.  Two reasons: keep it simple, and less=20
>surprises to the transport algorithms.

I agree in the basic shim protocol, that we should keep it simple.

>Now, AFAICT, the discussion below cornered around what to do=20
>next, after the initial goal has been reached.  What I=20
>remember from Paris we all seem to agree that we need some=20
>kind of an interface between the shim layer and ULP.
>
>My two messages (URLs below) define a strawman for such an=20
>interface, while trying to keep the system at the same time as=20
>simple as possible.  I don't see that changing at all the way=20
>the shim works with unmodified ULPs, or what state it maintains.
>
>On the other hand, what I find somewhat worrisome are=20
>allusions that we should make the state at the shim layer more=20
>complex as we add multi-addressing support to ULPs.  It looks=20
>to me that some folks would like to do locator-policy=20
>decisions at multi-addressing transports, and then push those=20
>policies somehow to the shim, to be executed there.  I find=20
>that unnecessarily complex.  Why can't the ULP just pick the=20
>locators, if it knows about them; in that case the shim just=20
>protects the transport from using locators that are no longer valid.

Isn't this an address selection problem?  A multi-interface host will
have a lot of IP addresses, esp. if there are additional tunnel
interfaces,
privacy addresses, transition mechanisms, etc.  The shim will somehow
need=20
to have some simple rules for deciding which of these are actually
useful.
There probably will be some issues with address selection at the shim
layer and then identity selection at the ULP.

John

Delivery-date: Tue, 23 Aug 2005 05:53:53 +0000
Content-class: urn:content-classes:message
Subject: RE: shim-aware transports
Date: Tue, 23 Aug 2005 08:53:41 +0300
From: <[email-masked]>
To: <[email-masked]>, <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>

Geoff,

>Brian noted that  making the shim  work with unmodified IP and=20
>unmodified upper layers needs to get shipped first, and thats=20
>a good priority.
>
>Brian also noted that this implies a two-way API between the=20
>shim and the ULP, in which the necessary events are=20
>transmitted up or down.
>
>So I'd like to float the concept of a placeholder about the=20
>interaction with transport that hopefully will allow the=20
>priority work to proceed while still keeping enough=20
>flexibility around to allow transport sessions to be able to=20
>be more explicit about what they want from the shim layer, but=20
>to define this interaction at a later stage in the WG's life.

I agree, and I think that this will buy us something - it will
hopefully give us a more light-weight mechanism without=20
overloading of a lot of functionality (i.e. - state) inside
of the sim layer.

>It seems to me that the basic mechanics of this can be=20
>satisfied by defining a rich enough signalling interface=20
>between the shim layer and the upper level sessions - perhaps=20
>allowing for the inclusion of the relevant locator set, and=20
>allowing attributes to be placed on individual locators as=20
>well as on the complete set in either direction. In other=20
>words this allows an upper level to be more specific about=20
>locator preference either by attribute or by explicit choice=20
>by passing a rich information bundle to the ULP via this=20
>signalling interface.

Agreed.

>Placing the potential functionality in the hands of the=20
>session layer also infers that the shim state may not=20
>necessarily be endpoint-to-endpoint, but may operate at a=20
>finer level of selection (the 'equivalence" topic I referred=20
>to at the WG meeting). It also infers that an individual=20
>session may want to trigger a locator change that appears to=20
>be an isolated request
>- in which case there appears to be a logical need for dynamic=20
>shim state forking in some form (i.e. a shim state may state=20
>in a general host-to-host state, but individual sessions may=20
>'fork' out of this general state by requesting individual=20
>locator change trigger settings, for example.)
>
>If you allow vertical (and peer-to-err horizontal) locator=20
>attribute signalling, and also dynamic shim state forking you=20
>can then say  that the mechanisms for the ULP to signal quite=20
>explicit hints to the shim layer as to when and what locator=20
>to use, and for the shim to respond with a forked state as=20
>required and signal this to the peer remote shim entity is=20
>within the capability of the protocol. How the fields are=20
>defined and used are up to a later incarnation of the or=20
>another WG, that will define the attributes and  their semantics.
>
>Does this help?

I think that this might be a useful way to procede.

John

Delivery-date: Tue, 23 Aug 2005 05:51:11 +0000
Content-class: urn:content-classes:message
Subject: RE: Thoughts about layering multi-addressing 
Date: Tue, 23 Aug 2005 08:50:37 +0300
From: <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>

Pekka,

Couple points embedded:

>So, before I/we start writing a draft in this space, I'd like=20
>to poll the WG opinion on the viability of the basic approach,=20
>as outlined below.  Note that this is an area where I really=20
>think that running code or simulations would help.  We also=20
>need to get some transport area people involved.

I agree with this, I also wonder if anyone has any input from
how upper-layer protocols interact with SCTP.  I know this isn't
directly equivalent, but as SCTP is providing a handle to ULPs
which acts as a locator; SCTP provides path-probing and some other
features, I think that we might already have some insights into
what it does well & what can be improved.

>Getting to the subject, the current "theory" of shim6=20
>implementation seems to be that the shim6 layer rewrites the=20
>locators to identifiers for incoming packets and vice versa=20
>for outgoing packets.  While that may be fine in the short run=20
>for TCP and UDP, we seem to want a different approach for=20
>SCTP, probably also for TCP in the longer run, and maybe even for UDP.

I agree.  At least with SCTP, my guess is that it might want to use
multiple locators or handles to the shim layer.  I don't think that
we should force multi-homing aware transports into using a single
locator.

[text clipped]

>The architecturally guiding principle in this is to keep IP=20
>only aware of the plain set of locators associated with each=20
>ULID, the required encapsulation/marking information (such as=20
>the flow label), =20
>and nothing else.   Annotating the ULIDs with locators (or vice =20
>versa) allows more capable transport protocols to keep track=20
>of them and associate path-related parameters such as RTT,=20
>bandwidth estimate, etc, separately with each locator pair. =20
>This allows the transport-related functionality to remain=20
>located at the transport layer.  At the same time, any changes=20
>in the locator set need to be secured only at the shim layer,=20
>i.e. only once per each ULID pair rather than separately for=20
>each transport connection.

I agree.  Geoff has said, in the past, that he sees some vertical
signaling needed; this also could be considered as part of the=20
'rich API' for the shim layer. =20

>In addition to this basic mechanism, some transport may need=20
>to figure out the complete set of locators available. =20
>However, are there any other functions that would be needed? =20
>For example, if the transport can give outgoing locators as=20
>hints to the shim-layer, it does not need to tell the=20
>shim-layer to deprecate any locators even though it suspects=20
>that some of they may not work any more.  Indeed, telling shim=20
>to prefer some over others may be dangerous since different=20
>locator pairs may work for different transports, due to the=20
>Internet not being fully transparent any more.

Pekka, who is doing the 'telling' to the shim layer?  If it is
the transport layer, then I think we are a bit safer, especially
if ULPs can fork-off of the main shim state if they realize
that the prefered locator pair isn't working for them.  Another
mechanism might be some sort of 'policy' object that gives a
preference or default for a locator.  My guess is that there
may be some overlap with the "Distribution of RFC 3484 address selection
policies"
thread in the IPv6 wg; basically being practical, multihoming support
in IPv6 seems especially critical in enterprise environments,
and I am sure that netadmins will want to inform users of
prefered paths.

>Hence, my question is if the transport are likely to need=20
>anything else than the following:
>   - both ULIDs and locators in incoming packets
>   - ability to define locators for outgoing packets
>     - what happens if the locators are not available any
>       more?  Host-internal ICMP-like response?
>   - ability to get the full set of locators related
>     to a given ULID
>Anything else?

Those seem like good first cut requirements, some of the other
things are more optional features but support would be needed,
IMO, for:

 - some level of failure detection, especially at interface=20
   (for example, the ethernet cable is unplugged or the access
   router is out).  If I am not mistaken, this might be the
   level that Paul is getting at.
 - support for some simple policy mechanism or for signaling
   prefered locator.

>One other practical problem here is to figure out how to make=20
>this all work with high speed servers, which are likely to=20
>have IP and TCP implemented with firmware at the NIC.  (Kudos=20
>to Christian for pointing this out.)

That's engineering!

John

Delivery-date: Tue, 23 Aug 2005 05:27:03 +0000
Content-class: urn:content-classes:message
Subject: RE: shim-aware transports
Date: Tue, 23 Aug 2005 08:26:20 +0300
From: <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>, <[email-masked]>

Erik & Geoff,

>> Placing the potential functionality in the hands of the session layer

>> also infers that the shim state may not necessarily be=20
>> endpoint-to-endpoint, but may operate at a finer level of selection=20
>> (the 'equivalence" topic I referred to at the WG meeting). It also=20
>> infers that an individual session may want to trigger a locator
change=20
>> that appears to be an isolated request - in which case there appears=20
>> to be a logical need for dynamic shim state forking in some form
(i.e.=20
>> a shim state may state in a general host-to-host state, but
individual=20
>> sessions may 'fork' out of this general state by requesting
individual=20
>> locator change trigger settings, for example.)
>
>Geoff,
>
>This notion of forking makes sense, but it isn't clear to me=20
>whether it is a local notion at the sending side, with the ULP=20
>at each end being able to make independent decisions (perhaps=20
>using some ULP signaling), or whether it is a shim-coordinated=20
>decision.
>
>I think we can define it to be a local thing at the sending=20
>side, where a ULP can signal that a given class of packets=20
>between a ULP pair (or even at the granularity of each packet)=20
>use a different locator pair than the default. The receiver=20
>will happily accept such packets, because the locator pair is=20
>in the negotiated set of locators for the ULP pair context.
>
>Do you think we'd loose something important if there isn't=20
>shim signaling to have the two ends agree to switch the=20
>locator pair for both directions?

I might be missing something here, but if a ULP forks-off of a shim
session toand uses a different locator pair than the default, shouldn't
this be signaled to the other end point, otherwise we run the risk
that each endpoint might get out of 'sync' from each other. =20

If endpoint a decides to fork-off and start using locater a1' rather
than
a1, the ULP at the opposite end might still be using a1 to send packets
to endpoint a. =20

John

Delivery-date: Mon, 22 Aug 2005 14:41:04 +0000
Date: Mon, 22 Aug 2005 15:40:36 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: shim proxy (was Re: failure detection)
Mail-Copies-To: [email-masked]

On Mon, 22 Aug 2005, marcelo bagnulo braun wrote:

> the problem is that the is no way to prove the binding between the 
> identifier and their locator sets... i.e. any prefix could be used 
> with any identifier and it would be ok, so any rewriting would be 
> ok, hence the potential attacks...

If, as a subset of all ULIDs, we allow a set of ULIDs to be composed 
of a network identifier (ie the first 64 bits) and a host identifier 
(last / least significant 64 bits), ie that the ULID essentially be a 
valid IPv6 address (which the shim6 drafts anticipate being 
possible), then the 'proxy' can have a static mapping which need only 
map the /network/ portion of the ULID to the network portion of a 
locator. Ie leaving the host portion unchanged.

The security implications are no different from normal static 
forwarding, as far as I can tell.

> Perhaps you could try to evaluate how would such solution cope with 
> the threats described in the threat analysis...

I don't see the threat.

> as i said, i consider this proxy capability to be really 
> interesting, but i am afraid you are underestimating the security 
> issues here.

Possible :).

> regards, marcelo

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Don't put off for tomorrow what you can do today because if you enjoy it today,
you can do it again tomorrow.

Delivery-date: Mon, 22 Aug 2005 14:27:11 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: shim proxy (was Re: failure detection)
Date: Mon, 22 Aug 2005 16:27:14 +0200
To: Paul Jakma <[email-masked]>

El 20/08/2005, a las 19:02, Paul Jakma escribi=F3:
>
> Yes, as a subset of shim6. Ie, using an 8+8 static mapping for the=20
> local ULID(s), and using whatever shim6 control messages are needed to=20=

> map the remote ULID to the correct remote locator(s). It would be=20
> assumed that the ULIDs are composed of a prefix and a host ID,=20
> obviously. The mapping would only change the prefix.
>
> I think it would be very useful to allow such a mapping, and hence=20
> allow split/proxy shim6.
>
>> i agree it would be useful but i still not sure how do you deal with=20=

>> security stuff in this case...
>
> There are no security implications to static mapping. It never =
changes.

the problem is that the is no way to prove the binding between the=20
identifier and their locator sets... i.e. any prefix could be used with=20=

any identifier and it would be ok, so any rewriting would be ok, hence=20=

the potential attacks...

Perhaps you could try to evaluate how would such solution cope with the=20=

threats described in the threat analysis...

as i said, i consider this proxy capability to be really interesting,=20
but i am afraid you are underestimating the security issues here.

regards, marcelo

>  No more than there are security implications to a non-shim6 host=20
> forwarding packets according to a static routing table of=20
> destination->nexthop.
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Those who can, do; those who can't, write.
> Those who can't write work for the Bell Labs Record.
>

Delivery-date: Sat, 20 Aug 2005 17:04:13 +0000
Date: Sat, 20 Aug 2005 18:02:46 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Sat, 20 Aug 2005, marcelo bagnulo braun wrote:

> ok, i guess we have come to key point here.

Maybe.

Note again, nothing in shim6 should preclude what you want. I just 
don't think the 'perfect' case you desire should be mandated as the 
default case.

> We agree that the proposed mechanism proposed for the shim is what 
> is needed to deal with all failure modes and to identify if there 
> is at least one working path right?

Yes.

> We seem to disagree about if the cost that implies is worth it, right?

Yes.

> you seem to consider that there are simpler methods that would deal with a 
> significant amount of the common failure modes, in particular the one you 
> detail above.

Yep.

> I guess that probably RFC3178 already provides a reasonable 
> solution that provides a the protection level that you ask for. I 
> mean RFC3178 protects from failures in the edges in a transparent 
> fashion

That's one solution.

Like I said, there are already /lots/ of other ways and as yet 
unknown future ways to cope with routing failures. Ignoring all the 
other possibilities and just mandating n^2 probing in shim6 seems 
unwise to me - particularly when those other ways could be far more 
efficient. (And particularly when, imho, the failure modes you're 
concerned about are not /that/ common imho).

>> A host could have the following default route:
>> 
>> default via ISP1-gateway
>> 	via ISP2-gateway

> what if there is a single router in a link of the multihomed site? 
> i mean, you cannot assume that in all links of the multihoemd site 
> there will be as many routers as ISPs the site is multihomed too, 
> right?

That's possible. But I don't know of many ISPs who 'share' their 
routers. And if they do, they must co-ordinate its configuration. Eg, 
if a router serves both ISP A and B, it obviously will not be 
filtering out packets with source of either A or B ;).

> In this point, i guess you end up requiring source address based 
> routing in the multihomed site, in order to allow the end host to 
> force routing through the selected exit ISP and the shim using the 
> source address to actually select the exit ISP hence the shim 
> selecting the source address, i guess

Maybe, depends.

>> ISP1-gateway device X src ISP1-PA-address
>> ISP2-gateway device X src ISP2-PA-address

> not sure what you mean.. are you thinking in something like GSE here?

Yes, as a subset of shim6. Ie, using an 8+8 static mapping for the 
local ULID(s), and using whatever shim6 control messages are needed 
to map the remote ULID to the correct remote locator(s). It would be 
assumed that the ULIDs are composed of a prefix and a host ID, 
obviously. The mapping would only change the prefix.

I think it would be very useful to allow such a mapping, and hence 
allow split/proxy shim6.

> i agree it would be useful but i still not sure how do you deal 
> with security stuff in this case...

There are no security implications to static mapping. It never 
changes. No more than there are security implications to a non-shim6 
host forwarding packets according to a static routing table of 
destination->nexthop.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Those who can, do; those who can't, write.
Those who can't write work for the Bell Labs Record.

Delivery-date: Sat, 20 Aug 2005 14:34:35 +0000
Cc: shim6 <[email-masked]>, Tony Li <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Sat, 20 Aug 2005 16:35:19 +0200
To: Pekka Nikander <[email-masked]>

El 20/08/2005, a las 9:18, Pekka Nikander escribi=F3:

...

> We have traditionally taken care of path properties at the transport. =20=

>  Now we are moving some of this functionality to the networking layer,=20=

> which may be the right direction in the very long run.   However, IMHO=20=

> we should aim for a conservative approach where we try to keep the=20
> shim as simple as possible, and keep the path property discovery at=20
> the transport layer.
>
> In some way we seem to have two contending goals here:
>
> 1. keep locator sets at the IP layer, as a locator set is a property=20=

> of a host and not a connection, and
>
> 2. keep path property discovery at the transport layer, as it has=20
> traditionally been done, separated from IP
>

...but we are including failure detection and path availability=20
exploration mechanisms in the shim, right?

I mean, current design of shim includes (within the shim) the=20
capability of detecting that an address pair in no longer working and=20
also the capability of exploring alternative working address pairs

So, somehow we are including some parts of the path property discovery=20=

in the shim. However, the path property that is being discovered by the=20=

shim is quiet basic, just working/not working address pair

I assume that what you mean above is that the discovery of more=20
advanced path properties like latency jitter RTT should not be=20
performed by the shim at this stage, is this correct?

> However, I don't consider these to be that incompatible; anyway, we=20
> can have joint data structures that allows this distinction to be=20
> kept.  Something like Dave's and Avri's CELP, or maybe what Brian=20
> alluded.  The flip side of this is, of course, that this breaks the=20
> layering abstraction, somewhat.  Consequently, I think that in the=20
> very long run we should reconsider where to place the traditional=20
> transport functionality; my gut feeling is that we may need to split=20=

> it into path-dependent and connection-dependent parts, separated by a=20=

> multi-addressing layer.
>
> So, it looks to me that we are facing something like the following=20
> steps:
>
> 1. Minimally intelligent shim, one primary locator, unchanged
>    transports
>
> 2. Shim as in #1, richer interface between shim and ULPs,
>    shim-aware transports being able to use multiple locators
>    at the same time
>
> 3. Further research on the proper new location(s) of traditional
>    transport functions in the stack
>
> #2 and #3 are clearly research at this point of time, and outside the=20=

> scope of the WG charter.  As I wrote in my first message in this=20
> thread, I would very much see running code or simulations in that=20
> space; without that we'd be driving in the dark.
>

imho #3 is out of the scope clearly but i don't have a strong position=20=

about #2, perhaps some work in this area would be interesting...

regards, marcelo

> However, I am worried that we may be trying to jump directly to #3 and=20=

> design too much functionality to the shim.  KISS vs. over-engineering,=20=

> if you will.
>
> --Pekka Nikander
>

Delivery-date: Sat, 20 Aug 2005 14:32:26 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: about reachability detection draft
Date: Sat, 20 Aug 2005 16:31:45 +0200
To: marcelo bagnulo braun <[email-masked]>

On 20-aug-2005, at 7:34, marcelo bagnulo braun wrote:

>>> So, A informs B that the B->A path has failed (this implies some  
>>> form of signaling from A to B, which is likely to be required to  
>>> be somehow reliable, which is why i see a potential difficulty here)

>> The way I see it, A just starts a path exploration procedure. This  
>> means it will try address pairs until it gets an answer from B, so  
>> unless the path from A to B is completely broken at some point B  
>> will see a path exploration packet from A, which indicates to B  
>> that there is probably something wrong, so B would start doing its  
>> own path exploration.

> but this is somehow weird, because A is performing a path  
> exploration procedure, even if the path that A is using to send  
> packet is working properly...

It's not so strange: A doesn't know if it's the path from A to B that  
has failed or the one from B to A (or both are still working and  
there was just some random packet loss).

> I mean, actually, in this scenario, you are using the path  
> exploration procedure initiated by A as a reliable mechanism to  
> allow A to inform B that something is wrong in the path that B is  
> using to send packets, so that B starts its own path exploration  
> procedure, which is the one that is the one that in fact is  
> relevant, since is the B->A path which is broken.

I guess so.

> So, in fact the FBD procedure has the following steps, as i see it:
> 1- Both ends send keepalives to preserve the traffic frequency  
> required.
> 2- when something is wrong, the receiving end detects the failure

Yes.

> 3- Then the receiving end convey the information about the failure  
> to the sender (which the one that has to react). Such signaling  
> must be performed in a somehow reliable way (in particular, you can  
> use the path exploration procedure since it tries multiple paths)
> 4- The the sender can react, performing in its turn a path  
> exploration procedure.

> agree?

Yes, except that we don't know for sure which direction failed.

> i think it is important to identify that step 3 is required in  
> order to complete the failure detection procedure, since it is not  
> only important to detect the failure but also to inform the party  
> that can actually do something about it (i.e. the sender) that the  
> failure has occurred

I suppose I didn't feel the need to state this so explicitly, but  
it's certainly true.

> If we use the other mechanism described in the draft (i don't  
> remember how did you call this in your presentation) the sender is  
> the one that actually detects the failure so there is no need for  
> step 3

No, that's not exactly true. Let me present a different model:

1. Traffic is flowing in both directions, transport is happy
2. Traffic is no longer flowing in both directions, transport is unhappy
3. Side A starts full path exploration
4a. It is determined that the path from A to B failed
5a. A changes to a different locator pair
4b. It is determined that the path from B to A failed
5b. B changes to a different locator pair

As you can see there is no difference, except in the inner workings  
in steps 1 and 2, where FBD just monitors incoming packets, and CUD  
either relies on the transport feedback or sends probes and gets back  
replies.

>> Agree, but on the other hand we don't want to give attackers the  
>> capability to make hosts think there are reachability problems,  
>> especially attackers that aren't able to sniff any of the setup  
>> traffic. So something simple like the TCP sequence number or the  
>> IPsec replay counter would be useful here.

> the attack that you have in mind then is that an attacker generates  
> a reachability test packet and the host reads this packets as the  
> other end starting the path exploration procedure, which implies  
> that itself should do the same because of a potential outage, right?

Yes.

> But this would be in the case of FBD right? i mean does this  
> applies in the other mechanism too? I think not, becuase in this  
> case, reachability tests are a two way exchange and wouldn't imply  
> that the other end starts doing its own reachability test, i think

?

> In any case, you could also include some cookie generated during  
> the session establishment exchange to even make this stronger.

Yes, should be no problem.

>>> Well, deprecation means that somewhere in the near future, the  
>>> address won't be available anymore, right?

>> Right, so we keep using it until it's not available anymore.  :-)

> Suppose that a prefix PrefA is being renumbered.
> In order to remove this prefix, all addresses containing PrefA are  
> deprecated. This means that this prefix should not be used for  
> establishing new communications, but can be used for ongoing  
> communications, right?

Right.

> Now a certain point in time the prefix is actually removed, this  
> means that packets containing a destination address with this  
> prefix won't reach the site any more.

Right.

> this would be that the prefix and the associated addresses are no  
> longer available according to your terminology right? So my  
> question is how does the host knows that the prefix is no longer  
> available? is there any other signaling tool besides deprecation  
> through RAdv?

First the "preferred lifetime" expires and the address is deprecated.  
A bit later, the "valid lifetime" expires and the address is no  
longer valid. Most OSes then remove it from the system.

I guess if we're rewriting addresses anyway, we may as well give  
deprecated addresses a slightly lower priority, so all else being  
equal non-deprecated addresses are used. However, it's probably  
better to stick in backward compatibility mode as long as we can (=  
no rewriting) rather than to avoid deprecated addresses.

> BEsides, in order to return to the original path, the shim needs to  
> the probing it all the time, so that it can detect when the  
> original path is back up. I mean, this wouldn't normally occur for  
> other paths, since the shim won't be probing alternative paths  
> while the current path is working, so if we want this feature, the  
> default behaviour needs to be modified i guess.

It's probably a good idea to make the probe frequency depend on the  
difference in address preference. So when A1-B1 has a preference of  
10 and A2-B2 also has a preference of 10, we really don't need to do  
much probing to see if we can return to A1-B1. But if A2-B2 has a  
preference of 1, then we'd want to move back sooner rather than later  
so we'd send probes relatively frequently. An interesting situation  
occurs when the original session uses A1-B1 which as preference 10,  
but A2-B2 has preference 25...

Iljitsch

Delivery-date: Sat, 20 Aug 2005 14:13:22 +0000
Cc: Pekka Nikander <[email-masked]>, shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Sat, 20 Aug 2005 16:11:47 +0200
To: marcelo bagnulo braun <[email-masked]>

On 19-aug-2005, at 16:33, marcelo bagnulo braun wrote:

> In addition, i guess that the shim will need to provide some form  
> of support for expressing policy/preferences.
> I guess that the additional complexity in this case, is that these  
> preferences are per ULP rather than per ULID... but this brings us  
> to a more general problem, which is: how do we handle the case  
> where there are two ULPs communicating with a given shim context  
> but they are asking to use different locators for the communication?

Well, for outgoing packets that shouldn't be a problem. If the FTP  
application wants to use the cheap high bandwidth link while the VoIP  
application wants to use the lower bandwidth, more expensive but  
bounded-delay link, the shim can just insert the right addresses and  
send the packets toward the right next hop and everything will work  
fine.

At the remote end, there really isn't much to do: the packet is  
received and the ULIDs rewritten, and that's that. The problem starts  
when we want the other side to send back their bulk data to the  
address tied to our cheap fast line and their VoIP data to the  
address tied to our high quality line.

One way to handle this would be to assume that the apps on the other  
side tell the shim what they want and things will work out without  
specific action from us. The other side still needs to know the  
characteristics of our addresses/links, though. So we need a good way  
to express them. A simple way to do this is two metrics: a high  
priority one, that makes sure only addresses with the highest value  
are chosen when those are available, and a second tie breaker metric  
that can be used for load balancing. So when there are two addresses  
with the highest priority and one as 25 for the second metric and the  
other 75, there is a 75% chance that the second address is used.

We can use the good old low delay/high bandwidth/high reliability bit  
mask to differentiate between the needs of different applications,  
but I'm not sure if this will work out in practice.

Another approach would be for the application on our side to ask the  
transport and for the transport to ask the shim for a certain  
behavior, such as "make incoming traffic for this FTP app come in at  
address X if possible", and the shim communicates this request to the  
shim layer at the correspondent, which then takes the appropriate  
action.

Note that there is no real reason to do a full "fork" of the shim  
state: the most important thing the shim has to do is allow for the  
rewriting and demux of packets. The rewriting is impacted, but not  
the demux. In fork terminology: rather than fork off a new process,  
we're starting a new thread within the same process, which isn't  
nearly as heavy an operation. I think all of this can happen based on  
transport demux info (i.e., port numbers).

It's interesting to note that by now we have several types of  
communication happening at several levels of granularity:

- service differentiation information that is ULP session specific
- rewriting, which happens based on ULID (or more than just ULID if  
we for the ULID state??? what would be the secondary differentiator?)
- demuxing, which is mostly specific to ULIDs but needs help from  
something else (flow label, tag, locator set)
- reachability detection, which happens address pair to address pair
- reachability acks, which happen from host to host

Delivery-date: Sat, 20 Aug 2005 07:50:20 +0000
Cc: shim6 <[email-masked]>
From: Pekka Nikander <[email-masked]>
Subject: Re: shim-aware transports
Date: Sat, 20 Aug 2005 09:25:01 +0200
To: Geoff Huston <[email-masked]>

Hi Geoff,

I just became worried when you started to infer that the IP layer  
might need to have state specific to different transport / session  
specific state:

>>> ... It also infers that an individual session may want to trigger  
>>> a locator change that appears to be an isolated request - in  
>>> which case there appears to be a logical need for dynamic shim  
>>> state forking in some form (i.e. a shim state may state in a  
>>> general host-to-host state, but individual sessions may 'fork'  
>>> out of this general state by requesting individual locator change  
>>> trigger settings, for example.)

But maybe I was misreading your words.

This is also very much related to what a locator really is, given  
that IP no more transparent...

--Pekka

Delivery-date: Sat, 20 Aug 2005 07:49:50 +0000
Cc: marcelo bagnulo braun <[email-masked]>, Tony Li <[email-masked]>
From: Pekka Nikander <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Sat, 20 Aug 2005 09:18:01 +0200
To: shim6 <[email-masked]>

> It is clear that the priority is that the shim provides multihoming  
> support for unchanged ULPs
>
> The question is: after the shim is available, wouldn't it make  
> sense to allow modified ULPs to have a richer signaling with the shim?
>
> I mean, in this scenario, the shim is still used to provide some  
> basic features for multiaddressing support, like the shim protocol  
> is used to securely exchange the address set available for each  
> end, for instance, and perhaps the path exploration procedure of  
> the shim can be used to identify available alternative paths. But  
> shim capable ULPs could actually do some functions that they would  
> perform more adequately, like failure detection more adjusted to  
> their own vision of what a failure is, or enhanced referral support  
> (including all the address set in the referral)

I concur.

Additionally:

To me it looks like the (currently available) locator set is a  
property of a host and not of a connection; locators are meant to  
denote topological locations in the network, anyway.  Hence,  
architecturally, locator set management seems to belong to the  
networking layer, i.e., IP.

We have traditionally taken care of path properties at the  
transport.   Now we are moving some of this functionality to the  
networking layer, which may be the right direction in the very long  
run.   However, IMHO we should aim for a conservative approach where  
we try to keep the shim as simple as possible, and keep the path  
property discovery at the transport layer.

In some way we seem to have two contending goals here:

1. keep locator sets at the IP layer, as a locator set is a property  
of a host and not a connection, and

2. keep path property discovery at the transport layer, as it has  
traditionally been done, separated from IP

However, I don't consider these to be that incompatible; anyway, we  
can have joint data structures that allows this distinction to be  
kept.  Something like Dave's and Avri's CELP, or maybe what Brian  
alluded.  The flip side of this is, of course, that this breaks the  
layering abstraction, somewhat.  Consequently, I think that in the  
very long run we should reconsider where to place the traditional  
transport functionality; my gut feeling is that we may need to split  
it into path-dependent and connection-dependent parts, separated by a  
multi-addressing layer.

So, it looks to me that we are facing something like the following  
steps:

1. Minimally intelligent shim, one primary locator, unchanged
    transports

2. Shim as in #1, richer interface between shim and ULPs,
    shim-aware transports being able to use multiple locators
    at the same time

3. Further research on the proper new location(s) of traditional
    transport functions in the stack

#2 and #3 are clearly research at this point of time, and outside the  
scope of the WG charter.  As I wrote in my first message in this  
thread, I would very much see running code or simulations in that  
space; without that we'd be driving in the dark.

However, I am worried that we may be trying to jump directly to #3  
and design too much functionality to the shim.  KISS vs. over- 
engineering, if you will.

--Pekka Nikander

Delivery-date: Sat, 20 Aug 2005 05:44:59 +0000
Cc: Pekka Nikander <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Sat, 20 Aug 2005 07:45:44 +0200
To: Tony Li <[email-masked]>

Hi Tony,
El 20/08/2005, a las 1:43, Tony Li escribi=F3:

>
> Just a historical note:
>
> The reason that we started talking about a shim layer in the first=20
> place was to prevent changes in the ULPs.  If such changes are in fact=20=

> permissible, it would be FAR preferable to make the ULPs locator agile=20=

> and dispense with the shim entirely.
>

Right and agree

It is clear that the priority is that the shim provides multihoming=20
support for unchanged ULPs

The question is: after the shim is available, wouldn't it make sense to=20=

allow modified ULPs to have a richer signaling with the shim?

I mean, in this scenario, the shim is still used to provide some basic=20=

features for multiaddressing support, like the shim protocol is used to=20=

securely exchange the address set available for each end, for instance,=20=

and perhaps the path exploration procedure of the shim can be used to=20
identify available alternative paths. But shim capable ULPs could=20
actually do some functions that they would perform more adequately,=20
like failure detection more adjusted to their own vision of what a=20
failure is, or enhanced referral support (including all the address set=20=

in the referral)

Do you think that this would make sense?

Regards, marcelo

> Regards,
> Tony
>

Delivery-date: Sat, 20 Aug 2005 05:34:41 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: about reachability detection draft
Date: Sat, 20 Aug 2005 07:34:41 +0200
To: Iljitsch van Beijnum <[email-masked]>

El 19/08/2005, a las 15:23, Iljitsch van Beijnum escribi=F3:

> On 17-aug-2005, at 16:24, marcelo bagnulo braun wrote:
>
>> Suppose that A is communicating with B
>> Suppose that we are using FBD so that a given frequency of packets=20
>> (data or just signaling) is guaranteed by the shim in each direction
>
>> Suppose now that A stops receiving packets. This implies a failure in=20=

>> the B->A path.
>
> Yes. (Assuming that A is still sending, if not the session could=20
> simply be idle.)
>
>> Now are you assuming that when A stops receiving packets A should try=20=

>> with alternative paths?
>> I am not sure this is good approach, since the path A->B may be=20
>> working properly.. moreover, maybe this path A->B is the only one=20
>> working.
>
> There are three possible actions:
>
> 1. Do a "lightweight" reachability test using the currently used=20
> locator pairs for both directions.
> 2. Start a full reachability evaluation procedure that will look at=20
> all locator pairs until it finds one that works.
> 3. Switch to a new address pair.
>
> I think 3. is the wrong thing to do. Most importantly, because if one=20=

> pair fails it's very likely that at least some other pairs also fail,=20=

> so jumping to a new address pair blindly makes little sense. Also, in=20=

> order to be able to do this, we must have previously authenticated the=20=

> alternative address for the correspondent (or the correspondent for=20
> our alternative address) to avoid redirection attacks. So this=20
> basically means authenticating all possible addresses for both sides=20=

> when the shim state is created. That's a waste of resources.
>
> 1. may make sense, but if we know there should be packets flowing in=20=

> both directions but they're missing in one direction, it's very=20
> unlikely that the current address pair is still working. We should=20
> probably still test it, but I think we should also start looking at=20
> the first secondary address pair because it's likely that the=20
> currently used pair is dead. So that lands us at 2.
>
>> As i see the FBD mechanism would be the following:
>
>> A and B are communicating
>> they are using FBD
>> A stops receiving packets
>
>> So, A informs B that the B->A path has failed (this implies some form=20=

>> of signaling from A to B, which is likely to be required to be=20
>> somehow reliable, which is why i see a potential difficulty here)
>
> The way I see it, A just starts a path exploration procedure. This=20
> means it will try address pairs until it gets an answer from B, so=20
> unless the path from A to B is completely broken at some point B will=20=

> see a path exploration packet from A, which indicates to B that there=20=

> is probably something wrong, so B would start doing its own path=20
> exploration.
>

but this is somehow weird, because A is performing a path exploration=20
procedure, even if the path that A is using to send packet is working=20
properly...

I mean, actually, in this scenario, you are using the path exploration=20=

procedure initiated by A as a reliable mechanism to allow A to inform B=20=

that something is wrong in the path that B is using to send packets, so=20=

that B starts its own path exploration procedure, which is the one that=20=

is the one that in fact is relevant, since is the B->A path which is=20
broken.

So, in fact the FBD procedure has the following steps, as i see it:
1- Both ends send keepalives to preserve the traffic frequency required.
2- when something is wrong, the receiving end detects the failure
3- Then the receiving end convey the information about the failure to=20
the sender (which the one that has to react). Such signaling must be=20
performed in a somehow reliable way (in particular, you can use the=20
path exploration procedure since it tries multiple paths)
4- The the sender can react, performing in its turn a path exploration=20=

procedure.

agree?

i think it is important to identify that step 3 is required in order to=20=

complete the failure detection procedure, since it is not only=20
important to detect the failure but also to inform the party that can=20
actually do something about it (i.e. the sender) that the failure has=20
occurred

If we use the other mechanism described in the draft (i don't remember=20=

how did you call this in your presentation) the sender is the one that=20=

actually detects the failure so there is no need for step 3

(FWIW I am not arguing for this latter mechanism or against FBD, just=20
trying to understand all the pieces that are required to make each one=20=

work)

>> So you think that we need one packet type for periodic keepalives for=20=

>> failure detection and another type of packet for exploring=20
>> alternative paths?
>
> Yes. Assuming we go with the forced bidirectional thing, in the event=20=

> that there is traffic from A to B, but no traffic from B to A, B needs=20=

> to generate a packet anyway in order to let A know that everything is=20=

> still working. Since A presumably monitors all packet types, the=20
> nature of this packet is of no importance. It could even be an IP=20
> header without any payload.
>
> But for the full reachability evaluation we need lots of extra stuff.
>
> So it makes sense to have this in different packet types, especially=20=

> since the simple filler packets will probably be the most common.
>

ok i see now what you meant... (skipped some questions now answered by=20=

your explanation)

...
>>>  And we need a reasonable level of authentication too, because we=20
>>> haven't previously established that these addresses indeed belong to=20=

>>> the correspondent.
>
>> I am not sure what is the level of security we need here yet, but in=20=

>> any case imho it will be very related to the security used during the=20=

>> shim context establishment.
>
>> I mean, imho the critical part of the security if the addition of new=20=

>> locators to the locator set. Once the locators have been validated,=20=

>> using them shouldn't require to tight security requirements imho
>
> Agree, but on the other hand we don't want to give attackers the=20
> capability to make hosts think there are reachability problems,=20
> especially attackers that aren't able to sniff any of the setup=20
> traffic. So something simple like the TCP sequence number or the IPsec=20=

> replay counter would be useful here.
>

the attack that you have in mind then is that an attacker generates a=20
reachability test packet and the host reads this packets as the other=20
end starting the path exploration procedure, which implies that itself=20=

should do the same because of a potential outage, right?
But this would be in the case of FBD right? i mean does this applies in=20=

the other mechanism too? I think not, becuase in this case,=20
reachability tests are a two way exchange and wouldn't imply that the=20
other end starts doing its own reachability test, i think

In any case, you could also include some cookie generated during the=20
session establishment exchange to even make this stronger.

>>> On the other hand, when A1 <-> B1 is working happily, there is no=20
>>> need to use such a complex protocol: we are only testing one pair in=20=

>>> each direction, and the correspondent has been authenticated=20
>>> earlier. If we wanted we could even use pings to determine whether=20=

>>> this still works. (Well, sort of...)
>
>> But as i see it, we are only going to try locators that are included=20=

>> in the locator set available for that shim context, which means that=20=

>> they have already been validated....
>
> Since we need to test them for reachability right before we use them=20=

> anyway, I tink it makes more sense to defer this validation until that=20=

> time.
>
>>>  Deprecation is irrelevant, as we can continue to use deprecated=20
>>> addresses.
>
>> Well, deprecation means that somewhere in the near future, the=20
>> address won't be available anymore, right?
>
> Right, so we keep using it until it's not available anymore.  :-)
>

ok, perhaps i am confused here so please correct me

Suppose that a prefix PrefA is being renumbered.
In order to remove this prefix, all addresses containing PrefA are=20
deprecated. This means that this prefix should not be used for=20
establishing new communications, but can be used for ongoing=20
communications, right?

Now a certain point in time the prefix is actually removed, this means=20=

that packets containing a destination address with this prefix won't=20
reach the site any more. this would be that the prefix and the=20
associated addresses are no longer available according to your=20
terminology right? So my question is how does the host knows that the=20
prefix is no longer available? is there any other signaling tool=20
besides deprecation through RAdv?

>> So i guess it make sense to keep on using it as ULID, but i wonder if=20=

>> it wouldn't be a good strategy to try to rehome the ongoing=20
>> communications to an alternative locator...?
>
> Are there any bad side effects when using deprecated addresses?
>
>>>  A more interesting case is when an address is removed from the=20
>>> system. I don't remember which session it was, but in Paris someone=20=

>>> was talking about how systems keep using addresses they no longer=20
>>> have because upper layers still use those addresses.
>
>> well, in the shim this address should be kept as a possible ULID but=20=

>> not as a valid locator i guess
>
> Yes, it makes sense to keep addresses as ULIDs for a while after=20
> they've become unavailable. On the other hand, in theory the address=20=

> could be assigned to someone else so we problably want to limit this.
>
> On the other hand, if the address is an HBA address, presumably nobody=20=

> else would be able to use it (certainly not as a ULID, only possibly=20=

> as an RFC 3041 semi-collision), so there is no harm in keeping it=20
> around as a ULID.
>
>>>  IMO this is a feature: if I unplug my ethernet from my powerbook=20
>>> and turn on my wifi, I get the same address on a different interface=20=

>>> and my sessions are still alive. Under windows, things like this=20
>>> kill your sessions immediately.
>
>> in this case i guess that this address should be:
>> - kept as a ULID during all the time
>> - should be removed from the locator set during the period which the=20=

>> address was not available (i.e. during the time it took to remove it=20=

>> from the ethernet and it is back in the wifi)
>
> Maybe we should keep using the ULID as long as there is shim state=20
> that refers to it, but it's removed when the shim state that uses the=20=

> ULID is removed.
>

agree

>>> The way I see it there is an ordered list of address pairs. The more=20=

>>> probes fail to make it to the other side the lower the address pair=20=

>>> ends up on the list, I imagine.  :-)
>
>> but what happens when you only have 2 address paris for instance? you=20=

>> may end up trying with the same two address pairs forever... I mean,=20=

>> i guess we need a mechanism to give up trying, right?
>
> Hm, maybe, maybe not... How does this work for current IP?
>
> I guess that while the shim is still trying to move the session to a=20=

> different address pair, we should block ICMP errors and such so the=20
> application won't give up, but when the shim can't find a working=20
> address pair, we should pass on the ICMP errors to the upper layer. At=20=

> the same time, I'm not sure if we should ever really give up as long=20=

> as upper layers keep sending traffic: connectivity can come back at=20
> any time.
>

well, the shim could perhaps generate an internal ICMP error for=20
informing ULPs that no path is available

>>> We need to think about the situation where a fast primary link fails=20=

>>> and we switch to a slow backup, though. Presumably, we'll want to=20
>>> switch back to the fast primary address pair when possible.
>
>> Right, the same arguments may apply for other reasons like cost of=20
>> the link, security/privacy features of the path... But the question=20=

>> is how does the shim is aware of such information? i guess that these=20=

>> are policy issues and we could include some means to express this=20
>> type of considerations in the shim
>
> Right. For destination addresses and to some degree, source addresses,=20=

> this can work through the RFC 3484 policy table, but it gets more=20
> difficult when we have to select an output interface or ISP/site exit,=20=

> possibly though the source address in the packet.
>
>

BEsides, in order to return to the original path, the shim needs to the=20=

probing it all the time, so that it can detect when the original path=20
is back up. I mean, this wouldn't normally occur for other paths, since=20=

the shim won't be probing alternative paths while the current path is=20
working, so if we want this feature, the default behaviour needs to be=20=

modified i guess.

regards, marcelo

Delivery-date: Sat, 20 Aug 2005 05:34:11 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Sat, 20 Aug 2005 07:34:43 +0200
To: Paul Jakma <[email-masked]>

El 19/08/2005, a las 17:11, Paul Jakma escribi=F3:

> On Fri, 19 Aug 2005, marcelo bagnulo braun wrote:
>
>>> Why must host1 detect this? Host2 could also ;).
>>
>> not in a unidirectional connectivity scenario
>>
>> consider the case where the failure implies that:
>> PrefA:Host1 -> Host2 is not working
>> PrefB:Host1 -> Host2 is working
>> Host2 -> PrefA:Host1 is working
>> Host2 -> PrefB:Host1 is not working
>>
>> How would you cope with this case?
>
> How important is this case?
>
> Further, in your scenario, this was due to a local-failure near Host1.=20=

> A failure which can easily be detected locally without any need for=20
> n^2 probing.
>
> What's needed is:
>
> - Host1 to detect the local failure and update the exit path to use
>   (and hence the source to use)
> 	- this is achievable in multiple ways
> 	- none of which need be in shim6
> 	- none of which require shim6 to be aware of SAS or egress
> 	  issues
>
> - Host2 shim6 to detect host1's valid locators have changed
> 	- Maybe because it receives a packet from Host1 with a new
> 	  source
> 	- Maybe because Host2's reachability probes detect PrefB
>
> How common is this failure mode?
>
> You want to specify that shim6 be able to work around /any/ kind of=20
> routing failure, anywhere on any part of the internet affecting any=20
> path between Host1 and Host2.
>
> My gut feelings though are:
>
> - Failures typically are near the edges
> - Failures are typically bi-directional for a given path
> - Uni-directional failures tend to be due to /congestion/, not
>   actual failures - again, typically at the edges. Congestion related
>   "failures" tend to be very transient/sporadic.
> - Failures in the 'middle' are uncommon, and tend to affect /huge/
>   numbers of paths (ie there's a decent chance it will take out /all/
>   your paths)
> - The problem of uni-directional failure on two /unrelated/ paths at
>   the same time is *tiny*
>
> Hence (as a gut feeling):
>
> - n^2 probing in shim6 is simply introducing huge expense in order to
>   solve a very uncommon problem
>
> You think the tradeoff in order to achieve perfection is worth it.
>
> I don't, I think the above is a general quality-of-internet-routing=20
> problem. I think it's something that should and will be tackled within=20=

> the routing area, where people have been and are continuing working on=20=

> optimising routing protocols (from OSPF to BGP) to cope gracefully=20
> with failures and restarts in order to eliminate some common scenarios=20=

> where routing-loops can occur in todays routing protocols.
>
> I don't see a compelling reason to consider problems in internet=20
> routing to be something shim6 needs to introduce great complexity for=20=

> in order to work around, when a simple approach (let underlying OS=20
> routing pick the local prefix) will likely allow 99% of failures to be=20=

> detectable and worked around.
>

ok, i guess we have come to key point here.

We agree that the proposed mechanism proposed for the shim is what is=20
needed to deal with all failure modes and to identify if there is at=20
least one working path right?

We seem to disagree about if the cost that implies is worth it, right?

you seem to consider that there are simpler methods that would deal=20
with a significant amount of the common failure modes, in particular=20
the one you detail above.

I guess that probably RFC3178 already provides a reasonable solution=20
that provides a the protection level that you ask for. I mean RFC3178=20
protects from failures in the edges in a transparent fashion

...

>
> There are many other possible mechanisms.
>
> A host could have the following default route:
>
> default via ISP1-gateway
> 	via ISP2-gateway
>

what if there is a single router in a link of the multihomed site? i=20
mean, you cannot assume that in all links of the multihoemd site there=20=

will be as many routers as ISPs the site is multihomed too, right?

In this point, i guess you end up requiring source address based=20
routing in the multihomed site, in order to allow the end host to force=20=

routing through the selected exit ISP and the shim using the source=20
address to actually select the exit ISP hence the shim selecting the=20
source address, i guess

> ISP1-gateway device X src ISP1-PA-address
> ISP2-gateway device X src ISP2-PA-address
>
> Some external mechanism could update this route as required. Be it=20
> gateway-probing, probing "well known hosts on the internet", a RIP=20
> default announced from border routes, or even an application which=20
> monitors route-lookups or the OS route-cache and probes those and=20
> updates routes accordingly.
>
> The wealth of possibilities if you at least /allow/ the=20
> source-selector mechanism to be external to shim seems a compelling=20
> reason for shim6 to /not/ (by default) get involved.
>
>> So far, we are assuming that the shim is a host based approach and=20
>> that each host performs all the functions of the shim
>
> Yes.
>
>> the case of the proxy that you are mentioning have been considered=20
>> and it is attractive but it presents some difficulties, especially=20
>> w.r.t security... perhaps you could try to consider the security=20
>> implications of that split that you are considering...
>
> There is one case where 'split' or 'proxy' mode shimming would be=20
> possible without security ramifications, I think. The case where the=20=

> ULID's in use are IPv6 addresses, network prefix and host identifier.
>
> Then a simple stateless static mapping on the "shimmers" (which would=20=

> be gateways into/out of the "shimmed" ULID network) will do. Security=20=

> then is simply not a concern, no more than it is for a normal router=20=

> with a static forwarding table.
>

not sure what you mean.. are you thinking in something like GSE here?

>> not sure what you mean here... ULID is upper layer identifier, right?=20=

>> so the ULID belongs to the shim host, so i guess they are in the same=20=

>> machine
>
> Right.
>
> As above, it seems possible to me that shim6 could allow for at least=20=

> one usage that would not require shim and "ULP" to be same machine. It=20=

> could be very useful for small/medium size multihomed sites (ie not=20
> large enough to get a global IPv6 prefix).
>

i agree it would be useful but i still not sure how do you deal with=20
security stuff in this case...

regards, marcelo

> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Don't let your status become too quo!
>
>

Delivery-date: Fri, 19 Aug 2005 23:01:16 +0000
Date: Sat, 20 Aug 2005 08:58:00 +1000
To: Brian E Carpenter <[email-masked]>, Pekka Nikander <[email-masked]>
From: Geoff Huston <[email-masked]>
Subject: Re: shim-aware transports
Cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>

I'd suggest that an API model may assist in the thinking here, particularly 
as I
suspect that there may also be some 'horizontal' signalling that may also
be required here when the SHIM layer wants to signal its remote
counterpart.

However if an implementation chooses to implement the API via
direct access to data structures in the stack then that's not
inconsistent with this API design approach.

   Geoff

At 01:30 AM 20/08/2005, Brian E Carpenter wrote:
>Pekka, we do agree, and I agree with what Dave Meyer wrote too.
>
>Simply put, the shim needs enough state to decide when to switch
>locators on behalf of dumb transports, and it needs a way to make that
>state accessible to smart transports. What I just realised is the
>access for a smart transport could be an API, or it could be
>direct access to the data structure maintained by the shim. We can
>certainly model it as an API, but direct access might be needed
>for performance reasons in an actual implementation.
>
>    Brian
>
>Pekka Nikander wrote:
>>Brian,
>>I have a feeling we have some sort of a communication problem here  but 
>>that we fundamentally agree; this message is an attempt to  clarify what 
>>I am trying to say.
>>I am all for the initial goal of keeping the ULPs unchanged.  That's
>>the whole point.  However, when doing so I also think that we
>>*probably* should adopt the conservative approach adopted by e.g.  SCTP, 
>>i.e., to have one preferred locator and to use the rest as  equal 
>>backups.  Two reasons: keep it simple, and less surprises to  the 
>>transport algorithms.
>>Now, AFAICT, the discussion below cornered around what to do next,
>>after the initial goal has been reached.  What I remember from Paris  we 
>>all seem to agree that we need some kind of an interface between  the 
>>shim layer and ULP.
>>My two messages (URLs below) define a strawman for such an interface,
>>while trying to keep the system at the same time as simple as
>>possible.  I don't see that changing at all the way the shim works  with 
>>unmodified ULPs, or what state it maintains.
>>On the other hand, what I find somewhat worrisome are allusions that  we 
>>should make the state at the shim layer more complex as we add
>>multi-addressing support to ULPs.  It looks to me that some folks  would 
>>like to do locator-policy decisions at multi-addressing  transports, and 
>>then push those policies somehow to the shim, to be  executed there.  I 
>>find that unnecessarily complex.  Why can't the  ULP just pick the 
>>locators, if it knows about them; in that case the  shim just protects 
>>the transport from using locators that are no  longer valid.
>>So, all I am saying is KISS, and so far the method Christian and I  deviced
>>
>>Going a little bit deeper,
>>On Aug 19, 2005, at 12:55, Brian E Carpenter wrote:
>>
>>>Pekka, since the initial goal is no change to the ULPs,
>>>the state is all going to be in the shim, isn't it?
>>>(Or perhaps better to say, in a data structure initially
>>>maintained exclusively by the shim, but perhaps accessed
>>>by shim6-aware ULPs in a later stage.)
>>>
>>>    Brian
>>>
>>>Pekka Nikander wrote:
>>>
>>>>I am just wondering how much state we want to push to the shim
>>>>layer  versus keep at the     ULPs.  See my other notes about that:
>>>>http://psg.com/lists/shim6/msg00663.html
>>>>http://psg.com/lists/shim6/msg00671.html
>>>>It looks to me that we may be able to keep the signalling  interface
>>>>pretty simple, without any loss of generality.  But  maybe I am
>>>>missing something there.
>>>>--Pekka Nikander
>>>>On Aug 19, 2005, at 1:53, Geoff Huston wrote:
>>>>
>>>>>Brian noted that  making the shim  work with unmodified IP and
>>>>>unmodified upper layers needs to get shipped first, and thats a
>>>>>good priority.
>>>>>
>>>>>Brian also noted that this implies a two-way API between the  shim
>>>>>and the ULP, in which the necessary events are transmitted  up or down.
>>>>>
>>>>>So I'd like to float the concept of a placeholder about the
>>>>>interaction with transport that hopefully will allow the  priority
>>>>>work to proceed while still keeping enough flexibility  around to
>>>>>allow transport sessions to be able to be more  explicit about what
>>>>>they want from the shim layer, but to define  this interaction at a
>>>>>later stage in the WG's life.
>>>>>
>>>>>It seems to me that the basic mechanics of this can be satisfied
>>>>>by  defining a rich enough signalling interface between the shim
>>>>>layer  and the upper level sessions - perhaps allowing for the
>>>>>inclusion  of the relevant locator set, and allowing attributes  to be 
>>>>>placed  on individual locators as well as on the complete  set in 
>>>>>either  direction. In other words this allows an upper  level to be 
>>>>>more  specific about locator preference either by  attribute or by
>>>>>explicit choice by passing a rich information  bundle to the ULP 
>>>>>via  this signalling interface.
>>>>>
>>>>>Placing the potential functionality in the hands of the session
>>>>>layer also infers that the shim state may not necessarily be
>>>>>endpoint-to-endpoint, but may operate at a finer level of
>>>>>selection  (the 'equivalence" topic I referred to at the WG
>>>>>meeting). It also  infers that an individual session may want to
>>>>>trigger a locator  change that appears to be an isolated request  - in 
>>>>>which case there  appears to be a logical need for dynamic  shim state 
>>>>>forking in some  form (i.e. a shim state may state in  a general 
>>>>>host-to-host state,  but individual sessions may 'fork'  out of this 
>>>>>general state by  requesting individual locator  change trigger 
>>>>>settings, for example.)
>>>>>
>>>>>If you allow vertical (and peer-to-err horizontal) locator
>>>>>attribute signalling, and also dynamic shim state forking you  can
>>>>>then say  that the mechanisms for the ULP to signal quite  explicit
>>>>>hints to the shim layer as to when and what locator to  use, and 
>>>>>for  the shim to respond with a forked state as required  and signal 
>>>>>this  to the peer remote shim entity is within the  capability of 
>>>>>the  protocol. How the fields are defined and used  are up to a 
>>>>>later  incarnation of the or another WG, that will  define the 
>>>>>attributes  and  their semantics.
>>>>>
>>>>>Does this help?
>>>>>
>>>>>   Geoff
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>

Delivery-date: Fri, 19 Aug 2005 23:00:46 +0000
Date: Sat, 20 Aug 2005 08:59:27 +1000
To: [email-masked], shim6 <[email-masked]>
From: Geoff Huston <[email-masked]>
Subject: Re: shim-aware transports

At 02:13 AM 20/08/2005, Spencer Dawkins wrote:
>Iljitsch van Beijnum wrote:
>
>>On 17-aug-2005, at 17:31, Spencer Dawkins wrote:
>>
>>>when we were discussing TCP in TRIGTRAN, we pretty quickly  converged on 
>>>NOT notifying transports when bandwidth available  changed, because TCPs 
>>>have to respond to losses, and adding  additional hints just adds stack 
>>>complexity beyond the complexity  that is unavoidable.
>>
>>Well, I'm sorry to have to say it, but that's just STUPID.
>
>... FWIW, I never said I thought it was the right answer :-)
>
>But my point in writing was mostly to say that I encourage postponing 
>"transport-aware" work until we make some progress on "transport-unaware" 
>SHIM6, and then making sure that, wherever the work gets done, we are 
>working closely with the transport community to meet their needs with 
>protocol mechanisms they will actually make use of.

And that approach and relative priorities within the charter is, I trust, 
pretty much where we are today.

regards,

   Geoff

Delivery-date: Fri, 19 Aug 2005 21:54:53 +0000
Date: Fri, 19 Aug 2005 22:53:40 +0100 (IST)
From: Paul Jakma <[email-masked]>
Reply-To: [email-masked], [email-masked]
To: Iljitsch van Beijnum <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Fri, 19 Aug 2005, Iljitsch van Beijnum wrote:

> RAs have very long lifetimes. I think the Cisco default is a week. 
> You can't bring this down to a minute or less without all kinds of 
> interesting side effects.

You have 2 lifetimes, preferred and valid. Valid can't go under 2 
hours. Preferred can be as short as you like, down to 3s, and 
prefered is the one we're interested in.

I have preferred set to 10s here.

What are the side effects?

> I agree that when certain information is available, it makes sense 
> to distribute it locally rather than have every host go out and 
> discover the same facts for itself. We'll have to come back to this 
> at some point.

Progress :).

>> - fix the problems in internet routing
>
> We await your suggestions...

Ha!

However, in a working group not so far away, there are people looking 
at these things.

> The trouble is that you need aggregation to make routing scale, and 
> with aggregation you lose all this interesting info that would have 
> been useful. Routing can still tell you some interesting things 
> when there are wide-spread catastrophes, but I'm not sure it's 
> worth the trouble to optimize for that. (Or rather: I'm pretty sure 
> it isn't.)

That's great, but we're discussing a host protocol (and maybe even a 
leaf-site border protocol). Hosts should behave like hosts and not 
try probe every possible path by default. Imagine:

   Host2(shimmed)
   |  \
   |   \
  ISP3  ISP4
   |  \   |
   |   \_tier-1
tier-2   |  \
     \ \__|_  |
      \   | \ |
       ISP1  ISP2
         \   /
          \ /
         Host1 (shimmed)

Host1 is communicating with Host2 using Host2's tier-2 ISP locator.

Tier-2 has a failure affecting the shim 'flow' Host is using. The 
tier-1's POP gets 2*X probe packets - for no good reason.

Further, within a minute, maybe less, of tier-2 failing, both ISP1 
and ISP2 have switched over their routes to ISP3 to go via tier-1.

Now imagine ISP1 and ISP2 have thousands upon thousands of shimmed 
customers. Now imagine these probes at an internet wide scale. Is it 
worth introducing all that n^2 probing noise into the internet when 
core internet routing likely will fix the problem anyway, maybe 
within a minute, maybe faster?

Probe for availability of the /remote/ locators sure. But don't 
combine it with every possible combination of local address please - 
it's not needed.

>> wow 2*4^2, ie 32 packets to complete probing (worst case). Imagine 50 such 
>> shim6 hosts on your network.
>
> Well you really want to send at least 3 probes to account for random packet 
> loss.  :-)

Ouch.

>> But it's easy. Shim6 is *not* TCP, it doesnt need to maintain any 
>> /specific/ consistency of addresses. Eg, in this example, why on 
>> earth is B replying with (B1,A1)? The reply from B (in my mind) 
>> would be (B3,A1).
>
> The reason why it's not easy is that at this point, the shim hasn't been 
> activated yet, we're just doing regular TCP.

I don't quite understand this. This is intended as an optimisation 
for the case where the ULID's are 'routeable' directly between the 2 
hosts right?

> This is necessary to maintain backward compatibility.

I'm not sure I fully understand this point.

I suspect this (having a ULID be a locator) can be done too. So you 
you may only have to shim if addresses change. But once shimmed, the 
/shim/ mapping can be of *sets* of addresses to the other set of 
addresses, not of specific tuples. Ie, the mapping should be:

( {A1,A2,A3,A4} , {B1,B2,B3,B4} ) -> .....

Then you can drop/add things out of the sets as required.

If the mapping maps A1 to A1, fair enough. (This is the "null 
transform" case in Geoff's architecture case right?)

> And even if you activate the shim at this point, the two sides 
> haven't been able to compare notes yet, so you can't start doing 
> strange tricks yet, or at least you run into security 
> complications.

Ah, I'm not familiar with these, would you be able to explain or 
refer me to something?

> Ah, good that you said so because we all thought you were 
> supporting this.

ROFL :). Just thought I'd make it clear ;).

> Don't forget that if the site exit router does its own version of 
> the ingress filtering, it can send back ICMP messages so the host 
> knows that this source address doesn't work and move on without 
> much delay.

Yep.

> So after a maximum of 3 messages with incorrect source addresses A 
> knows it should use A4, and then it only has to do B1, B2, B3 and 
> B4 to find the working A4-B4 pair.

Yep.

> Also, if the host has several sessions towards different 
> destinations, it may observe that if 2001:a900:456::1 isn't 
> working, so if it has to choose between trying 2001:a900:789::1 and 
> 3ffe:ffff:789::1 it will choose the latter because there is a 
> chance the whole 2001:a900::/32 block is affected.

That seems a possible strategy, yes.

> So in reality having to test 2*n^2 will be extremely unlikely.

I would agree, but more because imho the internet is reasonably 
reliable, so usually only a few will fail.

2*n^2 is for both sides btw - sent and received. n^2 (or local*remote 
locators, tending to n^2 for worst case) is probes sent. And the 2 is 
insignificant anyway compared to the square ;).

> The trouble is that many small ISPs around here connect to the rest 
> of the world through one location in Amsterdam, so when there is a 
> power failure at that location, not only their AMS-IX stuff goes 
> down but also their transit. Last time there was an AMS-IX power 
> failure (a month before the generator they were installing because 
> of the one-but-last power failure went online) about 25% of all 
> AMS-IX members were completely unreachable for me.

Yes, that seems to be a problem in BeNeLux - huge overdependence on 
AMS-IX. When you're that close to AMS-IX and everyone is there, 
there's not much 'push' to setup physically seperate transit and 
peerings. That though is an internet architecture issue, particularly 
for NL and BeNeLux.

The internet ecosystem will evolve further to cope with these things 
(outside of shim), eg my point about VoIP being a driver in the 
routing area (see mail to marcelo). Routing will get better and 
better.

> BTW, my ISP just had a very big DoS attack. The shim would have 
> enabled me to keep working to the extent possible, routing can't 
> really do anything in these cases.

Yep, I really want shim too. ;) I want it to allow for sanity though. 
;) (while still allowing for implementations to try full probing if 
they really must).

Do you want to add to the DoS of your ISP by having lots of shimmed 
hosts then go and DoS the *other* ISP? :)

Note, that for something like a DoS, where you're getting just poor 
service rather than no service, it might be easier to just update 
your routes or your local IPv6 RA prefixes to not use that ISP. 
That's an argument again for making shim6 interact with normal OS 
routing+SAS layer like any other application.

For extra-browny points, write a small daemon to monitor RTT and loss 
rates to each of your ISPs and adjust local routes/prefixes/whatever 
to suit to your tastes.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Pick another fortune cookie.

Delivery-date: Fri, 19 Aug 2005 20:36:38 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: failure detection
Date: Fri, 19 Aug 2005 22:36:04 +0200
To: Paul Jakma <[email-masked]>

On 19-aug-2005, at 20:57, Paul Jakma wrote:

> - Use IPv6 RAs to advertise both prefixes, with a preferred lifetime
>   set according to how quickly you want to switch, eg set it equal
>   to twice the RA interval, or even equal to it.

RAs have very long lifetimes. I think the Cisco default is a week.  
You can't bring this down to a minute or less without all kinds of  
interesting side effects.

> Anyway: Then with *one* message (or lack of, to be more specific),  
> from your DSL router all X hosts on your network deprecate use of  
> the ISP-B addressm and can start using others. Far more efficient  
> than *all* your X hosts doing n^2 probes.

I agree that when certain information is available, it makes sense to  
distribute it locally rather than have every host go out and discover  
the same facts for itself. We'll have to come back to this at some  
point.

>> Yes, I do. As a BGP jockey, I'm kind of like the health inspector  
>> who never eats out... There is a lot going on that regular users  
>> don't really know about.

> So the answer is:

> - have users run around every restaurant to try divine which one
>   serves edible food (n^2 probing)

Well, that's better than coming back to the one that serves crappy  
food each time because you don't want to visit so many restaurants.  :-)

> rather than:

> - fix the problems in internet routing

We await your suggestions...

The trouble is that you need aggregation to make routing scale, and  
with aggregation you lose all this interesting info that would have  
been useful. Routing can still tell you some interesting things when  
there are wide-spread catastrophes, but I'm not sure it's worth the  
trouble to optimize for that. (Or rather: I'm pretty sure it isn't.)

[test setup]

> wow 2*4^2, ie 32 packets to complete probing (worst case). Imagine  
> 50 such shim6 hosts on your network.

Well you really want to send at least 3 probes to account for random  
packet loss.  :-)

>> A initiates a TCP session with destination address B1. Let's  
>> assume that the system chooses interface 1 for output and A1 as a  
>> source address, so the packets have address pair A1-B1

>> Now it's entirely possible that B's default route is over ISP Q.  
>> So when B sends a reply to the A1-B1 session setup request, it  
>> sends a B1-A1 packet out on interface 2. Now either the site exit  
>> router will filter it, or it will end up at ISP Q or ISP R, which  
>> will filter it. This is the infamous ingress filtering problem  
>> that we have to figure out.

> But it's easy. Shim6 is *not* TCP, it doesnt need to maintain any / 
> specific/ consistency of addresses. Eg, in this example, why on  
> earth is B replying with (B1,A1)? The reply from B (in my mind)  
> would be (B3,A1).

The reason why it's not easy is that at this point, the shim hasn't  
been activated yet, we're just doing regular TCP. This is necessary  
to maintain backward compatibility.

And even if you activate the shim at this point, the two sides  
haven't been able to compare notes yet, so you can't start doing  
strange tricks yet, or at least you run into security complications.

> BTW: Note that I strongly disagree with trying to solve all the  
> internet's routing problems by making every end-host do n^2 probing.

Ah, good that you said so because we all thought you were supporting  
this.

>> So A tries:

>> A1-B2
>> A2-B1
>> A1-B3
>> A3-B1
>> A1-B4
>> A4-B1

>> and on and on and on, until it eventually determines that A4-B4  
>> works.

>> You don't want this to happen. So what's the alternative? Give up  
>> after the second try? The fourth? The n^2/2th?

> IMHO, you should only try:

> (unspecified) -> B1
> (unspecified) -> B3
> (unspecified) -> B4

Ah, ok.

Don't forget that if the site exit router does its own version of the  
ingress filtering, it can send back ICMP messages so the host knows  
that this source address doesn't work and move on without much delay.  
So after a maximum of 3 messages with incorrect source addresses A  
knows it should use A4, and then it only has to do B1, B2, B3 and B4  
to find the working A4-B4 pair.

Also, if the host has several sessions towards different  
destinations, it may observe that if 2001:a900:456::1 isn't working,  
so if it has to choose between trying 2001:a900:789::1 and 3ffe:ffff: 
789::1 it will choose the latter because there is a chance the whole  
2001:a900::/32 block is affected.

So in reality having to test 2*n^2 will be extremely unlikely.

> In my universe (which happens to correspond vaguely to how the  
> internet works today ;) ), when AMS-IX dies, within 1 to 3 minutes  
> or so, your ISP, in conjunction with other ISPs start propogating  
> WITHDRAWs and UPDATEs and converge so that packets flow again.

The trouble is that many small ISPs around here connect to the rest  
of the world through one location in Amsterdam, so when there is a  
power failure at that location, not only their AMS-IX stuff goes down  
but also their transit. Last time there was an AMS-IX power failure  
(a month before the generator they were installing because of the one- 
but-last power failure went online) about 25% of all AMS-IX members  
were completely unreachable for me.

>> Remember that while all of this is going on, the transport  
>> protocol sees a black hole. So at any time, the transport can  
>> decide to time out. The shim doesn't do anything that actually  
>> _hurts_ regular transport protocols.

> If you call the potential for a smallish network sending out near  
> to 1k probe packets "doesn't hurt" for not much gain, sure.

Well, I send out that many packets in 10 seconds over my DSL line  
when everything works, so why not do the same when there is a  
problem?  :-)

BTW, my ISP just had a very big DoS attack. The shim would have  
enabled me to keep working to the extent possible, routing can't  
really do anything in these cases.

Delivery-date: Fri, 19 Aug 2005 19:43:25 +0000
Date: Sat, 20 Aug 2005 05:36:11 +1000
To: Erik Nordmark <[email-masked]>
From: Geoff Huston <[email-masked]>
Subject: Re: shim-aware transports
Cc: Brian E Carpenter <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>

At 11:05 PM 19/08/2005, Erik Nordmark wrote:
>Geoff Huston wrote:
>
>>Placing the potential functionality in the hands of the session layer 
>>also infers that the shim state may not necessarily be 
>>endpoint-to-endpoint, but may operate at a finer level of selection (the 
>>'equivalence" topic I referred to at the WG meeting). It also infers that 
>>an individual session may want to trigger a locator change that appears 
>>to be an isolated request - in which case there appears to be a logical 
>>need for dynamic shim state forking in some form (i.e. a shim state may 
>>state in a general host-to-host state, but individual sessions may 'fork' 
>>out of this general state by requesting individual locator change trigger 
>>settings, for example.)
>
>Geoff,
>
>This notion of forking makes sense, but it isn't clear to me whether it is 
>a local notion at the sending side, with the ULP at each end being able to 
>make independent decisions (perhaps using some ULP signaling), or whether 
>it is a shim-coordinated decision.

I am not sure whether peer-to-peer coordination is a strict requirement or 
not, hence
my comment about a richer vertical and horizontal signalling API that at 
least leaves
a signalling placeholder there in any case.

>I think we can define it to be a local thing at the sending side, where a 
>ULP can signal that a given class of packets between a ULP pair (or even 
>at the granularity of each packet) use a different locator pair than the 
>default. The receiver will happily accept such packets, because the 
>locator pair is in the negotiated set of locators for the ULP pair context.

Exactly my thought as well! The other end may see the use of multiple 
source locators
in the incoming stream and will correctly map these to the same ULID 
and  then demux
them to the active transport entities as normal.

>Do you think we'd loose something important if there isn't shim signaling 
>to have the two ends agree to switch the locator pair for both directions?

I think we should allow for the possibility for one shim entity to signal 
to the remote shim
entity to attempt a switch to a specific locator set for a specific 
"state", forking its state
as necessary, but its a weakly formed thought at this stage!

regards,

     Geoff

Delivery-date: Fri, 19 Aug 2005 19:27:38 +0000
Date: Sat, 20 Aug 2005 05:25:27 +1000
To: Pekka Nikander <[email-masked]>
From: Geoff Huston <[email-masked]>
Subject: Re: shim-aware transports
Cc: Brian E Carpenter <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>

Hi Pekka,

I was attempting to suggest that a potentially richer vertical and
horizontal interface would allow us to press on with the more direct
case of Shim6 operating at a straightforward IP level, and allow
for subsequent ULP direction to be defined.  This is indeed a signalling
specification suggestion that was intended to keep most of the session
level information within the transport level.

regards,

   Geoff

At 06:35 PM 19/08/2005, Pekka Nikander wrote:
>I am just wondering how much state we want to push to the shim layer
>versus keep at the     ULPs.  See my other notes about that:
>
>http://psg.com/lists/shim6/msg00663.html
>http://psg.com/lists/shim6/msg00671.html
>
>It looks to me that we may be able to keep the signalling interface
>pretty simple, without any loss of generality.  But maybe I am
>missing something there.
>
>--Pekka Nikander
>
>On Aug 19, 2005, at 1:53, Geoff Huston wrote:
>
>>Brian noted that  making the shim  work with unmodified IP and
>>unmodified upper layers needs to get shipped first, and thats a
>>good priority.
>>
>>Brian also noted that this implies a two-way API between the shim
>>and the ULP, in which the necessary events are transmitted up or down.
>>
>>So I'd like to float the concept of a placeholder about the
>>interaction with transport that hopefully will allow the priority
>>work to proceed while still keeping enough flexibility around to
>>allow transport sessions to be able to be more explicit about what
>>they want from the shim layer, but to define this interaction at a
>>later stage in the WG's life.
>>
>>It seems to me that the basic mechanics of this can be satisfied by
>>defining a rich enough signalling interface between the shim layer
>>and the upper level sessions - perhaps allowing for the inclusion
>>of the relevant locator set, and allowing attributes to be placed
>>on individual locators as well as on the complete set in either
>>direction. In other words this allows an upper level to be more
>>specific about locator preference either by attribute or by
>>explicit choice by passing a rich information bundle to the ULP via
>>this signalling interface.
>>
>>Placing the potential functionality in the hands of the session
>>layer also infers that the shim state may not necessarily be
>>endpoint-to-endpoint, but may operate at a finer level of selection
>>(the 'equivalence" topic I referred to at the WG meeting). It also
>>infers that an individual session may want to trigger a locator
>>change that appears to be an isolated request - in which case there
>>appears to be a logical need for dynamic shim state forking in some
>>form (i.e. a shim state may state in a general host-to-host state,
>>but individual sessions may 'fork' out of this general state by
>>requesting individual locator change trigger settings, for example.)
>>
>>If you allow vertical (and peer-to-err horizontal) locator
>>attribute signalling, and also dynamic shim state forking you can
>>then say  that the mechanisms for the ULP to signal quite explicit
>>hints to the shim layer as to when and what locator to use, and for
>>the shim to respond with a forked state as required and signal this
>>to the peer remote shim entity is within the capability of the
>>protocol. How the fields are defined and used are up to a later
>>incarnation of the or another WG, that will define the attributes
>>and  their semantics.
>>
>>Does this help?
>>
>>   Geoff
>>
>>
>>
>>
>
>

Delivery-date: Fri, 19 Aug 2005 18:58:41 +0000
Date: Fri, 19 Aug 2005 19:57:38 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Fri, 19 Aug 2005, Iljitsch van Beijnum wrote:

> The 2*n^2 probing isn't necessary to detect failures (local or 
> otherwise), but to detect what's still working.

I don't agree.

> And even if side A can easily detect a failure at site A (which 
> isn't a given, if my DSL line goes down my router knows it but my 
> hosts don't),

We already have protocols for this, there is at least one 
possibility, IPv6 RAs, so:

- Use IPv6 RAs to advertise both prefixes, with a preferred lifetime
   set according to how quickly you want to switch, eg set it equal
   to twice the RA interval, or even equal to it.

   When the DSL router detects a link has gone down, it simply stops
   advertising the relevant prefix.

- We may need to invent protocols to carry valid-source prefix
   information across multiple routers in a site. (to update
   prefix-advertisement information). I think that's a general IPv6 RA
   problem though.

- Other stateful configuration protocols may be used, eg DHCPv6
   the server would need to become multi-address aware, an
   implementation detail.

NB: The latter two points would not be required if we allow shim6 to 
work in a 'split' mode. For then the "shimmed" network would only use 
ULIDs which would always remain valid (no need to change).

Anyway: Then with *one* message (or lack of, to be more specific), 
from your DSL router all X hosts on your network deprecate use of the 
ISP-B addressm and can start using others. Far more efficient than 
*all* your X hosts doing n^2 probes.

Eg, imagine a site multihomed with 3 ISPs, and 50 hosts, which are 
communicating with a similar network in site-B using shim6 (50 hosts 
with 3 locators each). That's possibly 50*3^2 packets to send - 450 
packets. Even though all that's needed is for the DSL router to 
broadcast locally "seems I lost my connection to ISP-A" - one packet 
(or even, lack of one packet).

How can you consider this probing to be at all sane? ;)

> how does side B learn this fact?

It starts receiving packets from Site-A with a new locator address.

> A host doesn't necessarily know which exit path a router will choose.

That's true. We can fix that in a better way than n^2 probing surely?

> So what happens when through means outside our view a packet gets a 
> destination address routed over ISP X, but a source address from 
> address space from ISP Y, and X filters Y's addresses?

See above.

>> - Host2 shim6 to detect host1's valid locators have changed
>>    - Maybe because it receives a packet from Host1 with a new
>>      source
>
> This doesn't allow for unidirectional reachability.

How so?
Host2 need not use the source address in its packets which Host1 is 
using.

>> You want to specify that shim6 be able to work around /any/ kind of routing 
>> failure, anywhere on any part of the internet affecting any path between 
>> Host1 and Host2.

> Yes, I do. As a BGP jockey, I'm kind of like the health inspector 
> who never eats out... There is a lot going on that regular users 
> don't really know about.

So the answer is:

- have users run around every restaurant to try divine which one
   serves edible food (n^2 probing)

rather than:

- fix the problems in internet routing

> Maybe those are a bit more common, but it's not like failures in 
> the core never happen.

To be honest, I don't know, it's my gut feeling. But I'm not the one 
arguing on a gut feeling to have the IETF specify a host protocol 
that mandates n^2 probing.

If we don't know enough about internet failures to say whether it is 
or is not worth building n^2 probing per-default into shim6, then we 
shouldn't do it. It can always be 'added in' later, or done by 
implementations if left out, but if you put it in and it gets 
deployed - it's hard to take back.

> Ingress filtering has the potential to create lots of 
> unidirectional reachability for a given address combination.

I don't buy that at all.

If your ISP-A filters out ISP-B sourced packets, then it wont be 
routing ISP-B destined packets to you.

> Nonsense. Congestion is rare these days, and the levels necessary 
> to break connectivity wholesale are almost unheard of.

No it's not. Go live at the edges. I experience congestion 
*regularly* on my DSL link. I did say "at the edges".

And "break connectivity wholesale .. unheard of" is precisely part of 
my point - TCP doesn't break. But what will shim do exactly in the 
face of lossage?

>> - n^2 probing in shim6 is simply introducing huge expense in order 
>> to
>>  solve a very uncommon problem
>
> Yeah I don't get this point you're arguing so energetically.

:)

Exponential scaling scares me - particularly when built-in to a 
protocol. As does the SAS stuff, but you want that because of the n^2 
probing issue i think.

> Let's build a test network. (I'll be talking about hosts, but 
> obviously many aspects are side-wide.)
>
> Host A has two interfaces, that both eventually connect to a router that 
> connects to two ISPs. So:
>
> Addr A1: int 1 - ISP K
> Addr A2: int 1 - ISP L
> Addr A3: int 2 - ISP M
> Addr A4: int 2 - ISP N
>
> Sanme thing for its correspondent host B:
>
> Addr B1: int 1 - ISP O
> Addr B2: int 1 - ISP P
> Addr B3: int 2 - ISP Q
> Addr B4: int 2 - ISP R

wow 2*4^2, ie 32 packets to complete probing (worst case). Imagine 50 
such shim6 hosts on your network.

> Let's assume that each router will do source address based routing for the 
> two ISPs it connects to, but the ISPs all do ingress filtering.

Ok.

> A initiates a TCP session with destination address B1. Let's assume 
> that the system chooses interface 1 for output and A1 as a source 
> address, so the packets have address pair A1-B1
>
> Now it's entirely possible that B's default route is over ISP Q. So 
> when B sends a reply to the A1-B1 session setup request, it sends a 
> B1-A1 packet out on interface 2. Now either the site exit router 
> will filter it, or it will end up at ISP Q or ISP R, which will 
> filter it. This is the infamous ingress filtering problem that we 
> have to figure out.

But it's easy. Shim6 is *not* TCP, it doesnt need to maintain 
any /specific/ consistency of addresses. Eg, in this example, why on 
earth is B replying with (B1,A1)? The reply from B (in my mind) would 
be (B3,A1).

No filtering problem at all. When this packet arrives at A, it 
associates (B3,A1) with the right mapping. Simple to do, given its 
own locator is in there. Why should A care whether replies with the 
same IP address?

You're thinking, it seems, as if shim6 is like TCP, where each 
different tuples of (source,destination) *must* refer to different 
connections. But there is no need for such a restriction.

> But let's assume we somehow fix this problem, and packets flow 
> without trouble between A1 and B1.

Yes we can easily fix it. The flows will look like:

A -> B using (A1,B1)
A <- B using (B3,A1) (note, B need not even use A1, it could use A2)

> The TCP session continues for a bit, and at some poin the shim 
> wakes up and decides that this is a long-term session that should 
> be protected from failures. So the shim layer on host A sends out a 
> packet with source A1 and destination B1 (= addresses from the TCP 
> session) which includes security stuff and the list of local 
> alternative locators: A2, A3 and A4. B also happens to implement 
> the shim, so it answers with some security stuff of its own and its 
> list of alternative locators: B2, B3 and B4.

Ok.

> So now we're ready for the internet to fail.
>
> Scenario 1: A's link to ISP K fails.
>
> Since this is something A's router can detect, presumably any 
> packets from A1 to B2 will get back an ICMP message, and after a 
> few RTTs TCP becomes really unhappy. The shim may also observe that 
> there are packets going from A1 to B1, but there is nothing coming 
> in from B1 to A1. Maybe the shim decides to fire off a probe from 
> A1 to B1 for good measure. But eventually, it's clear that A1 to B1 
> doesn't work anymore.

Ok.

> Now suppose that the reachability detection subsystem at A decides 
> to see if B2 works. If A sticks to source address A1, then the 
> packet will also incur an ICMP and not make it. So either A sees 
> the ICMP and selects a different source address, or it decides that 
> A1-B2 doesn't seem to work either and goes on to the next address 
> pair. For instance A could try A2-B1. And this one works!
>
> So from now on any outgoing packets with addresses A1-B1 in them are 
> rewritten into A2-B1 and sent on their way.
>
> Any complaints so far?

Well, I wouldn't have /shim/ in A picking A2, but I wouldn't preclude 
it either. So no complaints.

> Scenario 2: big failure, and everything is wiped out except A4-B4. (From 
> where I sit 99% of all traffic flows through Amsterdam, and most of that 99% 
> over the AMS-IX. A nice big power failure there really hurts my 
> connectivity.)

Ha, an AMS-IX outage would hurt me too ;). There are intra-Ireland 
paths that actually go via european IXes, typically LINX but I've 
seen AMS-IX paths too. A lot of european traffic goes via IXes.

BTW: Note that I strongly disagree with trying to solve all the 
internet's routing problems by making every end-host do n^2 probing.

> So A tries:
>
> A1-B2
> A2-B1
> A1-B3
> A3-B1
> A1-B4
> A4-B1
>
> and on and on and on, until it eventually determines that A4-B4 works.
>
> You don't want this to happen. So what's the alternative? Give up after the 
> second try? The fourth? The n^2/2th?

IMHO, you should only try:

(unspecified) -> B1
(unspecified) -> B3
(unspecified) -> B4

In my universe (which happens to correspond vaguely to how the 
internet works today ;) ), when AMS-IX dies, within 1 to 3 minutes or 
so, your ISP, in conjunction with other ISPs start propogating 
WITHDRAWs and UPDATEs and converge so that packets flow again.

I much prefer that ISPs take care of the business of getting packets 
from A to B than specify an n^2 end-host probing protocol within the 
IETF.

What next? What if there are multiple failures between A4-B4, such 
that somewhere between A4 and B4 there is one path which works and 
one which does not. One which we could work around by doing 
source-specified hop-by-hop routing?

Let's have shim6 probe all the intermediary paths too. And let's 
abolish the IETF routing area while we're at it :). (kidding, but you 
see the stretched point I'm making I hope).

> Remember that while all of this is going on, the transport protocol 
> sees a black hole. So at any time, the transport can decide to time 
> out. The shim doesn't do anything that actually _hurts_ regular 
> transport protocols.

If you call the potential for a smallish network sending out near to 
1k probe packets "doesn't hurt" for not much gain, sure.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Man usually avoids attributing cleverness to somebody else -- unless it
is an enemy.
 		-- Albert Einstein

Delivery-date: Fri, 19 Aug 2005 16:43:07 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: failure detection
Date: Fri, 19 Aug 2005 18:38:38 +0200
To: Paul Jakma <[email-masked]>

On 19-aug-2005, at 17:11, Paul Jakma wrote:

> A failure which can easily be detected locally without any need for  
> n^2 probing.

The 2*n^2 probing isn't necessary to detect failures (local or  
otherwise), but to detect what's still working. And even if side A  
can easily detect a failure at site A (which isn't a given, if my DSL  
line goes down my router knows it but my hosts don't), how does side  
B learn this fact?

> What's needed is:

> - Host1 to detect the local failure and update the exit path to use
>   (and hence the source to use)

A host doesn't necessarily know which exit path a router will choose.

>     - this is achievable in multiple ways
>     - none of which need be in shim6

The shim needs to do the address rewriting to make it work, so I  
guess it can be argued that the shim doesn't have to be the one to  
determine this information, but it does need to know it.

>     - none of which require shim6 to be aware of SAS or egress
>       issues

So what happens when through means outside our view a packet gets a  
destination address routed over ISP X, but a source address from  
address space from ISP Y, and X filters Y's addresses?

> - Host2 shim6 to detect host1's valid locators have changed
>     - Maybe because it receives a packet from Host1 with a new
>       source

This doesn't allow for unidirectional reachability.

> You want to specify that shim6 be able to work around /any/ kind of  
> routing failure, anywhere on any part of the internet affecting any  
> path between Host1 and Host2.

Yes, I do. As a BGP jockey, I'm kind of like the health inspector who  
never eats out... There is a lot going on that regular users don't  
really know about.

> My gut feelings though are:

> - Failures typically are near the edges

Maybe those are a bit more common, but it's not like failures in the  
core never happen.

> - Failures are typically bi-directional for a given path

Ingress filtering has the potential to create lots of unidirectional  
reachability for a given address combination.

> - Uni-directional failures tend to be due to /congestion/, not

Nonsense. Congestion is rare these days, and the levels necessary to  
break connectivity wholesale are almost unheard of.

> - Failures in the 'middle' are uncommon, and tend to affect /huge/
>   numbers of paths (ie there's a decent chance it will take out /all/
>   your paths)

So?

> Hence (as a gut feeling):

> - n^2 probing in shim6 is simply introducing huge expense in order to
>   solve a very uncommon problem

Yeah I don't get this point you're arguing so energetically. Let's  
build a test network. (I'll be talking about hosts, but obviously  
many aspects are side-wide.)

Host A has two interfaces, that both eventually connect to a router  
that connects to two ISPs. So:

Addr A1: int 1 - ISP K
Addr A2: int 1 - ISP L
Addr A3: int 2 - ISP M
Addr A4: int 2 - ISP N

Sanme thing for its correspondent host B:

Addr B1: int 1 - ISP O
Addr B2: int 1 - ISP P
Addr B3: int 2 - ISP Q
Addr B4: int 2 - ISP R

Let's assume that each router will do source address based routing  
for the two ISPs it connects to, but the ISPs all do ingress filtering.

A initiates a TCP session with destination address B1. Let's assume  
that the system chooses interface 1 for output and A1 as a source  
address, so the packets have address pair A1-B1

Now it's entirely possible that B's default route is over ISP Q. So  
when B sends a reply to the A1-B1 session setup request, it sends a  
B1-A1 packet out on interface 2. Now either the site exit router will  
filter it, or it will end up at ISP Q or ISP R, which will filter it.  
This is the infamous ingress filtering problem that we have to figure  
out.

But let's assume we somehow fix this problem, and packets flow  
without trouble between A1 and B1. The TCP session continues for a  
bit, and at some poin the shim wakes up and decides that this is a  
long-term session that should be protected from failures. So the shim  
layer on host A sends out a packet with source A1 and destination B1  
(= addresses from the TCP session) which includes security stuff and  
the list of local alternative locators: A2, A3 and A4. B also happens  
to implement the shim, so it answers with some security stuff of its  
own and its list of alternative locators: B2, B3 and B4.

So now we're ready for the internet to fail.

Scenario 1: A's link to ISP K fails.

Since this is something A's router can detect, presumably any packets  
from A1 to B2 will get back an ICMP message, and after a few RTTs TCP  
becomes really unhappy. The shim may also observe that there are  
packets going from A1 to B1, but there is nothing coming in from B1  
to A1. Maybe the shim decides to fire off a probe from A1 to B1 for  
good measure. But eventually, it's clear that A1 to B1 doesn't work  
anymore.

Now suppose that the reachability detection subsystem at A decides to  
see if B2 works. If A sticks to source address A1, then the packet  
will also incur an ICMP and not make it. So either A sees the ICMP  
and selects a different source address, or it decides that A1-B2  
doesn't seem to work either and goes on to the next address pair. For  
instance A could try A2-B1. And this one works!

So from now on any outgoing packets with addresses A1-B1 in them are  
rewritten into A2-B1 and sent on their way.

Any complaints so far?

Scenario 2: big failure, and everything is wiped out except A4-B4.  
(From where I sit 99% of all traffic flows through Amsterdam, and  
most of that 99% over the AMS-IX. A nice big power failure there  
really hurts my connectivity.)

So A tries:

A1-B2
A2-B1
A1-B3
A3-B1
A1-B4
A4-B1

and on and on and on, until it eventually determines that A4-B4 works.

You don't want this to happen. So what's the alternative? Give up  
after the second try? The fourth? The n^2/2th?

Remember that while all of this is going on, the transport protocol  
sees a black hole. So at any time, the transport can decide to time  
out. The shim doesn't do anything that actually _hurts_ regular  
transport protocols.

Delivery-date: Fri, 19 Aug 2005 16:13:54 +0000
Date: Fri, 19 Aug 2005 11:13:11 -0500
From: Spencer Dawkins <[email-masked]>
Reply-To:  [email-masked]
To: shim6 <[email-masked]>
Subject: Re: shim-aware transports

Iljitsch van Beijnum wrote:

> On 17-aug-2005, at 17:31, Spencer Dawkins wrote:
> 
>> when we were discussing TCP in TRIGTRAN, we pretty quickly  converged 
>> on NOT notifying transports when bandwidth available  changed, because 
>> TCPs have to respond to losses, and adding  additional hints just adds 
>> stack complexity beyond the complexity  that is unavoidable.
> 
> 
> Well, I'm sorry to have to say it, but that's just STUPID.

... FWIW, I never said I thought it was the right answer :-)

But my point in writing was mostly to say that I encourage postponing 
"transport-aware" work until we make some progress on 
"transport-unaware" SHIM6, and then making sure that, wherever the work 
gets done, we are working closely with the transport community to meet 
their needs with protocol mechanisms they will actually make use of.

Spencer

Delivery-date: Fri, 19 Aug 2005 15:31:32 +0000
Date: Fri, 19 Aug 2005 17:30:56 +0200
From: Brian E Carpenter <[email-masked]>
Organization: IBM
To: Pekka Nikander <[email-masked]>
CC: Geoff Huston <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: shim-aware transports

Pekka, we do agree, and I agree with what Dave Meyer wrote too.

Simply put, the shim needs enough state to decide when to switch
locators on behalf of dumb transports, and it needs a way to make that
state accessible to smart transports. What I just realised is the
access for a smart transport could be an API, or it could be
direct access to the data structure maintained by the shim. We can
certainly model it as an API, but direct access might be needed
for performance reasons in an actual implementation.

    Brian

Pekka Nikander wrote:
> Brian,
> 
> I have a feeling we have some sort of a communication problem here  but 
> that we fundamentally agree; this message is an attempt to  clarify what 
> I am trying to say.
> 
> I am all for the initial goal of keeping the ULPs unchanged.  That's  
> the whole point.  However, when doing so I also think that we  
> *probably* should adopt the conservative approach adopted by e.g.  SCTP, 
> i.e., to have one preferred locator and to use the rest as  equal 
> backups.  Two reasons: keep it simple, and less surprises to  the 
> transport algorithms.
> 
> Now, AFAICT, the discussion below cornered around what to do next,  
> after the initial goal has been reached.  What I remember from Paris  we 
> all seem to agree that we need some kind of an interface between  the 
> shim layer and ULP.
> 
> My two messages (URLs below) define a strawman for such an interface,  
> while trying to keep the system at the same time as simple as  
> possible.  I don't see that changing at all the way the shim works  with 
> unmodified ULPs, or what state it maintains.
> 
> On the other hand, what I find somewhat worrisome are allusions that  we 
> should make the state at the shim layer more complex as we add  
> multi-addressing support to ULPs.  It looks to me that some folks  would 
> like to do locator-policy decisions at multi-addressing  transports, and 
> then push those policies somehow to the shim, to be  executed there.  I 
> find that unnecessarily complex.  Why can't the  ULP just pick the 
> locators, if it knows about them; in that case the  shim just protects 
> the transport from using locators that are no  longer valid.
> 
> So, all I am saying is KISS, and so far the method Christian and I  deviced
> 
> 
> Going a little bit deeper,
> 
> On Aug 19, 2005, at 12:55, Brian E Carpenter wrote:
> 
>> Pekka, since the initial goal is no change to the ULPs,
>> the state is all going to be in the shim, isn't it?
>> (Or perhaps better to say, in a data structure initially
>> maintained exclusively by the shim, but perhaps accessed
>> by shim6-aware ULPs in a later stage.)
>>
>>    Brian
>>
>> Pekka Nikander wrote:
>>
>>> I am just wondering how much state we want to push to the shim  
>>> layer  versus keep at the     ULPs.  See my other notes about that:
>>> http://psg.com/lists/shim6/msg00663.html
>>> http://psg.com/lists/shim6/msg00671.html
>>> It looks to me that we may be able to keep the signalling  interface  
>>> pretty simple, without any loss of generality.  But  maybe I am  
>>> missing something there.
>>> --Pekka Nikander
>>> On Aug 19, 2005, at 1:53, Geoff Huston wrote:
>>>
>>>> Brian noted that  making the shim  work with unmodified IP and   
>>>> unmodified upper layers needs to get shipped first, and thats a   
>>>> good priority.
>>>>
>>>> Brian also noted that this implies a two-way API between the  shim  
>>>> and the ULP, in which the necessary events are transmitted  up or down.
>>>>
>>>> So I'd like to float the concept of a placeholder about the   
>>>> interaction with transport that hopefully will allow the  priority  
>>>> work to proceed while still keeping enough flexibility  around to  
>>>> allow transport sessions to be able to be more  explicit about what  
>>>> they want from the shim layer, but to define  this interaction at a  
>>>> later stage in the WG's life.
>>>>
>>>> It seems to me that the basic mechanics of this can be satisfied  
>>>> by  defining a rich enough signalling interface between the shim  
>>>> layer  and the upper level sessions - perhaps allowing for the  
>>>> inclusion  of the relevant locator set, and allowing attributes  to 
>>>> be placed  on individual locators as well as on the complete  set in 
>>>> either  direction. In other words this allows an upper  level to be 
>>>> more  specific about locator preference either by  attribute or by  
>>>> explicit choice by passing a rich information  bundle to the ULP 
>>>> via  this signalling interface.
>>>>
>>>> Placing the potential functionality in the hands of the session   
>>>> layer also infers that the shim state may not necessarily be   
>>>> endpoint-to-endpoint, but may operate at a finer level of  
>>>> selection  (the 'equivalence" topic I referred to at the WG  
>>>> meeting). It also  infers that an individual session may want to  
>>>> trigger a locator  change that appears to be an isolated request  - 
>>>> in which case there  appears to be a logical need for dynamic  shim 
>>>> state forking in some  form (i.e. a shim state may state in  a 
>>>> general host-to-host state,  but individual sessions may 'fork'  out 
>>>> of this general state by  requesting individual locator  change 
>>>> trigger settings, for example.)
>>>>
>>>> If you allow vertical (and peer-to-err horizontal) locator   
>>>> attribute signalling, and also dynamic shim state forking you  can  
>>>> then say  that the mechanisms for the ULP to signal quite  explicit  
>>>> hints to the shim layer as to when and what locator to  use, and 
>>>> for  the shim to respond with a forked state as required  and signal 
>>>> this  to the peer remote shim entity is within the  capability of 
>>>> the  protocol. How the fields are defined and used  are up to a 
>>>> later  incarnation of the or another WG, that will  define the 
>>>> attributes  and  their semantics.
>>>>
>>>> Does this help?
>>>>
>>>>   Geoff
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>>
> 

Delivery-date: Fri, 19 Aug 2005 15:12:11 +0000
Date: Fri, 19 Aug 2005 16:11:34 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]

On Fri, 19 Aug 2005, marcelo bagnulo braun wrote:

>> Why must host1 detect this? Host2 could also ;).
>> 
>
> not in a unidirectional connectivity scenario
>
> consider the case where the failure implies that:
> PrefA:Host1 -> Host2 is not working
> PrefB:Host1 -> Host2 is working
> Host2 -> PrefA:Host1 is working
> Host2 -> PrefB:Host1 is not working
>
> How would you cope with this case?

How important is this case?

Further, in your scenario, this was due to a local-failure near 
Host1. A failure which can easily be detected locally without any 
need for n^2 probing.

What's needed is:

- Host1 to detect the local failure and update the exit path to use
   (and hence the source to use)
 	- this is achievable in multiple ways
 	- none of which need be in shim6
 	- none of which require shim6 to be aware of SAS or egress
 	  issues

- Host2 shim6 to detect host1's valid locators have changed
 	- Maybe because it receives a packet from Host1 with a new
 	  source
 	- Maybe because Host2's reachability probes detect PrefB

How common is this failure mode?

You want to specify that shim6 be able to work around /any/ kind of 
routing failure, anywhere on any part of the internet affecting any 
path between Host1 and Host2.

My gut feelings though are:

- Failures typically are near the edges
- Failures are typically bi-directional for a given path
- Uni-directional failures tend to be due to /congestion/, not
   actual failures - again, typically at the edges. Congestion related
   "failures" tend to be very transient/sporadic.
- Failures in the 'middle' are uncommon, and tend to affect /huge/
   numbers of paths (ie there's a decent chance it will take out /all/
   your paths)
- The problem of uni-directional failure on two /unrelated/ paths at
   the same time is *tiny*

Hence (as a gut feeling):

- n^2 probing in shim6 is simply introducing huge expense in order to
   solve a very uncommon problem

You think the tradeoff in order to achieve perfection is worth it.

I don't, I think the above is a general quality-of-internet-routing 
problem. I think it's something that should and will be tackled 
within the routing area, where people have been and are continuing 
working on optimising routing protocols (from OSPF to BGP) to cope 
gracefully with failures and restarts in order to eliminate some 
common scenarios where routing-loops can occur in todays routing 
protocols.

I don't see a compelling reason to consider problems in internet 
routing to be something shim6 needs to introduce great complexity for 
in order to work around, when a simple approach (let underlying OS 
routing pick the local prefix) will likely allow 99% of failures to 
be detectable and worked around.

I'd like to see more information on path failure modes seen on the 
internet, what is common, what is not, before I would change my 
position.

> this last point is influencing SAS and trying with alternative 
> source locators, which is basically what we are considering here 
> and that you are opposing right?

Correct.

As I stated later in that mail, you can achieve nearly the same thing 
by simply relying on external means to shim6 to update which egress 
interface/address the OS uses.

My position is that "nearly the same thing" is more than good enough, 
particularly when it could implement a great amount of complexity 
from shim6.

Note that such external mechanisms would *not* be precluded from 
doing n^2 probing, or any other fancy scheme they want to implement. 
(I wouldn't care to implement it, but..). Ie I'm not arguing shim6 
should preclude such probing, only that the base spec should assume 
simple mechanisms and provide the protocol tools to allow more 
complex probing (eg a 'PROBE' message or somesuch).

So weirder, more complex (and unneeded imho) probing and getting 
involved in SAS would be an implementation detail ;).

> we have already discussed this point (in multi6) and imho it is not 
> such a good idea to have the hosts to receive a full BGP feed.. i 
> think there was ssome kind of concensous on this point, but maybe 
> now has changed...

BGP feed (not advertising anything) is simply /one/ option - I gave a 
list of possibilities, using a routing protocol was just one of them. 
Using BGP would only be suitable for enterprise sites (presuming 
shim6 allows for a split/proxy mode).

Eg, DSL connected hosts: My experience, with my ISP, is that the only 
failures I actually notice are telco/local-loop related, or related 
to the DSL cable run in my house - and my PPP stack (which implements 
a type of keepalive) is the first to notice.

Maybe I just have a good ISP.

Oh, there's yet another option, in the long-run: If your ISP(s) sucks 
so much that odd uni-directional path failures are regular enough in 
occurance -> drop that ISP and go to another.

> The idea is that shim performs e2e failure detection (as i already 
> mentioned earlier) so that the fate of the communicating parties 
> i.e. the apps is shared with the fault tolerance mechanism and that 
> the shim can detect all the potential outages and recover from 
> them. Having different mechanisms deal with different types of 
> failures would result in a reduced protection i guess

I can see why the goal is attractive. I wonder though whether the 
required complexity is worth it.

> but you are assuming that host1 detects local failures through 
> other means

Correct.

> and this other means are likely to be injecting a full 
> BGP feed into end hosts, right?

No, that's not likely at all.

> Your math is correct

Phew, cause it's been a while ;).

> now let me ask: how many unidirectional address pairs are available 
> between two hosts having y and n addresses each one?

> i guess that we agree that there are y^2+n^2 different 
> unidirectional address pairs, right?

Right.

> So the point is: if you want to provide full fault tolerance you 
> need to explore them all, if you don't, there may exists available 
> paths that you are discarding, hence there are communications that 
> could be preserved but you are not finding the available path to 
> use.

Right.

Is it worth though?

You're coming from a position where you do not wish to have to rely 
on the stability of internet routing. In your world the "internet 
cloud" is likely to be swiss-cheese (full of "black holes" ;) ), and 
can't be relied upon.

That isn't in line with my experience of the internet. IMHO, it's 
reliable "enough". Further, it there are reliability problems in 
internet routing, then surely the best way to deal with them is in 
the /routing area/ working groups? ;)

Ie, is it worth it?

Unreliable ISPs will (eventually) be taken care of by market 
pressures. They'll either fix their problems, maybe take advantage of 
graceful-failure mechanisms which are becoming more prevalent in 
routing protocols and implementations, or they will slowly die as 
their customers move elsewhere.

Another factor to consider in routing reliability, that (IMHO) 
obviates need to worry about so much about it in shim6: VoIP. As more 
and more telcos switch over to unified IP core for both their voice 
and data services, they're putting ever resources into researchers, 
implementors and IETF to make routing 'perfect', at least for 
intra-AS - they can't tolerate packet loss or odd latencies because 
customers will *hear* it. In time I suspect even inter-AS routing 
will be optimised to be much more stable than it is today, as 
eventually (i suspect) VoIP inter-telco peering will replace SS7 (and 
they'll get researchers/implementors and IETF to optimise that case 
too).

Anyway, I think that's a fairly exhaustive explanation of why I think 
perfection in using available paths should /not/ be a core shim6 
objective.

I'll stop harping on on that topic. I would like to see justification 
as to why it should be though.

> Of course, you may have optimizations, like testing two 
> unidirectional paths (one in each directions) with a single packet 
> exchange (2 packets instead of 4), using local information for 
> discarding some addresses and so on, but again these are only 
> optimizations.

You could I guess. I wouldn't, but you could. I suspect most people 
would be happy with just assuming the internet "cloud" is mostly 
reliable.

Particularly: If it means that shim6 drafts are simplified, easier to 
write, easier to get reviewed and approved, easier to implement the 
basic functionality, etc.. then that means shim6 gets "to market" 
(ick) quicker.

If I can get to use shim6 in N years because it's simple, rather than 
N+2 because it tries to cope with every possible path-failure..

> how? i guess that you are asuming a BGP feed on hosts right?

No, I was assuming a split-mode of operation, with 
shimmed-address/ULID using hosts not doing the shimming, but edge 
'shim routers' doing it (hence BGP would be confined on those edges).

The drafts consistently refer to shim sitting on each host. So my 
assumption appears to be wrong. Though, I don't see why split-mode 
would not be possible (particularly if ULID's are IPv6 addresses and 
composed of a prefix and host identifier..).

> I mean, a hosts with a single deafult route, wouldn't really know 
> which of the n addresses available in its single interface to use 
> for a given destiantion address... i mean DAS would not result in 
> any particular address and the source address would be selected 
> randomly...

There are many other possible mechanisms.

A host could have the following default route:

default via ISP1-gateway
 	via ISP2-gateway

ISP1-gateway device X src ISP1-PA-address
ISP2-gateway device X src ISP2-PA-address

Some external mechanism could update this route as required. Be it 
gateway-probing, probing "well known hosts on the internet", a RIP 
default announced from border routes, or even an application which 
monitors route-lookups or the OS route-cache and probes those and 
updates routes accordingly.

The wealth of possibilities if you at least /allow/ the 
source-selector mechanism to be external to shim seems a compelling 
reason for shim6 to /not/ (by default) get involved.

> So far, we are assuming that the shim is a host based approach and 
> that each host performs all the functions of the shim

Yes.

> the case of the proxy that you are mentioning have been considered 
> and it is attractive but it presents some difficulties, especially 
> w.r.t security... perhaps you could try to consider the security 
> implications of that split that you are considering...

There is one case where 'split' or 'proxy' mode shimming would be 
possible without security ramifications, I think. The case where the 
ULID's in use are IPv6 addresses, network prefix and host identifier.

Then a simple stateless static mapping on the "shimmers" (which would 
be gateways into/out of the "shimmed" ULID network) will do. Security 
then is simply not a concern, no more than it is for a normal router 
with a static forwarding table.

> not sure what you mean here... ULID is upper layer identifier, 
> right? so the ULID belongs to the shim host, so i guess they are in 
> the same machine

Right.

As above, it seems possible to me that shim6 could allow for at least 
one usage that would not require shim and "ULP" to be same machine. 
It could be very useful for small/medium size multihomed sites (ie 
not large enough to get a global IPv6 prefix).

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Don't let your status become too quo!

Delivery-date: Fri, 19 Aug 2005 14:32:44 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Fri, 19 Aug 2005 16:33:12 +0200
To: Pekka Nikander <[email-masked]>

El 19/08/2005, a las 10:30, Pekka Nikander escribi=F3:

>
>> In any of these approaches, the locator annotated for outgoing=20
>> packets are a suggestion of they must be honored? i mean, i guess=20
>> this is related with how does the shim informs the ULP that a given=20=

>> locator is no longer reachable, right?
>
> I don't know what would be the right approach.  Currently I think that=20=

> the best approach might be to signal the ULP that the shim layer was=20=

> unable to send the packet due to the suggested locator no more being=20=

> associated with the ULID.  At that point the ULP could ask the shim=20
> for the new set of active locators, and pick a suitable one from=20
> there.
>
> The alternative would be the shim layer to pick the locator, in the=20
> same way as it would pick if there was none given by the ULP.  The=20
> drawback of this approach is that the ULP has no information about the=20=

> locator having been overwritten...
>

I think that the distinction of the different types of addresses and=20
address pairs defined in draft-ietf-shim6-failure-detection-00.txt=20
would be very useful to understand this point.

According to this draft we have

- Available Addresses
- Locally Operational Addresses
- Operational Address Pairs (unidirectional and bidirectional)

Now the question is which type of addresses should the shim inform the=20=

ULP

I guess this very much depends on what the ULP will do with these=20
addresses...

I mean, we can think the case of an app that performs referrals and=20
that it wants to find out all the addresses (its own or the remote=20
host) so that it can send them in a referral to a third party. In this=20=

case, since we don't know when the addresses will be used, i guess it=20
make sense that the shim inform the ULP about the "available=20
addresses". The "locally operational addresses" could be used. I guess=20=

that address pairs don't fit here.

Now, if the shim wants to inform the ULP about which source and=20
destination addresses can be used for communicating, i guess that the=20
shim should present the "locally operational addresses" sine not all=20
the available addresses can be used for the communication.

Finally, if the shim wants to inform the ULP about a failure in the=20
current addresses used, the information provided by the shim should=20
involve the "operational address pair" since in the case that the shim=20=

detects a failure, the information available in the shim is that a=20
given address pair is no longer working. I mean, in this situation it=20
is more appropriate to consider a failed address pair rather than an=20
address

So potentially the shim could inform the ULP about:
- available addresses (local and remote host)
- locally operational address
- changes in the available addresses
- changes in the locally operational addresses
- failure in an address pair (in particular in the currently used one)

>>> Hence, my question is if the transport are likely to need anything=20=

>>> else than the following:
>>>   - both ULIDs and locators in incoming packets
>>>   - ability to define locators for outgoing packets
>>>     - what happens if the locators are not available any
>>>       more?  Host-internal ICMP-like response?
>>
>> coming back to this point.. by locators you mean locator pair, right?=20=

>> i mean the shim should inform the ULP that a given locator pair is no=20=

>> longer reachable, right?
>
> As I indicated above, I am no that far yet in my thinking.  So, the=20
> brief answer is that I don't know.
>
>>> Anything else?
>>
>> I wonder if it wouldn't be useful to allow the ULP to inform the shim=20=

>> about some sort of preferences w.r.t. which alternative locator pair=20=

>> to try with in case of an outage.
>
> Hmm.  Let's consider the complexity implications.  If the shim just=20
> fails and signals the ULP about the failure, the ULP will have all the=20=

> necessary information about picking a locator to use.  It can easily=20=

> update its preferences dynamically based on multiple source of input,=20=

> e.g., based on flows on different locators.  That would keep the shim=20=

> simple, pushing the complexity to the transport.
>

in this case, all the path exploration (probing) would be managed by=20
the ULP, right?

my concern here is that path exploration is proving to be quite=20
complex, especially dealing with unidirectional paths, so probably the=20=

shim could provide a good support for this

i mean, i am not sure how the ulps would deal with unidirectional=20
paths, path exploration and so on...

> On the other hand, if the shim has more information, such as=20
> preferences per transport, that brings more complexity to the shim=20
> layer.  Conversely, if we accept the architectural principle that I=20
> suggested, we shouldn't pick this way, as in this case the shim knows=20=

> more than is absolutely necessary.
>
> I am currently tempted to try to keep the shim as simple as possible.

right, but as i understand it, the shim already has to deal with=20
unidirectional path support, path exploration and failure detection. In=20=

addition, i guess that the shim will need to provide some form of=20
support for expressing policy/preferences.
I guess that the additional complexity in this case, is that these=20
preferences are per ULP rather than per ULID... but this brings us to a=20=

more general problem, which is: how do we handle the case where there=20
are two ULPs communicating with a given shim context but they are=20
asking to use different locators for the communication?

I mean suppose that two apps are using a single shim context and that=20
the different ulps announce different locators for the communication=20
(with the same ulids), would this be an issue?

>  Due to having transports that won't know about locators, we clearly=20=

> need to have some sort of preferred locator at the shim, and=20
> associated fail over mechanisms.  However, to me it "smells" better if=20=

> we would not use them, at least by default, with more intelligent=20
> transports that supposedly know better.
>

agree

> But there is one point here:  As we apparently need to have a=20
> preferred locator at the shim, we should allow the transports to learn=20=

> about that, too.

i am not sure i can see why...

>   Apparently we also need some administrative interface for changing=20=

> the preferred locator dynamically....
>

the preferred locator per ULP you mean?

Regards, marcelo
> --Pekka
>

Delivery-date: Fri, 19 Aug 2005 13:55:51 +0000
Date: Fri, 19 Aug 2005 14:55:07 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: [email-masked], shim6 <[email-masked]>
Subject: Re: shim-aware transports
Mail-Copies-To: [email-masked]

On Fri, 19 Aug 2005, Iljitsch van Beijnum wrote:

> Well, I'm sorry to have to say it, but that's just STUPID.

I'd have thought "wise", but hey.

> What if I have a 119 MB/s gigabit ethernet link and a 1 kB/s GSM 
> data connection, and I'm downloading a nice big file with my fully 
> RFC 1323-compliant IP stack, and then the gigabit link goes down so 
> I rehome to the GSM link.

Jammer[1].

We're not here to solve problems in transport protocols. Even if try 
accomodate shortcomings in certain protocols by introducing 
additional interactions, what happen when those shortcomings are 
fixed (in that protocol or by a superceding one?) - you're left with 
a then redundant set of interactions. Interactions which make for 
(redundant) implementation difficulty.

Further, you have to accomodate all transports. Many of which have 
quite different expectations and information.

> Suppose the file is coming from 100 ms away, so my window is 12 MB 
> or so. That means that worst case, my GSM link could be saturated 
> for more than THREE HOURS because of that single window worth of 
> data that's in flight.

Jammer :).

But the receiver can close the window, can't it?

> This "complexity is scary" belief in IETF is getting ridiculous and then 
> some. (While at the same time you need to read some 20 RFCs to implement 
> SNMPv3. Good thing they never went for a "complex network management 
> protocol".)

Complexity /is/ scary.

It raises the bar to understanding, hence introduces additional 
potential for protocol 'bugs' to be overlooked. It raises the bar for 
implemention, in prototyping, debugging, etc.

Make things as complex as they /need/ to be, no more.

1. roughly translated: tough luck

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Politics makes strange bedfellows, and journalism makes strange politics.
 		-- Amy Gorin

Delivery-date: Fri, 19 Aug 2005 13:53:58 +0000
Date: Fri, 19 Aug 2005 06:53:36 -0700
From: David Meyer <[email-masked]>
To: Brian E Carpenter <[email-masked]>
Cc: Pekka Nikander <[email-masked]>, Geoff Huston <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: shim-aware transports

--qlTNgmc+xy1dBmNv
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Aug 19, 2005 at 12:55:12PM +0200, Brian E Carpenter wrote:
> Pekka, since the initial goal is no change to the ULPs,
> the state is all going to be in the shim, isn't it?
> (Or perhaps better to say, in a data structure initially
> maintained exclusively by the shim, but perhaps accessed
> by shim6-aware ULPs in a later stage.)

	Interstingly, when the "shim-aware transport" thread
	started on the list, I reviewed the charter and was
	somewhat confused as to the intention behind (what I
	believe to be) the operative/relevant requirement there,
	specifically    =20

o Changes in the addresses that are used below the shim will be
  invisible to the upper layers, which will see a fixed address
  (termed Upper Layer Identifier or ULID).

	and asked the co-chairs about this very topic. Geoff
	suggested several ways that this could be dealt with,
	including the fact that seeing a "fixed" ULID doesn't
	preclude annotating that ULID with various forms of
	information that may be useful to a ULP (i.e., a
	transport protocol).=20

	You seem to be suggesting a variant of that
	thinking. In all of the cases I discussed with Geoff, as
	well as what you are proposing here, the shim state is
	maintained by the shim layer. That seems in-scope and
	perhaps more importantly, reasonable (for now anyway).

	Dave

--qlTNgmc+xy1dBmNv
Content-Type: application/pgp-signature
Content-Disposition: inline

--qlTNgmc+xy1dBmNv--

Delivery-date: Fri, 19 Aug 2005 13:53:49 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Fri, 19 Aug 2005 15:53:50 +0200
To: Paul Jakma <[email-masked]>

El 18/08/2005, a las 16:13, Paul Jakma escribi=F3:

> On Thu, 18 Aug 2005, marcelo bagnulo braun wrote:
>
>> so far this is exactly what is contained in Iljitsch draft, Jari's=20
>> draft and this is the basic concesous we have so far as i understand=20=

>> it.
>
> Well, unless I'm reading the wrong draft (the one Iljitsch did the=20
> presentation on at IETF?), my understanding of the draft is that it=20
> explores the various possible solutions and discusses some the=20
> considerations, but does not draw a conclusion.
>

right, but at least as i read it, there are some basic assumptions=20
made, including:

- the mechanism used to explore different paths is to change the source=20=

and destination address
- the host will explore different paths to find at least one available=20=

if exists (the maximum number of explorations is hence 2^n but there=20
may be not need to explore them all to find a working path)
- unidiractional paths will be supported
- no bgp feed to end hosts

what is being discussed is which is the failure detection mechanism and=20=

eventually what is is the path exploration mechanism (this is more in=20
Jari's draft)

> So yes, I'm rehashing things in that draft, but I'm trying to focus=20
> the discussion by discounting some of the possible solutions (or at=20
> least relegating them to "implementation discretionary" and=20
> concentrate on specifics ;).
>
> Is my understanding correct?
>
>> At this point, what we are discussing is how this monitoring=20
>> mechanism should look like
>
> Ok, good.
>
>> and how this ULP feedback should look like
>
> I would discount this from consideration, other than to ensure any=20
> mechanism allows an implementation to make use of local positive=20
> feedback for optimising things.
>

well, negative feedback is also in consideration

>>> - You can't mandate a shim6 implementation /must/ send probes
>>>   regularly, to allow other side to use for reachability detection
>
>> not sure what you mean here....
>
> You can't have one side expect to get regular 'pings' in order to=20
> determine reachability, otherwise local-positive-feedback to suppress=20=

> probes/keepalives/pings/whatever-you-call-them would not be possible.
>
>> if you are saying that ULP feedback could result in avoiding shim=20
>> probing, then i agree and this is exactly what we have been=20
>> cosnidering
>
> Great.
>
>> We have also agreed that failure detection need to be unidirection,=20=

>> the discussion now is whether the proposed mechanisms are actually=20
>> unidirectional or whether they have an underlying bidirectional=20
>> assumption
>
> Unidirectional would, I guess, require that all the information needed=20=

> to setup a shim is available in one message.
>

not really because the idea is that there may be two unidrectional=20
paths (one in each direction) so that address pair used for=20
communicating in one direction differs from the one used in the other=20
direction

regards, marcelo

> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Faith is under the left nipple.
> 		-- Martin Luther
>

Delivery-date: Fri, 19 Aug 2005 13:53:18 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Fri, 19 Aug 2005 15:53:59 +0200
To: Paul Jakma <[email-masked]>

El 18/08/2005, a las 17:06, Paul Jakma escribi=F3:

>
...
>> Suppose you have a multihomed site with ISPA and ISPB and that they=20=

>> have assigned PrefA and PrefB respectivelly
>>
>> Suppose you have Host1 in the multihomed host and that it is=20
>> communicating with Host2 outside the multihomed site. For that=20
>> communication, Host1 is using address PrefA:host1 both as locator and=20=

>> as ULID. I assume that host2 has a single address host1
>
> Is host2 shim-aware?

yes

> It must be for scenario in next sentence to work. Particularly given=20=

> RPF filtering at ISPs on "PA" customers.
>
>> Now, suppose that an outage in the link between the multihomed site=20=

>> and ISPA occurs.
>
> Ok.
>
>> Host1 detects and needs to do something about it, how can he try with=20=

>> an alternative path?
>
> Why must host1 detect this? Host2 could also ;).
>

not in a unidirectional connectivity scenario

consider the case where the failure implies that:
PrefA:Host1 -> Host2 is not working
PrefB:Host1 -> Host2 is working
Host2 -> PrefA:Host1 is working
Host2 -> PrefB:Host1 is not working

How would you cope with this case?

> If, as part of shim6 control, host1 had, /in advance/, informed host2=20=

> that prefB:host1 was /also/ a valid locator, then:
>
> - if host2 was first to detect the problem, it could simply switch to
>   using the prefB:host1 locator
>
> - if host1 notices there is a problem with bidirectional
>   communication, it can simply start using the PrefB based locator
>   (which host2 already knows about and should accept)
>

this last point is influencing SAS and trying with alternative source=20
locators, which is basically what we are considering here and that you=20=

are opposing right?

> This presumes host1 is actively managing its source address according=20=

> to state specific to shim6. However, there is a *better* way:
>
> - dont have shim6 pick the ISP dependent part of the locator
> - let the OS's existing SAS mechanisms pick the source address
>   according to routing policy, as per normal, for each and every
>   packet
> - let some external mechanism detect the local failure
>
> Some (non-exhaustive) examples of such mechanisms:
>
> - link status information
>
> (eg PPP has its own heartbeat mechanism. PPP is often used for DSL=20
> connections, which will be a common case for the scenario described=20
> above for host1, today at least)
>
> - Routing information
>
> (eg, host1 /could/ get routing information from either or both of ISP1=20=

> and ISP2. Be it just a RIP default announced regularly on up to full=20=

> *read-only* BGP feeds)

we have already discussed this point (in multi6) and imho it is not=20
such a good idea to have the hosts to receive a full BGP feed.. i think=20=

there was ssome kind of concensous on this point, but maybe now has=20
changed...

>
> - Link probing / dead-gateway detection software
>
> (At least one OS provides software to actively probe gateways and=20
> adjust routes as/when required)
>
> - Well-known remote host probing software
>
> And so on.
>
> So, I'm not sure shim6 is at all the best place to take on detection=20=

> of /local/ failures.
>

The idea is that shim performs e2e failure detection (as i already=20
mentioned earlier) so that the fate of the communicating parties i.e.=20
the apps is shared with the fault tolerance mechanism and that the shim=20=

can detect all the potential outages and recover from them. Having=20
different mechanisms deal with different types of failures would result=20=

in a reduced protection i guess

> Ie, in your scenario above, what I'm saying is:
>
> - Shim6 on host1 doesnt and shouldn't be involved in detecting local
>   failures
>

but you are assuming that host1 detects local failures through other=20
means and this other means are likely to be injecting a full BGP feed=20
into end hosts, right?

> - Only host2 needs to know and detect whether it must use PrefA or
>   PrefB based locator for host1.
>
> Ie, this should all be about reachability of the *remote* locator.=20
> Which local locator to use should simply *not* be in scope for shim6 -=20=

> then things get easier, and you can avail of one or more existing=20
> mechanisms for detecting local failures.
>

as i mention above, i am not sure if this supports the unidirectional=20
case that i mention above.

...
>>
>> This implies that in order to explore all the possible paths, we need=20=

>> to make n^2 probes.
>
> Yes, if you refuse to consider the option of leaving SAS to something=20=

> else (a wealth of mechanisms exist /already/), then you'll become=20
> convinced you need n^2 probes. :)
>
>> Now, it is important to realize that n^2 is just an upper bound, and=20=

>> that n^2 probes will only be performed when all paths have failed=20
>> except one and this is the last one you have tried with
>
> Even better is to realise that local address selection simply does not=20=

> need to be something shim6 has to concern itself with. Then the number=20=

> of probes, in the worst case, goes down to n, where n is the number of=20=

> locators you have to hand for the remote shim6.
>
> Don't forget, n^2 is per side. Each side will have to have to do it.
> So you're looking at, worst case, for host1 with n locators and host2=20=

> with y locators, by my potentially incorrect algebra:
>
> shim6 doing SAS and probing each side:
>
> 	y^2 + n^2
>
> In the worse case, both y and n tend to infinity, so generally n =3D =
y,=20
> so the worst case therefore is:
>
> 	2*n^2
>
> as n and y -> infinity.
>

Your math is correct
now let me ask: how many unidirectional address pairs are available=20
between two hosts having y and n addresses each one?

i guess that we agree that there are y^2+n^2 different unidirectional=20
address pairs, right?

So the point is: if you want to provide full fault tolerance you need=20
to explore them all, if you don't, there may exists available paths=20
that you are discarding, hence there are communications that could be=20
preserved but you are not finding the available path to use.

Of course, you may have optimizations, like testing two unidirectional=20=

paths (one in each directions) with a single packet exchange (2 packets=20=

instead of 4), using local information for discarding some addresses=20
and so on, but again these are only optimizations.

...

> If you figure out that the local source to use is better left to=20
> other, existing, mechanisms, and only probe the remote side,
> That's great, but you have 0 idea whether its the right address.
>
> Determining the right address involves:
>
> - route lookup on the destination to find the local output
> - lookup source for this output
>
> Shim6 /could/ do this, but on the other hand it could just leave the=20=

> local address in packets sent down the stack as unspecified, and the=20=

> OS will just do it for shim6.
>

how? i guess that you are asuming a BGP feed on hosts right?

I mean, a hosts with a single deafult route, wouldn't really know which=20=

of the n addresses available in its single interface to use for a given=20=

destiantion address... i mean DAS would not result in any particular=20
address and the source address would be selected randomly...

> What is the benefit for shim6 to do this, that it MUST? (It doesn't=20
> need to. Any such optimisations are purely implementation specific).
>
> Ergo: No need for SAS in shim6.
>
>> No, shim will try to honor the policy table (or any other tool to=20
>> express policy we need to define) Obviously, if the path preffered by=20=

>> the policy is not available, then shim will have to use others of=20
>> course
>
> See, there's an existing way to send a packet to a remote host and=20
> have the source be set appropriately for the preferred path - don't=20
> specify the source address...
>
> No complications needed.
>
>>> That's entirely scaleable and well within reason for deployment at=20=

>>> 'enterprise' shim6 sites.
>>
>> agree, but this is not enough to preserve established communications
>
> Why exactly?
>
>> Are you considering the case that full BGP feed is injected to the=20
>> hosts, so that the hosts can find out which path is available towards=20=

>> its final destination?
>
> Well, in a BGP fed and shimmed network, I would imagine the shimming=20=

> would be done just on one or two hosts (or routers) at the edge of the=20=

> shimmed network. The BGP feed would be confined to those.
>

well, this seems a bit far from what is currently being considered imho

So far, we are assuming that the shim is a host based approach and that=20=

each host performs all the functions of the shim

the case of the proxy that you are mentioning have been considered and=20=

it is attractive but it presents some difficulties, especially w.r.t=20
security... perhaps you could try to consider the security implications=20=

of that split that you are considering...

> As far as I know, the ULID and shimming host need not be same machine,=20=

> is my understanding correct?

not sure what you mean here... ULID is upper layer identifier, right?
so the ULID belongs to the shim host, so i guess they are in the same=20
machine

regards, marcelo

Delivery-date: Fri, 19 Aug 2005 13:35:30 +0000
Date: Fri, 19 Aug 2005 14:34:48 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: marcelo bagnulo braun <[email-masked]>, shim6 <[email-masked]>
Subject: Re: about reachability detection draft
Mail-Copies-To: [email-masked]

On Fri, 19 Aug 2005, Iljitsch van Beijnum wrote:

> application won't give up, but when the shim can't find a working 
> address pair, we should pass on the ICMP errors to the upper layer. 
> At the same time, I'm not sure if we should ever really give up as 
> long as upper layers keep sending traffic: connectivity can come 
> back at any time.

For unmodified ULPs, is that wise?

The ICMP error has no context in the ULP, it's only context is the 
shim packet. Rather you need to interpret ICMP errors received and 
/maybe/ generate ICMP or other errors back up (there may or may not 
be a close correspondance).

Needs further thought imho.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Two wrongs don't make a right, but three lefts do.

Delivery-date: Fri, 19 Aug 2005 13:35:27 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: shim-aware transports
Date: Fri, 19 Aug 2005 15:34:45 +0200
To: [email-masked]

On 17-aug-2005, at 17:31, Spencer Dawkins wrote:

> when we were discussing TCP in TRIGTRAN, we pretty quickly  
> converged on NOT notifying transports when bandwidth available  
> changed, because TCPs have to respond to losses, and adding  
> additional hints just adds stack complexity beyond the complexity  
> that is unavoidable.

Well, I'm sorry to have to say it, but that's just STUPID.

What if I have a 119 MB/s gigabit ethernet link and a 1 kB/s GSM data  
connection, and I'm downloading a nice big file with my fully RFC  
1323-compliant IP stack, and then the gigabit link goes down so I  
rehome to the GSM link.

Suppose the file is coming from 100 ms away, so my window is 12 MB or  
so. That means that worst case, my GSM link could be saturated for  
more than THREE HOURS because of that single window worth of data  
that's in flight.

This "complexity is scary" belief in IETF is getting ridiculous and  
then some. (While at the same time you need to read some 20 RFCs to  
implement SNMPv3. Good thing they never went for a "complex network  
management protocol".)

Delivery-date: Fri, 19 Aug 2005 13:24:19 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: about reachability detection draft
Date: Fri, 19 Aug 2005 15:23:51 +0200
To: marcelo bagnulo braun <[email-masked]>

On 17-aug-2005, at 16:24, marcelo bagnulo braun wrote:

> Suppose that A is communicating with B
> Suppose that we are using FBD so that a given frequency of packets  
> (data or just signaling) is guaranteed by the shim in each direction

> Suppose now that A stops receiving packets. This implies a failure  
> in the B->A path.

Yes. (Assuming that A is still sending, if not the session could  
simply be idle.)

> Now are you assuming that when A stops receiving packets A should  
> try with alternative paths?
> I am not sure this is good approach, since the path A->B may be  
> working properly.. moreover, maybe this path A->B is the only one  
> working.

There are three possible actions:

1. Do a "lightweight" reachability test using the currently used  
locator pairs for both directions.
2. Start a full reachability evaluation procedure that will look at  
all locator pairs until it finds one that works.
3. Switch to a new address pair.

I think 3. is the wrong thing to do. Most importantly, because if one  
pair fails it's very likely that at least some other pairs also fail,  
so jumping to a new address pair blindly makes little sense. Also, in  
order to be able to do this, we must have previously authenticated  
the alternative address for the correspondent (or the correspondent  
for our alternative address) to avoid redirection attacks. So this  
basically means authenticating all possible addresses for both sides  
when the shim state is created. That's a waste of resources.

1. may make sense, but if we know there should be packets flowing in  
both directions but they're missing in one direction, it's very  
unlikely that the current address pair is still working. We should  
probably still test it, but I think we should also start looking at  
the first secondary address pair because it's likely that the  
currently used pair is dead. So that lands us at 2.

> As i see the FBD mechanism would be the following:

> A and B are communicating
> they are using FBD
> A stops receiving packets

> So, A informs B that the B->A path has failed (this implies some  
> form of signaling from A to B, which is likely to be required to be  
> somehow reliable, which is why i see a potential difficulty here)

The way I see it, A just starts a path exploration procedure. This  
means it will try address pairs until it gets an answer from B, so  
unless the path from A to B is completely broken at some point B will  
see a path exploration packet from A, which indicates to B that there  
is probably something wrong, so B would start doing its own path  
exploration.

> A continues using the path from A->B while B starts exploring  
> alternative paths

Right. You don't throw away your old shoes until you have new ones.  
Maybe the failure is temporary, so keep sending packets makes sense.  
And it's also the simplest thing to do.  :-)

> So you think that we need one packet type for periodic keepalives  
> for failure detection and another type of packet for exploring  
> alternative paths?

Yes. Assuming we go with the forced bidirectional thing, in the event  
that there is traffic from A to B, but no traffic from B to A, B  
needs to generate a packet anyway in order to let A know that  
everything is still working. Since A presumably monitors all packet  
types, the nature of this packet is of no importance. It could even  
be an IP header without any payload.

But for the full reachability evaluation we need lots of extra stuff.

So it makes sense to have this in different packet types, especially  
since the simple filler packets will probably be the most common.

> I am not sure about this since when two different unidirectional  
> paths are being used for a communication then the keepalive packet  
> will need to carry information about the locator pair used for  
> incoming packets.

Not sure what you mean.

> I mean, consider that a and Bare communicating, but they are using  
> two different unidirectional paths i.e. A->B is using locators A1  
> and B2, while B->A is using locators A2 and B2

> In this case when A sends a reachabilitity test request to B it  
> will use A1 and B1. However, when B replies, it will use B2 and A2,

Yes. Why would that be a problem?

> so i guess it would a good idea to include in this reply packet  
> information about the address used for the request packet i.e. A1  
> and B1

No, don't get the problem...

>>  And we need a reasonable level of authentication too, because we  
>> haven't previously established that these addresses indeed belong  
>> to the correspondent.

> I am not sure what is the level of security we need here yet, but  
> in any case imho it will be very related to the security used  
> during the shim context establishment.

> I mean, imho the critical part of the security if the addition of  
> new locators to the locator set. Once the locators have been  
> validated, using them shouldn't require to tight security  
> requirements imho

Agree, but on the other hand we don't want to give attackers the  
capability to make hosts think there are reachability problems,  
especially attackers that aren't able to sniff any of the setup  
traffic. So something simple like the TCP sequence number or the  
IPsec replay counter would be useful here.

>> On the other hand, when A1 <-> B1 is working happily, there is no  
>> need to use such a complex protocol: we are only testing one pair  
>> in each direction, and the correspondent has been authenticated  
>> earlier. If we wanted we could even use pings to determine whether  
>> this still works. (Well, sort of...)

> But as i see it, we are only going to try locators that are  
> included in the locator set available for that shim context, which  
> means that they have already been validated....

Since we need to test them for reachability right before we use them  
anyway, I tink it makes more sense to defer this validation until  
that time.

>>  Deprecation is irrelevant, as we can continue to use deprecated  
>> addresses.

> Well, deprecation means that somewhere in the near future, the  
> address won't be available anymore, right?

Right, so we keep using it until it's not available anymore.  :-)

> So i guess it make sense to keep on using it as ULID, but i wonder  
> if it wouldn't be a good strategy to try to rehome the ongoing  
> communications to an alternative locator...?

Are there any bad side effects when using deprecated addresses?

>>  A more interesting case is when an address is removed from the  
>> system. I don't remember which session it was, but in Paris  
>> someone was talking about how systems keep using addresses they no  
>> longer have because upper layers still use those addresses.

> well, in the shim this address should be kept as a possible ULID  
> but not as a valid locator i guess

Yes, it makes sense to keep addresses as ULIDs for a while after  
they've become unavailable. On the other hand, in theory the address  
could be assigned to someone else so we problably want to limit this.

On the other hand, if the address is an HBA address, presumably  
nobody else would be able to use it (certainly not as a ULID, only  
possibly as an RFC 3041 semi-collision), so there is no harm in  
keeping it around as a ULID.

>>  IMO this is a feature: if I unplug my ethernet from my powerbook  
>> and turn on my wifi, I get the same address on a different  
>> interface and my sessions are still alive. Under windows, things  
>> like this kill your sessions immediately.

> in this case i guess that this address should be:
> - kept as a ULID during all the time
> - should be removed from the locator set during the period which  
> the address was not available (i.e. during the time it took to  
> remove it from the ethernet and it is back in the wifi)

Maybe we should keep using the ULID as long as there is shim state  
that refers to it, but it's removed when the shim state that uses the  
ULID is removed.

>> The way I see it there is an ordered list of address pairs. The  
>> more probes fail to make it to the other side the lower the  
>> address pair ends up on the list, I imagine.  :-)

> but what happens when you only have 2 address paris for instance?  
> you may end up trying with the same two address pairs forever... I  
> mean, i guess we need a mechanism to give up trying, right?

Hm, maybe, maybe not... How does this work for current IP?

I guess that while the shim is still trying to move the session to a  
different address pair, we should block ICMP errors and such so the  
application won't give up, but when the shim can't find a working  
address pair, we should pass on the ICMP errors to the upper layer.  
At the same time, I'm not sure if we should ever really give up as  
long as upper layers keep sending traffic: connectivity can come back  
at any time.

>> We need to think about the situation where a fast primary link  
>> fails and we switch to a slow backup, though. Presumably, we'll  
>> want to switch back to the fast primary address pair when possible.

> Right, the same arguments may apply for other reasons like cost of  
> the link, security/privacy features of the path... But the question  
> is how does the shim is aware of such information? i guess that  
> these are policy issues and we could include some means to express  
> this type of considerations in the shim

Right. For destination addresses and to some degree, source  
addresses, this can work through the RFC 3484 policy table, but it  
gets more difficult when we have to select an output interface or ISP/ 
site exit, possibly though the source address in the packet.

Delivery-date: Fri, 19 Aug 2005 13:04:53 +0000
Date: Fri, 19 Aug 2005 06:05:30 -0700
From: Erik Nordmark <[email-masked]>
To: Geoff Huston <[email-masked]>
CC: Brian E Carpenter <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: shim-aware transports

Geoff Huston wrote:

> Placing the potential functionality in the hands of the session layer 
> also infers that the shim state may not necessarily be 
> endpoint-to-endpoint, but may operate at a finer level of selection (the 
> 'equivalence" topic I referred to at the WG meeting). It also infers 
> that an individual session may want to trigger a locator change that 
> appears to be an isolated request - in which case there appears to be a 
> logical need for dynamic shim state forking in some form (i.e. a shim 
> state may state in a general host-to-host state, but individual sessions 
> may 'fork' out of this general state by requesting individual locator 
> change trigger settings, for example.)

Geoff,

This notion of forking makes sense, but it isn't clear to me whether it 
is a local notion at the sending side, with the ULP at each end being 
able to make independent decisions (perhaps using some ULP signaling), 
or whether it is a shim-coordinated decision.

I think we can define it to be a local thing at the sending side, where 
a ULP can signal that a given class of packets between a ULP pair (or 
even at the granularity of each packet) use a different locator pair 
than the default. The receiver will happily accept such packets, because 
the locator pair is in the negotiated set of locators for the ULP pair 
context.

Do you think we'd loose something important if there isn't shim 
signaling to have the two ends agree to switch the locator pair for both 
directions?

    Erik

Delivery-date: Fri, 19 Aug 2005 12:53:47 +0000
Date: Fri, 19 Aug 2005 05:54:14 -0700
From: Erik Nordmark <[email-masked]>
To: Pekka Nikander <[email-masked]>
CC: shim6 <[email-masked]>, Christian Huitema <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, [email-masked]
Subject: Re: Thoughts about layering multi-addressing

Pekka Nikander wrote:

> Hence, my question is if the transport are likely to need anything  else 
> than the following:
>   - both ULIDs and locators in incoming packets
>   - ability to define locators for outgoing packets
>     - what happens if the locators are not available any
>       more?  Host-internal ICMP-like response?
>   - ability to get the full set of locators related
>     to a given ULID
> Anything else?

One additional thing comes to mind. If we are thinking a bit about 
multihomed hosts and not just multihomed sites, it might be that the 
locator pair is really part of an abstraction of the (host visible) 
path; another element of this would be the interface used to send and 
receive packets. Thus carrying <src locator, dst locator, interface id> 
as the annotation might make sense.

When it comes to the available locator pairs (or host-visible path 
abstractions), then I think we'd want the ability for the ULP to request 
notifications not only of the current set, but also of any changes to 
the set. (A locally generated ICMP error might be an implementation of 
this, but I find that missing in its inability to indicate new locators 
to the ULP.)

The thing that is the least clear to me if the role/split between the 
shim and the locator aware transport when it comes to the testing of the 
paths. It might be that the shim has some information about the state of 
the paths (which are known to have recently worked and not), which it 
can share with the ULP. But the ULP might have more data on the paths it 
is currently using (rtt, cwnd) which it may or may not make sense to 
share with other ULPs or instances of the same ULP.
How much do we need in this space? A full fledged mechanisms for sharing 
akin to the congestion manager (what Dave Crocker called CELP)? Or do we 
assume it is a unidirectional mechanism for the shim  to tell the ULP 
the status?

    Erik

Delivery-date: Fri, 19 Aug 2005 11:42:10 +0000
Cc: Geoff Huston <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
From: Pekka Nikander <[email-masked]>
Subject: Re: shim-aware transports
Date: Fri, 19 Aug 2005 13:41:54 +0200
To: Brian E Carpenter <[email-masked]>

Brian,

I have a feeling we have some sort of a communication problem here  
but that we fundamentally agree; this message is an attempt to  
clarify what I am trying to say.

I am all for the initial goal of keeping the ULPs unchanged.  That's  
the whole point.  However, when doing so I also think that we  
*probably* should adopt the conservative approach adopted by e.g.  
SCTP, i.e., to have one preferred locator and to use the rest as  
equal backups.  Two reasons: keep it simple, and less surprises to  
the transport algorithms.

Now, AFAICT, the discussion below cornered around what to do next,  
after the initial goal has been reached.  What I remember from Paris  
we all seem to agree that we need some kind of an interface between  
the shim layer and ULP.

My two messages (URLs below) define a strawman for such an interface,  
while trying to keep the system at the same time as simple as  
possible.  I don't see that changing at all the way the shim works  
with unmodified ULPs, or what state it maintains.

On the other hand, what I find somewhat worrisome are allusions that  
we should make the state at the shim layer more complex as we add  
multi-addressing support to ULPs.  It looks to me that some folks  
would like to do locator-policy decisions at multi-addressing  
transports, and then push those policies somehow to the shim, to be  
executed there.  I find that unnecessarily complex.  Why can't the  
ULP just pick the locators, if it knows about them; in that case the  
shim just protects the transport from using locators that are no  
longer valid.

So, all I am saying is KISS, and so far the method Christian and I  
deviced

Going a little bit deeper,

On Aug 19, 2005, at 12:55, Brian E Carpenter wrote:

> Pekka, since the initial goal is no change to the ULPs,
> the state is all going to be in the shim, isn't it?
> (Or perhaps better to say, in a data structure initially
> maintained exclusively by the shim, but perhaps accessed
> by shim6-aware ULPs in a later stage.)
>
>    Brian
>
> Pekka Nikander wrote:
>
>> I am just wondering how much state we want to push to the shim  
>> layer  versus keep at the     ULPs.  See my other notes about that:
>> http://psg.com/lists/shim6/msg00663.html
>> http://psg.com/lists/shim6/msg00671.html
>> It looks to me that we may be able to keep the signalling  
>> interface  pretty simple, without any loss of generality.  But  
>> maybe I am  missing something there.
>> --Pekka Nikander
>> On Aug 19, 2005, at 1:53, Geoff Huston wrote:
>>
>>> Brian noted that  making the shim  work with unmodified IP and   
>>> unmodified upper layers needs to get shipped first, and thats a   
>>> good priority.
>>>
>>> Brian also noted that this implies a two-way API between the  
>>> shim  and the ULP, in which the necessary events are transmitted  
>>> up or down.
>>>
>>> So I'd like to float the concept of a placeholder about the   
>>> interaction with transport that hopefully will allow the  
>>> priority  work to proceed while still keeping enough flexibility  
>>> around to  allow transport sessions to be able to be more  
>>> explicit about what  they want from the shim layer, but to define  
>>> this interaction at a  later stage in the WG's life.
>>>
>>> It seems to me that the basic mechanics of this can be satisfied  
>>> by  defining a rich enough signalling interface between the shim  
>>> layer  and the upper level sessions - perhaps allowing for the  
>>> inclusion  of the relevant locator set, and allowing attributes  
>>> to be placed  on individual locators as well as on the complete  
>>> set in either  direction. In other words this allows an upper  
>>> level to be more  specific about locator preference either by  
>>> attribute or by  explicit choice by passing a rich information  
>>> bundle to the ULP via  this signalling interface.
>>>
>>> Placing the potential functionality in the hands of the session   
>>> layer also infers that the shim state may not necessarily be   
>>> endpoint-to-endpoint, but may operate at a finer level of  
>>> selection  (the 'equivalence" topic I referred to at the WG  
>>> meeting). It also  infers that an individual session may want to  
>>> trigger a locator  change that appears to be an isolated request  
>>> - in which case there  appears to be a logical need for dynamic  
>>> shim state forking in some  form (i.e. a shim state may state in  
>>> a general host-to-host state,  but individual sessions may 'fork'  
>>> out of this general state by  requesting individual locator  
>>> change trigger settings, for example.)
>>>
>>> If you allow vertical (and peer-to-err horizontal) locator   
>>> attribute signalling, and also dynamic shim state forking you  
>>> can  then say  that the mechanisms for the ULP to signal quite  
>>> explicit  hints to the shim layer as to when and what locator to  
>>> use, and for  the shim to respond with a forked state as required  
>>> and signal this  to the peer remote shim entity is within the  
>>> capability of the  protocol. How the fields are defined and used  
>>> are up to a later  incarnation of the or another WG, that will  
>>> define the attributes  and  their semantics.
>>>
>>> Does this help?
>>>
>>>   Geoff
>>>
>>>
>>>
>>>
>>>
>>>
>
>
>

Delivery-date: Fri, 19 Aug 2005 10:57:08 +0000
Date: Fri, 19 Aug 2005 12:55:12 +0200
From: Brian E Carpenter <[email-masked]>
Organization: IBM
To: Pekka Nikander <[email-masked]>
CC: Geoff Huston <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: shim-aware transports

Pekka, since the initial goal is no change to the ULPs,
the state is all going to be in the shim, isn't it?
(Or perhaps better to say, in a data structure initially
maintained exclusively by the shim, but perhaps accessed
by shim6-aware ULPs in a later stage.)

    Brian

Pekka Nikander wrote:
> I am just wondering how much state we want to push to the shim layer  
> versus keep at the     ULPs.  See my other notes about that:
> 
> http://psg.com/lists/shim6/msg00663.html
> http://psg.com/lists/shim6/msg00671.html
> 
> It looks to me that we may be able to keep the signalling interface  
> pretty simple, without any loss of generality.  But maybe I am  missing 
> something there.
> 
> --Pekka Nikander
> 
> On Aug 19, 2005, at 1:53, Geoff Huston wrote:
> 
>> Brian noted that  making the shim  work with unmodified IP and  
>> unmodified upper layers needs to get shipped first, and thats a  good 
>> priority.
>>
>> Brian also noted that this implies a two-way API between the shim  and 
>> the ULP, in which the necessary events are transmitted up or down.
>>
>> So I'd like to float the concept of a placeholder about the  
>> interaction with transport that hopefully will allow the priority  
>> work to proceed while still keeping enough flexibility around to  
>> allow transport sessions to be able to be more explicit about what  
>> they want from the shim layer, but to define this interaction at a  
>> later stage in the WG's life.
>>
>> It seems to me that the basic mechanics of this can be satisfied by  
>> defining a rich enough signalling interface between the shim layer  
>> and the upper level sessions - perhaps allowing for the inclusion  of 
>> the relevant locator set, and allowing attributes to be placed  on 
>> individual locators as well as on the complete set in either  
>> direction. In other words this allows an upper level to be more  
>> specific about locator preference either by attribute or by  explicit 
>> choice by passing a rich information bundle to the ULP via  this 
>> signalling interface.
>>
>> Placing the potential functionality in the hands of the session  layer 
>> also infers that the shim state may not necessarily be  
>> endpoint-to-endpoint, but may operate at a finer level of selection  
>> (the 'equivalence" topic I referred to at the WG meeting). It also  
>> infers that an individual session may want to trigger a locator  
>> change that appears to be an isolated request - in which case there  
>> appears to be a logical need for dynamic shim state forking in some  
>> form (i.e. a shim state may state in a general host-to-host state,  
>> but individual sessions may 'fork' out of this general state by  
>> requesting individual locator change trigger settings, for example.)
>>
>> If you allow vertical (and peer-to-err horizontal) locator  attribute 
>> signalling, and also dynamic shim state forking you can  then say  
>> that the mechanisms for the ULP to signal quite explicit  hints to the 
>> shim layer as to when and what locator to use, and for  the shim to 
>> respond with a forked state as required and signal this  to the peer 
>> remote shim entity is within the capability of the  protocol. How the 
>> fields are defined and used are up to a later  incarnation of the or 
>> another WG, that will define the attributes  and  their semantics.
>>
>> Does this help?
>>
>>   Geoff
>>
>>
>>
>>
>>
> 
> 

Delivery-date: Fri, 19 Aug 2005 08:35:47 +0000
Cc: Brian E Carpenter <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
From: Pekka Nikander <[email-masked]>
Subject: Re: shim-aware transports
Date: Fri, 19 Aug 2005 10:35:40 +0200
To: Geoff Huston <[email-masked]>

I am just wondering how much state we want to push to the shim layer  
versus keep at the     ULPs.  See my other notes about that:

http://psg.com/lists/shim6/msg00663.html
http://psg.com/lists/shim6/msg00671.html

It looks to me that we may be able to keep the signalling interface  
pretty simple, without any loss of generality.  But maybe I am  
missing something there.

--Pekka Nikander

On Aug 19, 2005, at 1:53, Geoff Huston wrote:

> Brian noted that  making the shim  work with unmodified IP and  
> unmodified upper layers needs to get shipped first, and thats a  
> good priority.
>
> Brian also noted that this implies a two-way API between the shim  
> and the ULP, in which the necessary events are transmitted up or down.
>
> So I'd like to float the concept of a placeholder about the  
> interaction with transport that hopefully will allow the priority  
> work to proceed while still keeping enough flexibility around to  
> allow transport sessions to be able to be more explicit about what  
> they want from the shim layer, but to define this interaction at a  
> later stage in the WG's life.
>
> It seems to me that the basic mechanics of this can be satisfied by  
> defining a rich enough signalling interface between the shim layer  
> and the upper level sessions - perhaps allowing for the inclusion  
> of the relevant locator set, and allowing attributes to be placed  
> on individual locators as well as on the complete set in either  
> direction. In other words this allows an upper level to be more  
> specific about locator preference either by attribute or by  
> explicit choice by passing a rich information bundle to the ULP via  
> this signalling interface.
>
> Placing the potential functionality in the hands of the session  
> layer also infers that the shim state may not necessarily be  
> endpoint-to-endpoint, but may operate at a finer level of selection  
> (the 'equivalence" topic I referred to at the WG meeting). It also  
> infers that an individual session may want to trigger a locator  
> change that appears to be an isolated request - in which case there  
> appears to be a logical need for dynamic shim state forking in some  
> form (i.e. a shim state may state in a general host-to-host state,  
> but individual sessions may 'fork' out of this general state by  
> requesting individual locator change trigger settings, for example.)
>
> If you allow vertical (and peer-to-err horizontal) locator  
> attribute signalling, and also dynamic shim state forking you can  
> then say  that the mechanisms for the ULP to signal quite explicit  
> hints to the shim layer as to when and what locator to use, and for  
> the shim to respond with a forked state as required and signal this  
> to the peer remote shim entity is within the capability of the  
> protocol. How the fields are defined and used are up to a later  
> incarnation of the or another WG, that will define the attributes  
> and  their semantics.
>
> Does this help?
>
>   Geoff
>
>
>
>
>

Delivery-date: Fri, 19 Aug 2005 08:30:58 +0000
Cc: shim6 <[email-masked]>
From: Pekka Nikander <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Fri, 19 Aug 2005 10:30:30 +0200
To: marcelo bagnulo braun <[email-masked]>

> In any of these approaches, the locator annotated for outgoing  
> packets are a suggestion of they must be honored? i mean, i guess  
> this is related with how does the shim informs the ULP that a given  
> locator is no longer reachable, right?

I don't know what would be the right approach.  Currently I think  
that the best approach might be to signal the ULP that the shim layer  
was unable to send the packet due to the suggested locator no more  
being associated with the ULID.  At that point the ULP could ask the  
shim for the new set of active locators, and pick a suitable one from  
there.

The alternative would be the shim layer to pick the locator, in the  
same way as it would pick if there was none given by the ULP.  The  
drawback of this approach is that the ULP has no information about  
the locator having been overwritten...

>> Hence, my question is if the transport are likely to need anything  
>> else than the following:
>>   - both ULIDs and locators in incoming packets
>>   - ability to define locators for outgoing packets
>>     - what happens if the locators are not available any
>>       more?  Host-internal ICMP-like response?
>
> coming back to this point.. by locators you mean locator pair,  
> right? i mean the shim should inform the ULP that a given locator  
> pair is no longer reachable, right?

As I indicated above, I am no that far yet in my thinking.  So, the  
brief answer is that I don't know.

>> Anything else?
>
> I wonder if it wouldn't be useful to allow the ULP to inform the  
> shim about some sort of preferences w.r.t. which alternative  
> locator pair to try with in case of an outage.

Hmm.  Let's consider the complexity implications.  If the shim just  
fails and signals the ULP about the failure, the ULP will have all  
the necessary information about picking a locator to use.  It can  
easily update its preferences dynamically based on multiple source of  
input, e.g., based on flows on different locators.  That would keep  
the shim simple, pushing the complexity to the transport.

On the other hand, if the shim has more information, such as  
preferences per transport, that brings more complexity to the shim  
layer.  Conversely, if we accept the architectural principle that I  
suggested, we shouldn't pick this way, as in this case the shim knows  
more than is absolutely necessary.

I am currently tempted to try to keep the shim as simple as  
possible.  Due to having transports that won't know about locators,  
we clearly need to have some sort of preferred locator at the shim,  
and associated fail over mechanisms.  However, to me it "smells"  
better if we would not use them, at least by default, with more  
intelligent transports that supposedly know better.

But there is one point here:  As we apparently need to have a  
preferred locator at the shim, we should allow the transports to  
learn about that, too.  Apparently we also need some administrative  
interface for changing the preferred locator dynamically....

--Pekka

Delivery-date: Fri, 19 Aug 2005 00:07:43 +0000
Date: Fri, 19 Aug 2005 10:07:12 +1000
To: <[email-masked]>, <[email-masked]>, <[email-masked]>
From: Geoff Huston <[email-masked]>
Subject: RE: failure detection

At 09:36 PM 15/08/2005, [email-masked] wrote:
>Iljitsch,
>
>
> >Somewhat to my disappointment, there were no opinions about
> >which of the two failure detection mechanisms is better,
> >either during the sessions or afterward on the list.
>
>At least for me, I'm still puzzeling over how to capture failure ...
>Between 2 hosts, tcp traffic may work but not udp traffic (mostly due
>to stupid middleboxes). Is this a failure from the shim point of view?

Two potential cases:

a) the SHIM layer is conduction its own probe / heartbeat on the current 
locator pair and there is no vertical signalling taking place. The locator 
change will be triggered by a fail in the heartbeat
(the heartbeat may be either explicit, or may be derived from existing 
traffic, or a heatbeat may kick in in the absence of existing traffic)

b) the SHIM layer is responsive to ULP signals. If the UDP session signals 
a locator failure then the Shim layer would undertake a locator change (** 
if you want to include dynamic shim sate forking in this model it may well 
be that the shim state would fork at this point, and the forked state that 
retained association with the UDP traffic may undertake the locator switch)

>Also, different transports have different concepts of what 'failure'
>is. When is a path considered failed? Classical TCP failure (retransmission
>of the the same packet x times)?  Too high bit-error rate?  I think we might
>run into circumstances where if we let the shim layer decide conclusively
>that a path has failed, it might decide much later that the path is bad
>than what the transport layer knows.

Again this depends on the model:

In model a)  (no vertical signalling) there is no ULP information to use - 
in which case the SHIM trigger needs to be based on shim state information, 
derived from a heatbeat. i.e. failure is heartbeat failure

In model b) the shim layer should not care what conditions exist to cause 
the ULP to think that a change is appropriate - the ULP makes the call and 
then generates the signal to the shim layer.

>I think we should avoid any transport layer functions in the shim layer
>(or at least as much as possible).  I'm leaning more towards the rich-api
>type of functionality, where the shim provides as much info as it can
>to the transport layers, and let the transport layers make decisions.

precisely.

>I'm also wondering if the failure detection mechanism should be a pluggable
>option; I've been thinking how the shim layer would work in wireless 
>environments,
>and I don't think I'd like to have fast heartbeating, as that would drain
>batteries unnesscesarily.  Could it be possible to have this
>functionality as optional?

I would think so - i.e. in a rich API with forkable shim states a ULP 
should be able to say to the SHIM state: "don't heatbeat for me - I'll 
generate explicit triggers"

    Geoff

Delivery-date: Thu, 18 Aug 2005 23:54:19 +0000
Date: Fri, 19 Aug 2005 09:53:14 +1000
To: Brian E Carpenter <[email-masked]>, Iljitsch van Beijnum <[email-masked]>
From: Geoff Huston <[email-masked]>
Subject: Re: shim-aware transports
Cc: shim6 <[email-masked]>

Brian noted that  making the shim  work with unmodified IP and unmodified 
upper layers needs to get shipped first, and thats a good priority.

Brian also noted that this implies a two-way API between the shim and the 
ULP, in which the necessary events are transmitted up or down.

So I'd like to float the concept of a placeholder about the interaction 
with transport that hopefully will allow the priority work to proceed while 
still keeping enough flexibility around to allow transport sessions to be 
able to be more explicit about what they want from the shim layer, but to 
define this interaction at a later stage in the WG's life.

It seems to me that the basic mechanics of this can be satisfied by 
defining a rich enough signalling interface between the shim layer and the 
upper level sessions - perhaps allowing for the inclusion of the relevant 
locator set, and allowing attributes to be placed on individual locators as 
well as on the complete set in either direction. In other words this allows 
an upper level to be more specific about locator preference either by 
attribute or by explicit choice by passing a rich information bundle to the 
ULP via this signalling interface.

Placing the potential functionality in the hands of the session layer also 
infers that the shim state may not necessarily be endpoint-to-endpoint, but 
may operate at a finer level of selection (the 'equivalence" topic I 
referred to at the WG meeting). It also infers that an individual session 
may want to trigger a locator change that appears to be an isolated request 
- in which case there appears to be a logical need for dynamic shim state 
forking in some form (i.e. a shim state may state in a general host-to-host 
state, but individual sessions may 'fork' out of this general state by 
requesting individual locator change trigger settings, for example.)

If you allow vertical (and peer-to-err horizontal) locator attribute 
signalling, and also dynamic shim state forking you can then say  that the 
mechanisms for the ULP to signal quite explicit hints to the shim layer as 
to when and what locator to use, and for the shim to respond with a forked 
state as required and signal this to the peer remote shim entity is within 
the capability of the protocol. How the fields are defined and used are up 
to a later incarnation of the or another WG, that will define the 
attributes and  their semantics.

Does this help?

   Geoff

Delivery-date: Thu, 18 Aug 2005 19:46:50 +0000
Date: Thu, 18 Aug 2005 20:46:27 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: David Meyer <[email-masked]>
cc: [email-masked]
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Thu, 18 Aug 2005, Paul Jakma wrote:

> I'm presuming shim6 is to allow a 'split' mode of operation, where 
> end-systems on a 'shimmed' network don't know of shimming, but the 
> shimming is instead on a subset of hosts (at the edges).

Ah, well, given the following text in Geoff's rather nice 
architecture draft:

"Shim6 operates as a per-host header address mapping function."

I might be presuming incorrectly. Though, I don't /yet/ see any 
intrinsic reason why a split mode would not be possible..

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
A dead atheist is someone who's all dressed up with no place to go.

Delivery-date: Thu, 18 Aug 2005 19:30:47 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Thoughts about layering multi-addressing 
Date: Thu, 18 Aug 2005 21:30:54 +0200
To: Pekka Nikander <[email-masked]>

Hi Pekka,

thanks for describing this so clearly... some comments/questions=20
below...

El 18/08/2005, a las 17:03, Pekka Nikander escribi=F3:
>
> Approach one:  Annotate ULIDs with locators
>
>                  ^  v
>                  |  |  ULIDs (annotated with locators)
>              +----------+
>              |  shim6   |
>              +----------+
>                  |  |
>                  ^  v  Locators (and only locators)
>
> In outgoing packets, the optional locator annotation could indicate=20
> the locators that the transport prefers to use, if they are available=20=

> according to the shim6 knowledge.  In incoming packets, the locator=20
> annotation indicates the locators as found in the incoming packets. =20=

> ULIDs that don't understand the annotations continue to function=20
> according to the current "theory".
>
> Approach two: Send locators up but annotate them with ULIDs
>
>                  ^  v
>                  |  |  Locators (annotated with ULIDs)
>              +----------+
>              |  shim6   |
>              +----------+
>                  |  |
>                  ^  v  Locators (and only locators)
>
> While the information content here is the same as in the previous=20
> case, the difference lies in what transports see by default.  This=20
> might work better with unmodified or minimally modified multi-homing=20=

> aware transports, such as SCTP and DCCP.
>

In any of these approaches, the locator annotated for outgoing packets=20=

are a suggestion of they must be honored? i mean, i guess this is=20
related with how does the shim informs the ULP that a given locator is=20=

no longer reachable, right?

> The architecturally guiding principle in this is to keep IP only aware=20=

> of the plain set of locators associated with each ULID, the required=20=

> encapsulation/marking information (such as the flow label), and=20
> nothing else.   Annotating the ULIDs with locators (or vice versa)=20
> allows more capable transport protocols to keep track of them and=20
> associate path-related parameters such as RTT, bandwidth estimate,=20
> etc, separately with each locator pair.  This allows the=20
> transport-related functionality to remain located at the transport=20
> layer.  At the same time, any changes in the locator set need to be=20
> secured only at the shim layer, i.e. only once per each ULID pair=20
> rather than separately for each transport connection.
>
> In addition to this basic mechanism, some transport may need to figure=20=

> out the complete set of locators available.  However, are there any=20
> other functions that would be needed?  For example, if the transport=20=

> can give outgoing locators as hints to the shim-layer, it does not=20
> need to tell the shim-layer to deprecate any locators even though it=20=

> suspects that some of they may not work any more.  Indeed, telling=20
> shim to prefer some over others may be dangerous since different=20
> locator pairs may work for different transports, due to the Internet=20=

> not being fully transparent any more.
>
> Hence, my question is if the transport are likely to need anything=20
> else than the following:
>   - both ULIDs and locators in incoming packets
>   - ability to define locators for outgoing packets
>     - what happens if the locators are not available any
>       more?  Host-internal ICMP-like response?

coming back to this point.. by locators you mean locator pair, right? i=20=

mean the shim should inform the ULP that a given locator pair is no=20
longer reachable, right?

>   - ability to get the full set of locators related
>     to a given ULID
> Anything else?
>

I wonder if it wouldn't be useful to allow the ULP to inform the shim=20
about some sort of preferences w.r.t. which alternative locator pair to=20=

try with in case of an outage.

I mean, the question is if it wouldn't be interesting to let the ULP to=20=

inform about which locator to try if an outage occur. This may make=20
some sense in the case that the different access links have different=20
characteristics and that each app has a strong preference on which=20
links to use.
A somehow related aspect was brought up the other day by Iljitsch,=20
w.r.t. to returning to the initial locator once that the failure has=20
been repaired.

Regards, marcelo

> One other practical problem here is to figure out how to make this all=20=

> work with high speed servers, which are likely to have IP and TCP=20
> implemented with firmware at the NIC.  (Kudos to Christian for=20
> pointing this out.)
>
> --Pekka Nikander
>
>
>

Delivery-date: Thu, 18 Aug 2005 15:13:39 +0000
Date: Thu, 18 Aug 2005 16:13:23 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: David Meyer <[email-masked]>
cc: [email-masked]
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Thu, 18 Aug 2005, David Meyer wrote:

> 	Are you saying that you believe it is scalable for every
> 	end-system in the Internet (or even every enterprise) to
> 	have some kind of quasi-real-time access to the
> 	equivalent of the DFZ RIB somewhere so that they (the
> 	end-systems/hosts) can make decisions about which locator
> 	sets to use?

No, I said it's scaleable for enterprise /sites/.

I'm presuming shim6 is to allow a 'split' mode of operation, where 
end-systems on a 'shimmed' network don't know of shimming, but the 
shimming is instead on a subset of hosts (at the edges).

I don't know whether that presumption is valid.

> 	Further, are you saying that you believe it is scalable
> 	for every end-system in the Internet to *require* such
> 	information to support site multi-homing?

I didn't and would not say that.

> 	And BTW, if so,
> 	how do I communicate this information to my transport
> 	stack?

Typically, one avails of routing information by not /requiring/ IP 
protocols to be involved in the details of routing. Ie by layering 
things correctly.

That's my prime reason for arguing against involving shim6 
preemptively in the business of which local locator/source address to 
use in packets it outputs.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
It'll be just like Beggars' Canyon back home.
 		-- Luke Skywalker

Delivery-date: Thu, 18 Aug 2005 15:07:40 +0000
Date: Thu, 18 Aug 2005 08:07:04 -0700
From: David Meyer <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: failure detection

--/04w6evG8XlLl3ft
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

	Paul,

> I'm saying you can still get a "read-only" routing feed (BGP or=20
> whatever), purely for informational purposes, to help decide which of=20
> your ISPs has the best path.
>
> That's entirely scaleable and well within reason for deployment at=20
> 'enterprise' shim6 sites.

	Are you saying that you believe it is scalable for every
	end-system in the Internet (or even every enterprise) to
	have some kind of quasi-real-time access to the
	equivalent of the DFZ RIB somewhere so that they (the
	end-systems/hosts) can make decisions about which locator
	sets to use?  =20

	Further, are you saying that you believe it is scalable
	for every end-system in the Internet to *require* such
	information to support site multi-homing? And BTW, if so,
	how do I communicate this information to my transport
	stack?=20

	Dave

--/04w6evG8XlLl3ft
Content-Type: application/pgp-signature
Content-Disposition: inline

--/04w6evG8XlLl3ft--

Delivery-date: Thu, 18 Aug 2005 15:06:31 +0000
Date: Thu, 18 Aug 2005 16:06:11 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Thu, 18 Aug 2005, marcelo bagnulo braun wrote:

> Hi Paul,
>
> My understanding is that you seem to be uncomfortable with the fact that n^2 
> probes may be needed when we try with different source and destination 
> locators, right?

If needed == MUST or SHOULD, then not uncomfortable but vehemently 
opposed. It's not needed at all imho.

My reason for this particularly is because I fear this will risk 
precluding (*better*) mechanisms external to shim6 for detecting 
failures.

> Suppose you have a multihomed site with ISPA and ISPB and that they have 
> assigned PrefA and PrefB respectivelly
>
> Suppose you have Host1 in the multihomed host and that it is communicating 
> with Host2 outside the multihomed site. For that communication, Host1 is 
> using address PrefA:host1 both as locator and as ULID. I assume that host2 
> has a single address host1

Is host2 shim-aware? It must be for scenario in next sentence to 
work. Particularly given RPF filtering at ISPs on "PA" customers.

> Now, suppose that an outage in the link between the multihomed site and ISPA 
> occurs.

Ok.

> Host1 detects and needs to do something about it, how can he try 
> with an alternative path?

Why must host1 detect this? Host2 could also ;).

If, as part of shim6 control, host1 had, /in advance/, informed host2 
that prefB:host1 was /also/ a valid locator, then:

- if host2 was first to detect the problem, it could simply switch to
   using the prefB:host1 locator

- if host1 notices there is a problem with bidirectional
   communication, it can simply start using the PrefB based locator
   (which host2 already knows about and should accept)

This presumes host1 is actively managing its source address according 
to state specific to shim6. However, there is a *better* way:

- dont have shim6 pick the ISP dependent part of the locator
- let the OS's existing SAS mechanisms pick the source address
   according to routing policy, as per normal, for each and every
   packet
- let some external mechanism detect the local failure

Some (non-exhaustive) examples of such mechanisms:

- link status information

(eg PPP has its own heartbeat mechanism. PPP is often used for DSL 
connections, which will be a common case for the scenario described 
above for host1, today at least)

- Routing information

(eg, host1 /could/ get routing information from either or both of 
ISP1 and ISP2. Be it just a RIP default announced regularly on up to 
full *read-only* BGP feeds)

- Link probing / dead-gateway detection software

(At least one OS provides software to actively probe gateways and 
adjust routes as/when required)

- Well-known remote host probing software

And so on.

So, I'm not sure shim6 is at all the best place to take on detection 
of /local/ failures.

Ie, in your scenario above, what I'm saying is:

- Shim6 on host1 doesnt and shouldn't be involved in detecting local
   failures

- Only host2 needs to know and detect whether it must use PrefA or
   PrefB based locator for host1.

Ie, this should all be about reachability of the *remote* locator. 
Which local locator to use should simply *not* be in scope for shim6 
- then things get easier, and you can avail of one or more existing 
mechanisms for detecting local failures.

> Well, it needs to retry using an alternative source address,

Indeed, but as per above, let /something else/ figure out which local 
path (and hence which source) is the best to use.

> so that packets can be routed through ISPB (in the outgoing 
> direction this is due to ingress filtering compatibility and in the 
> incoming direction becuase of the usage of PA addressing)

The keyword here is "routed", of (to paraphrase) the local output to 
take. That implies routing, that implies (to my mind) "Not shim6's 
job".

What shim6 /does/ need to concern itself with is which /remote/ 
locator address it needs to use.

> This implies that when a host within a multihomed site needs to try 
> alternative paths, it needs to use different source addreses, and 
> of course different destiantion addresses in a more general 
> scenario.
>
> This implies that in order to explore all the possible paths, we 
> need to make n^2 probes.

Yes, if you refuse to consider the option of leaving SAS to something 
else (a wealth of mechanisms exist /already/), then you'll become 
convinced you need n^2 probes. :)

> Now, it is important to realize that n^2 is just an upper bound, 
> and that n^2 probes will only be performed when all paths have 
> failed except one and this is the last one you have tried with

Even better is to realise that local address selection simply does 
not need to be something shim6 has to concern itself with. Then the 
number of probes, in the worst case, goes down to n, where n is the 
number of locators you have to hand for the remote shim6.

Don't forget, n^2 is per side. Each side will have to have to do it.
So you're looking at, worst case, for host1 with n locators and host2 
with y locators, by my potentially incorrect algebra:

shim6 doing SAS and probing each side:

 	y^2 + n^2

In the worse case, both y and n tend to infinity, so generally n = y, 
so the worst case therefore is:

 	2*n^2

as n and y -> infinity.

If you figure out that the local source to use is better left to 
other, existing, mechanisms, and only probe the remote side, worst 
case is:

 	n + y

similar to above, this worst case will tend towards:

 	2n

as n->infinity, y->infinity.

O(n^2) scaling is horrid, it causes big performance problems even 
with relatively low orders of N. Be it in run-time or space. (Eg, you 
may have to keep state for those probes. Imagine a shim6 'translator' 
shimming for a big network, or a tiny little embedded consumer 
device, eg a DSL/802.11 gateway device).

I strongly urge n^2 probing to /not/ be considered, to instead 
examine possibility to *not* do SAS in shim6. It just simplifies 
everything, allows *better* mechanisms (like routing, PPP 
link-status, etc.) to do their job, and scales linearly.

> (which may occur very often according to Murphy's law :-)

I live in Ireland, so I run into Murphy a lot ;).

> One of the main concerns of the people designing this mechanisms is 
> how to achieve clever mechanisms to reduce as much as possible the 
> number of probes. The idea is not to send the n^2 probes at once, 
> but to perform some form of exploration phases in which different 
> combinations are tried.

How about considering /not/ doing SAS in shim6 at all? For the 
reasons I describe above? :)

Even simpler.

> sure, n^2 is just the worst case

My initial thoughts, as above, simply is that it is not required. We 
can get linear scaling if we just drop the desire for shim6 to do 
SAS. :)

> no, i am saying that source address selection is influenced by 
> RFC3484 policy table and that this table is the right place to 
> express policy.

Agreed.

> In addition, that SHIM can honor this table as much as possible, so 
> that policy can be expressed when using the shim

Disagree. Shim6 shouldn't know /anything/ about this table. SAS 
shouldn't be its concern in the first place, imho.

> no, shim can try to use first the addresses as expressed in the 
> policy table

That's great, but you have 0 idea whether its the right address.

Determining the right address involves:

- route lookup on the destination to find the local output
- lookup source for this output

Shim6 /could/ do this, but on the other hand it could just leave the 
local address in packets sent down the stack as unspecified, and the 
OS will just do it for shim6.

What is the benefit for shim6 to do this, that it MUST? (It doesn't 
need to. Any such optimisations are purely implementation specific).

Ergo: No need for SAS in shim6.

> No, shim will try to honor the policy table (or any other tool to 
> express policy we need to define) Obviously, if the path preffered 
> by the policy is not available, then shim will have to use others 
> of course

See, there's an existing way to send a packet to a remote host and 
have the source be set appropriately for the preferred path - don't 
specify the source address...

No complications needed.

>> That's entirely scaleable and well within reason for deployment at 
>> 'enterprise' shim6 sites.
>> 
>
> agree, but this is not enough to preserve established communications

Why exactly?

> Are you considering the case that full BGP feed is injected to the 
> hosts, so that the hosts can find out which path is available 
> towards its final destination?

Well, in a BGP fed and shimmed network, I would imagine the shimming 
would be done just on one or two hosts (or routers) at the edge of 
the shimmed network. The BGP feed would be confined to those.

As far as I know, the ULID and shimming host need not be same 
machine, is my understanding correct?

> when i said that is not specific i meant that other protocols may 
> require similar procedures, but it is a key component of the shim 
> protocol, so that is why it needs to be performed.

I dont think it needs to be, imho.

> can you expand on that?

I'll try to write down my thought on a specific mechanism. It's been 
forming as part of these discussions.

> As i pointed out in the example above, different source/destiantion 
> address combinations are required to deal with failures in the 
> edges (i.e. ISP - multihomed site links)

Yep. And how to do that tends to be described in routing tables :) 
(Of which, in many implementations, the source-address-selection 
policy is also described).

I'll try write down my thought on a specific mechanism in a seperate 
mail/doc. Bit busy this week though.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
We don't really understand it, so we'll give it to the programmers.

Delivery-date: Thu, 18 Aug 2005 15:03:43 +0000
Cc: Christian Huitema <[email-masked]>, Iljitsch van Beijnum <[email-masked]>, [email-masked]
From: Pekka Nikander <[email-masked]>
Subject: Thoughts about layering multi-addressing 
Date: Thu, 18 Aug 2005 17:03:06 +0200
To: shim6 <[email-masked]>

[Disclaimer: I am badly behind my reading of shim6 list.  Please keep  
me in CC: in this particular discussion.]

This is related to the AP I got together with some other people, as  
recorded in the minutes:

> Pekka Nikander, John Loughney and Christian Huitema to prepare some  
> initial ideas (slide pack / draft) on a bi-directional SHIM / ULP  
> API that would include the ability for the ULP to obtain/share  
> locator pair path info and then express a locator pair preference  
> to the SHIM (i.e. keep the notion of binding sessions at the  
> transport level but allow ULP signaling to include Loc Pair  
> preferences to be expressed) in addition to sending the SHIM the  
> trigger / heartbeat information associated with the current  
> locator   pair.

According to my own notes, also Iljitch was volunteering to work on  
this.

So, before I/we start writing a draft in this space, I'd like to poll  
the WG opinion on the viability of the basic approach, as outlined  
below.  Note that this is an area where I really think that running  
code or simulations would help.  We also need to get some transport  
area people involved.

Getting to the subject, the current "theory" of shim6 implementation  
seems to be that the shim6 layer rewrites the locators to identifiers  
for incoming packets and vice versa for outgoing packets.  While that  
may be fine in the short run for TCP and UDP, we seem to want a  
different approach for SCTP, probably also for TCP in the longer run,  
and maybe even for UDP.

The current "theory" can be visualised as follows:

                  ^  v
                  |  |  ULIDs (and only ULIDs)
              +----------+
              |  shim6   |
              +----------+
                  |  |
                  ^  v  Locators (and only locators)

There are couple of other approaches, both of which seem to be  
architecturally compatible, AFAICT, but also create the kind of  
enriched interface (not really an API) between the shim6 layer and ULs.

Approach one:  Annotate ULIDs with locators

                  ^  v
                  |  |  ULIDs (annotated with locators)
              +----------+
              |  shim6   |
              +----------+
                  |  |
                  ^  v  Locators (and only locators)

In outgoing packets, the optional locator annotation could indicate  
the locators that the transport prefers to use, if they are available  
according to the shim6 knowledge.  In incoming packets, the locator  
annotation indicates the locators as found in the incoming packets.   
ULIDs that don't understand the annotations continue to function  
according to the current "theory".

Approach two: Send locators up but annotate them with ULIDs

                  ^  v
                  |  |  Locators (annotated with ULIDs)
              +----------+
              |  shim6   |
              +----------+
                  |  |
                  ^  v  Locators (and only locators)

While the information content here is the same as in the previous  
case, the difference lies in what transports see by default.  This  
might work better with unmodified or minimally modified multi-homing  
aware transports, such as SCTP and DCCP.

The architecturally guiding principle in this is to keep IP only  
aware of the plain set of locators associated with each ULID, the  
required encapsulation/marking information (such as the flow label),  
and nothing else.   Annotating the ULIDs with locators (or vice  
versa) allows more capable transport protocols to keep track of them  
and associate path-related parameters such as RTT, bandwidth  
estimate, etc, separately with each locator pair.  This allows the  
transport-related functionality to remain located at the transport  
layer.  At the same time, any changes in the locator set need to be  
secured only at the shim layer, i.e. only once per each ULID pair  
rather than separately for each transport connection.

In addition to this basic mechanism, some transport may need to  
figure out the complete set of locators available.  However, are  
there any other functions that would be needed?  For example, if the  
transport can give outgoing locators as hints to the shim-layer, it  
does not need to tell the shim-layer to deprecate any locators even  
though it suspects that some of they may not work any more.  Indeed,  
telling shim to prefer some over others may be dangerous since  
different locator pairs may work for different transports, due to the  
Internet not being fully transparent any more.

Hence, my question is if the transport are likely to need anything  
else than the following:
   - both ULIDs and locators in incoming packets
   - ability to define locators for outgoing packets
     - what happens if the locators are not available any
       more?  Host-internal ICMP-like response?
   - ability to get the full set of locators related
     to a given ULID
Anything else?

One other practical problem here is to figure out how to make this  
all work with high speed servers, which are likely to have IP and TCP  
implemented with firmware at the NIC.  (Kudos to Christian for  
pointing this out.)

--Pekka Nikander

Delivery-date: Thu, 18 Aug 2005 14:13:54 +0000
Date: Thu, 18 Aug 2005 15:13:26 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Thu, 18 Aug 2005, marcelo bagnulo braun wrote:

> so far this is exactly what is contained in Iljitsch draft, Jari's 
> draft and this is the basic concesous we have so far as i 
> understand it.

Well, unless I'm reading the wrong draft (the one Iljitsch did the 
presentation on at IETF?), my understanding of the draft is that 
it explores the various possible solutions and discusses some the 
considerations, but does not draw a conclusion.

So yes, I'm rehashing things in that draft, but I'm trying to focus 
the discussion by discounting some of the possible solutions (or at 
least relegating them to "implementation discretionary" and 
concentrate on specifics ;).

Is my understanding correct?

> At this point, what we are discussing is how this monitoring 
> mechanism should look like

Ok, good.

> and how this ULP feedback should look like

I would discount this from consideration, other than to ensure any 
mechanism allows an implementation to make use of local positive 
feedback for optimising things.

>> - You can't mandate a shim6 implementation /must/ send probes
>>   regularly, to allow other side to use for reachability detection

> not sure what you mean here....

You can't have one side expect to get regular 'pings' in order to 
determine reachability, otherwise local-positive-feedback to suppress 
probes/keepalives/pings/whatever-you-call-them would not be possible.

> if you are saying that ULP feedback could result in avoiding shim probing, 
> then i agree and this is exactly what we have been cosnidering

Great.

> We have also agreed that failure detection need to be unidirection, 
> the discussion now is whether the proposed mechanisms are actually 
> unidirectional or whether they have an underlying bidirectional 
> assumption

Unidirectional would, I guess, require that all the information 
needed to setup a shim is available in one message.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Faith is under the left nipple.
 		-- Martin Luther

Delivery-date: Thu, 18 Aug 2005 14:01:11 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Thu, 18 Aug 2005 16:01:56 +0200
To: Paul Jakma <[email-masked]>

El 18/08/2005, a las 14:08, Paul Jakma escribi=F3:

> On Thu, 18 Aug 2005, Paul Jakma wrote:
>
>> - reachability detection itself is unidirectional
>>
>>  (iljitsch's draft mentions "unidirectional", i know, but wrt
>>   outside unidirectional issues. I'm proposing the reachability
>>   mechanism itself be unidirectional in nature. Two unidirectional
>>   paths =3D=3D bi-directional :) )
>
> I'd should explain this better:
>
> - the protocol would be to implement "I can hear you", rather than
>   "We can both hear each other" (as, eg, some other path-state
>   protocols do - like OSPF Hello)
>
> - As an optimisation, if the other side asks "Can you hear me?", you
>   can at least infer "I can hear them" - and hence suppress a probe.
>
> Aside: The address tuples used in each direction could be different
>

right see Jari's draft for more details

the difficutly is how to deal with unidirectional paths
but imho the mechanism included in Jari's draft can deal with this.

regards, marcelo

> PS: I know all the reachability/probing/ULP-feedback talk is all still=20=

> at exploratory stages. Hope I've not been /too/ presumptive in=20
> attacking ULP-feedback and n^2 probing (ie, attacking these when=20
> they're only being /considered/, rather than decided on. Just trying=20=

> to nip these two particular ideas in the bud - at least for anything=20=

> other than appendix 'implemention note' type material).
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> I can't live the button-down life like you.  I want it all!  The
> terrifying lows, the dizzying highs, the creamy middles!
>
> 		-- Homer Simpson
> 		   Lisa's Rival
>

Delivery-date: Thu, 18 Aug 2005 13:59:47 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Thu, 18 Aug 2005 16:00:30 +0200
To: Paul Jakma <[email-masked]>

Hi Paul,

My understanding is that you seem to be uncomfortable with the fact=20
that n^2 probes may be needed when we try with different source and=20
destination locators, right?

Ok, i am uncomfortable too with this, but i fail to see any other=20
option for dealing with this.
Let me explain why:

Suppose you have a multihomed site with ISPA and ISPB and that they=20
have assigned PrefA and PrefB respectivelly

Suppose you have Host1 in the multihomed host and that it is=20
communicating with Host2 outside the multihomed site. For that=20
communication, Host1 is using address PrefA:host1 both as locator and=20
as ULID. I assume that host2 has a single address host1

Now, suppose that an outage in the link between the multihomed site and=20=

ISPA occurs.
Host1 detects and needs to do something about it, how can he try with=20
an alternative path? Well, it needs to retry using an alternative=20
source address, so that packets can be routed through ISPB (in the=20
outgoing direction this is due to ingress filtering compatibility and=20
in the incoming direction becuase of the usage of PA addressing)

This implies that when a host within a multihomed site needs to try=20
alternative paths, it needs to use different source addreses, and of=20
course different destiantion addresses in a more general scenario.

This implies that in order to explore all the possible paths, we need=20
to make n^2 probes.

Now, it is important to realize that n^2 is just an upper bound, and=20
that n^2 probes will only be performed when all paths have failed=20
except one and this is the last one you have tried with (which may=20
occur very often according to Murphy's law :-)

One of the main concerns of the people designing this mechanisms is how=20=

to achieve clever mechanisms to reduce as much as possible the number=20
of probes. The idea is not to send the n^2 probes at once, but to=20
perform some form of exploration phases in which different combinations=20=

are tried.

Ok, i will move on to some comments you have below...

El 18/08/2005, a las 12:14, Paul Jakma escribi=F3:
>
>
>>> 1. Why this is a compelling argument given that it's been possible=20=

>>> to publish multiple addresses in DNS for a long long time, yet there=20=

>>> has been 0 demand for either applications to implement n^2=20
>>> path-probing of each local address to every remote address, or for=20=

>>> OSes to implement some kind of 'path-probe' shim to provide such=20
>>> functionality for all applications?
>>
>> I am afraid you are missing our goal here. this is not a matter of=20
>> oportunity but the way we can preserve established communication=20
>> through outages. see above
>
> But you can preserve comms without n^2 probing.
>

sure, n^2 is just the worst case

>> With the shim, path are closely related to addresses used, in=20
>> particular exit paths of the multihomed site are related to source=20
>> addresses used.
>
> Agreed.
>
>> So in order to provide this type of features, source address=20
>> selection has to be influenced, for instance using RFC 3484 policy=20
>> table
>
> So in order to determine the source, you're saying the table which=20
> affects source selection has to be influenced by shim6.

no, i am saying that source address selection is influenced by RFC3484=20=

policy table and that this table is the right place to express policy.
In addition, that SHIM can honor this table as much as possible, so=20
that policy can be expressed when using the shim

>  Surely that's like saying "shim6 will use whatever source pleases it"=20=

> (ie simply ignoring SAS for the final shim6 output packet).
>

no, shim can try to use first the addresses as expressed in the policy=20=

table

>
>>> Ie: The best way to honour local policy is to use INADDR_ANY and let=20=

>>> the OS decide the source address by consulting local routing policy=20=

>>> - alternatively, an administratively specified address. Why exactly=20=

>>> is shim6 so different from everything else on the internet and=20
>>> special that this would not work for it?
>>
>> this is exactly how the shim would support policing see above
>
> No, you said shim6 likely will have to influence SAS. Thats quite=20
> different from not worrying about SAS at all and letting the OS decide=20=

> according to its local policy.
>

No, shim will try to honor the policy table (or any other tool to=20
express policy we need to define) Obviously, if the path preffered by=20
the policy is not available, then shim will have to use others of=20
course

>>> 3. The traditional way on the internet to guard against path=20
>>> failures is to get a routing feed (and no, that does *not* imply you=20=

>>> advertise anything), why is shim6 so special that it can't defer to=20=

>>> existing practice?
>>
>> scalability. traditional IPv4 routing based multihoming lacks of it
>
> Note again "does *not* imply you advertise anything". Everyone is=20
> (sort of) agreed "multihome by advertising your prefix" doesn't scale=20=

> and should not be considered for IPv6 - hence shim6.
>
> I'm saying you can still get a "read-only" routing feed (BGP or=20
> whatever), purely for informational purposes, to help decide which of=20=

> your ISPs has the best path.
>
> That's entirely scaleable and well within reason for deployment at=20
> 'enterprise' shim6 sites.
>

agree, but this is not enough to preserve established communications

Are you considering the case that full BGP feed is injected to the=20
hosts, so that the hosts can find out which path is available towards=20
its final destination?

>>> 4. How will you decide which path is best?
>
>> local policy can be expressed to some degree with the policy table=20
>> defined in RFC 3484. If more fine grained expression is needed (e.g.=20=

>> per app) additional parameters need to be included in the policy=20
>> table
>
> Sorry, my question was more about the metrics you will use to=20
> determine whether path (local A, remote B) is better than (local C,=20
> remote D). Eg, RTT, packet loss, etc.
>

this is an ongoing discussinon in this list right now

>> we seem to be assuming that multihoming support is something useful=20=

>> and that it will be needed in IPv6.
>
> I'd agree with that assumption. :)
>
>> This multihoming support seems to require communications to be=20
>> preserved through outages.
>
> Agreed, but I still don't understand why this requires shim6 to=20
> specify overriding existing SAS policy and do n^2 probing.
>

i addressed this point above

>>> 6. If path-probing really is desired, explain why this is shim6=20
>>> specific?
>>
>> path exploration is a fundamental part of the shim protocol-. maybe=20=

>> is not shim specific and ideas from other similar protocols can be=20
>> used, but it is imho a key part of the shim protocol and need to be=20=

>> part of it.
>
> If it is not specific to shim6, why should it be solved in shim6? (In=20=

> the 'for every possible path of the combinations of local and remote=20=

> addresses')
>

when i said that is not specific i meant that other protocols may=20
require similar procedures, but it is a key component of the shim=20
protocol, so that is why it needs to be performed.

>
...
>>> Note that it's the probing using every single local address which=20
>>> bothers the most. Simple heartbeats and monitoring which set of=20
>>> locators are reachable and just picking one and sticking to it till=20=

>>> you needed to switch, I could agree with.
>>
>> AFAICT this is the approach being considered here or at least one of=20=

>> them.
>
> One of them yes, the n^2 probing is one of the options being=20
> considered - I hope to quash its further development :) - at least in=20=

> terms of something that is mandated officially as part of shim6 specs=20=

> (can still be mentioned in some kind of implementors note).
>
>> I mean, imho, we would only need to perform path exploration after an=20=

>> outage
>
> You can do it more cleverly than that.
>

can you expand on that?

>> I fail to understand what you are missing. Failure detection and Path=20=

>> exploration are key components of the shim, and they are needed to=20
>> preserve established communications through outages.
>
> Sure, I agree. But why not do it in the simplest way possible?
>

and that would be...

>>> on that. It's the n^2 probing I want to ensure is /not/ considered=20=

>>> for inclusion in shim6 RFCs, other than as something mentioned as a=20=

>>> possible implementation detail.
>>
>> Ok, i think i see now. Your problem is with probing with different=20
>> source locators, right?
>
> Yup.
>
>> Well, this is needed because the source address determines the exit=20=

>> path from the multihomed site. I mean, because we are assuming PA=20
>> addressing, changing the source address results in using a different=20=

>> ISP in the multihomed site. That is why different source address need=20=

>> to be explored
>
> Explored and then dismissed as a bad idea, I strongly hope.
>
> The source to use for the shim should be determined by the=20
> /destination/ address in the shim6 packet you receive (eg) first from=20=

> the other side.
>
> Ie, given two shim6 stacks, A and B, that want to talk to each other=20=

> via the global internet (eg to exchange info needed to create shim=20
> mapping between them, A initiates), each with 3 addresses say (A1, A2,=20=

> so on). The communication required is:
>
> A retrieves the required locator information, and gets a list "B1, B2,=20=

> B3". It then sends a packet to each remote address, n packets at a=20
> time, in series, with whatever shim6 control message is required:
>
> A(IN6_ADDRANY) -> B1
> A(IN6_ADDRANY) -> B2
> A(IN6_ADDRANY) -> B3
>
> some time later a reply is received:
>
> A(x)  <-  B3
>
> The source address A should prefer to use (if it must prefer one over=20=

> IN6_ADDRANY) is 'x', as determined by which destination address was in=20=

> the packet of B's that got through.
>
> See, that's simple and works - no probing required.
>

first of all, this is probing since you sent 3 packets. Second, which=20
source addresses has A used for sending these packets?

> If "n^2 probing is simply not an option" is in your mind, then you'll=20=

> start realising there are other, simpler, better ways of achieving the=20=

> same end-goal - which don't require messing with local SAS policy=20
> either, but rather use it as intended (without modification).
>
>>> There's many many years of existing deployment of IP using systems=20=

>>> and applications that simply don't consider such complex probing=20
>>> worth it.
>
>> right, because they are not assuming the usage of multiple PA=20
>> addresses in a single host
>
> That's a fair point, and likely part of the answer.
>
> Another possibility is simply that path failures in the 'middle' of=20
> internet are rare and hence users and apps have not had any compelling=20=

> need to explore this possibility.
>

As i pointed out in the example above, different source/destiantion=20
address combinations are required to deal with failures in the edges=20
(i.e. ISP - multihomed site links)

>
Regards, marcelo

Delivery-date: Thu, 18 Aug 2005 13:59:25 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Thu, 18 Aug 2005 15:59:52 +0200
To: Paul Jakma <[email-masked]>

El 18/08/2005, a las 13:26, Paul Jakma escribi=F3:

> On Thu, 18 Aug 2005, marcelo bagnulo braun wrote:
>
>> There are two possibilities to deal with this, as explained in=20
>> Ilijtsch draft
>
> Yes, ULP feedback and address-pair exploration (ie n^2 probing).
>
> - n^2 probing, I think I've already given my opinion
>
> - ULP feedback, gave some opinion already
>
> I think the latter is actually promising, honest. However I don't=20
> think shim6 should rely on it (seems obvious). Simply because:
>
> - Obviously, there likely are implementations without any feedback
>   mechanisms and we should not preclude them from shim6, or force
>   potentially significant stack reworking to accomodate shim6
>
> - Implementations which do already provide ULP feedback mechanisms
>   might /still/ not (initially or ever) be able to provide this
>   feedback to shim6, as shim6 sits under IP, not (eg) TCP. Even if
>   there is a way for TCP to indicate progress to IP, that information
>   might not be able in shim6 (and there mightn't be a clean way to do
>   it)
>
> Given the ULP points:
>
> - Shim6 specs *should* provide for a shim6 contained reachibility
>   mechanism, to avoid raising bar for implementation
>
> (that seems an uncontraversial point to me)
>
> So what I think we /should/ do is:
>
> - Design some kind of monitoring/reachability protocol inside shim6
>   (as Iljitsch discusses in his draft, but dislikes because of
>   potential overhead) and in its design simply keep a mind to
>   possible optimisations using ULP feedback (eg as NUD does, as you
>   pointed out)
>
> Eg, it should be possible for one side to not have ULP feedback=20
> available and do reachability/monitoring via shim6 while the other=20
> side /does/ have and use ULP feedback. Seems quite possible to me.
>
> This implies:
>
> - ULP feedback should allow sending of probes to be optional
>

so far this is exactly what is contained in Iljitsch draft, Jari's=20
draft and this is the basic concesous we have so far as i understand=20
it.

At this point, what we are discussing is how this monitoring mechanism=20=

should look like and how this ULP feedback should look like

> Hence:
>
> - You can't mandate a shim6 implementation /must/ send probes
>   regularly, to allow other side to use for reachability detection
>

not sure what you mean here....

if you are saying that ULP feedback could result in avoiding shim=20
probing, then i agree and this is exactly what we have been cosnidering

> Hence:
>
> - reachability detection itself is unidirectional
>
>   (iljitsch's draft mentions "unidirectional", i know, but wrt
>    outside unidirectional issues. I'm proposing the reachability
>    mechanism itself be unidirectional in nature. Two unidirectional
>    paths =3D=3D bi-directional :) )
>

We have also agreed that failure detection need to be unidirection, the=20=

discussion now is whether the proposed mechanisms are actually=20
unidirectional or whether they have an underlying bidirectional=20
assumption

Regards, marcelo

> On those principles we can design something, I think. It mightn't be=20=

> far away from NUD either (which I hadn't considered previously, I=20
> admit). Iljitsch's draft reckons this is too much overhead, but I'm=20
> not sure why this should be so.
>
>> not sure what you mean by this...
>
> See above.
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> My doctor told me to stop having intimate dinners for four. Unless=20
> there
> are three other people.
>

Delivery-date: Thu, 18 Aug 2005 12:09:15 +0000
Date: Thu, 18 Aug 2005 13:08:48 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Thu, 18 Aug 2005, Paul Jakma wrote:

> - reachability detection itself is unidirectional
>
>  (iljitsch's draft mentions "unidirectional", i know, but wrt
>   outside unidirectional issues. I'm proposing the reachability
>   mechanism itself be unidirectional in nature. Two unidirectional
>   paths == bi-directional :) )

I'd should explain this better:

- the protocol would be to implement "I can hear you", rather than
   "We can both hear each other" (as, eg, some other path-state
   protocols do - like OSPF Hello)

- As an optimisation, if the other side asks "Can you hear me?", you
   can at least infer "I can hear them" - and hence suppress a probe.

Aside: The address tuples used in each direction could be different

PS: I know all the reachability/probing/ULP-feedback talk is all 
still at exploratory stages. Hope I've not been /too/ presumptive in 
attacking ULP-feedback and n^2 probing (ie, attacking these when 
they're only being /considered/, rather than decided on. Just trying 
to nip these two particular ideas in the bud - at least for anything 
other than appendix 'implemention note' type material).

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
I can't live the button-down life like you.  I want it all!  The
terrifying lows, the dizzying highs, the creamy middles!

 		-- Homer Simpson
 		   Lisa's Rival

Delivery-date: Thu, 18 Aug 2005 11:27:15 +0000
Date: Thu, 18 Aug 2005 12:26:47 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Thu, 18 Aug 2005, marcelo bagnulo braun wrote:

> There are two possibilities to deal with this, as explained in 
> Ilijtsch draft

Yes, ULP feedback and address-pair exploration (ie n^2 probing).

- n^2 probing, I think I've already given my opinion

- ULP feedback, gave some opinion already

I think the latter is actually promising, honest. However I don't 
think shim6 should rely on it (seems obvious). Simply because:

- Obviously, there likely are implementations without any feedback
   mechanisms and we should not preclude them from shim6, or force
   potentially significant stack reworking to accomodate shim6

- Implementations which do already provide ULP feedback mechanisms
   might /still/ not (initially or ever) be able to provide this
   feedback to shim6, as shim6 sits under IP, not (eg) TCP. Even if
   there is a way for TCP to indicate progress to IP, that information
   might not be able in shim6 (and there mightn't be a clean way to do
   it)

Given the ULP points:

- Shim6 specs *should* provide for a shim6 contained reachibility
   mechanism, to avoid raising bar for implementation

(that seems an uncontraversial point to me)

So what I think we /should/ do is:

- Design some kind of monitoring/reachability protocol inside shim6
   (as Iljitsch discusses in his draft, but dislikes because of
   potential overhead) and in its design simply keep a mind to
   possible optimisations using ULP feedback (eg as NUD does, as you
   pointed out)

Eg, it should be possible for one side to not have ULP feedback 
available and do reachability/monitoring via shim6 while the other 
side /does/ have and use ULP feedback. Seems quite possible to me.

This implies:

- ULP feedback should allow sending of probes to be optional

Hence:

- You can't mandate a shim6 implementation /must/ send probes
   regularly, to allow other side to use for reachability detection

Hence:

- reachability detection itself is unidirectional

   (iljitsch's draft mentions "unidirectional", i know, but wrt
    outside unidirectional issues. I'm proposing the reachability
    mechanism itself be unidirectional in nature. Two unidirectional
    paths == bi-directional :) )

On those principles we can design something, I think. It mightn't be 
far away from NUD either (which I hadn't considered previously, I 
admit). Iljitsch's draft reckons this is too much overhead, but I'm 
not sure why this should be so.

> not sure what you mean by this...

See above.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
My doctor told me to stop having intimate dinners for four. Unless there
are three other people.

Delivery-date: Thu, 18 Aug 2005 10:14:54 +0000
Date: Thu, 18 Aug 2005 11:14:09 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

Hi Marcelo,

On Thu, 18 Aug 2005, marcelo bagnulo braun wrote:

> SHIM wg is about providing multihoming support for IPv6. In 
> particular a solution for IPv6 multihoming must be able to preserve 
> communications through outages in the communicating path. Such 
> functionality is provided in IPv4 through BGP features but it 
> requires the injection of site routes in the interdomain routing.

> In IPv6 PA addressing is used, so we need additional mechanisms 
> (the shim) to provide equivalent functionalities, in particular to 
> preserve established communications through outages.

Ok. We're on the same page here I think.

>> 1. Why this is a compelling argument given that it's been possible to 
>> publish multiple addresses in DNS for a long long time, yet there has been 
>> 0 demand for either applications to implement n^2 path-probing of each 
>> local address to every remote address, or for OSes to implement some kind 
>> of 'path-probe' shim to provide such functionality for all applications?
>> 
>
> I am afraid you are missing our goal here. this is not a matter of 
> oportunity but the way we can preserve established communication 
> through outages. see above

But you can preserve comms without n^2 probing.

> With the shim, path are closely related to addresses used, in 
> particular exit paths of the multihomed site are related to source 
> addresses used.

Agreed.

> So in order to provide this type of features, source address 
> selection has to be influenced, for instance using RFC 3484 policy 
> table

So in order to determine the source, you're saying the table which 
affects source selection has to be influenced by shim6. Surely that's 
like saying "shim6 will use whatever source pleases it" (ie simply 
ignoring SAS for the final shim6 output packet).

The SAS aspects of this n^2 probing talk bother me greatly. It will 
ignore local policy, policy which might be there for very good 
reasons, which shim6 has no oversight of.

>> Ie: The best way to honour local policy is to use INADDR_ANY and let the OS 
>> decide the source address by consulting local routing policy - 
>> alternatively, an administratively specified address. Why exactly is shim6 
>> so different from everything else on the internet and special that this 
>> would not work for it?
>> 
>
> this is exactly how the shim would support policing see above

No, you said shim6 likely will have to influence SAS. Thats quite 
different from not worrying about SAS at all and letting the OS 
decide according to its local policy.

>> 3. The traditional way on the internet to guard against path failures is to 
>> get a routing feed (and no, that does *not* imply you advertise anything), 
>> why is shim6 so special that it can't defer to existing practice?
>> 
>
> scalability. traditional IPv4 routing based multihoming lacks of it

Note again "does *not* imply you advertise anything". Everyone is 
(sort of) agreed "multihome by advertising your prefix" doesn't scale 
and should not be considered for IPv6 - hence shim6.

I'm saying you can still get a "read-only" routing feed (BGP or 
whatever), purely for informational purposes, to help decide which of 
your ISPs has the best path.

That's entirely scaleable and well within reason for deployment at 
'enterprise' shim6 sites.

>> 4. How will you decide which path is best?

> local policy can be expressed to some degree with the policy table 
> defined in RFC 3484. If more fine grained expression is needed 
> (e.g. per app) additional parameters need to be included in the 
> policy table

Sorry, my question was more about the metrics you will use to 
determine whether path (local A, remote B) is better than (local C, 
remote D). Eg, RTT, packet loss, etc.

> we seem to be assuming that multihoming support is something useful 
> and that it will be needed in IPv6.

I'd agree with that assumption. :)

> This multihoming support seems to require communications to be 
> preserved through outages.

Agreed, but I still don't understand why this requires shim6 to 
specify overriding existing SAS policy and do n^2 probing.

>> 6. If path-probing really is desired, explain why this is shim6 specific?
>
> path exploration is a fundamental part of the shim protocol-. maybe 
> is not shim specific and ideas from other similar protocols can be 
> used, but it is imho a key part of the shim protocol and need to be 
> part of it.

If it is not specific to shim6, why should it be solved in shim6? (In 
the 'for every possible path of the combinations of local and remote 
addresses')

>> - software that does local path-probing to determine reachability of
>>   locally attached gateways (eg IPMP in Solaris for one)
>
> this seems to be local, while shim is defined e2e

Indeed. But it's already implemented. And local probing (with no SAS 
messing about) likely will do for many cases.

>> - BFD and some other protocols in development
>
> B means bidirectional, and we are not assuming bidirectional paths here

Ah, yes.

>> - software that monitors the systems route-cache and does
>>   path-probing for destinations that currently see flows
>> 
>
>
> not sure what you mean by those but in any case, i am sure we can 
> benefit from these designs as well from the others you emntioned to 
> design the shim path exploration.

Yes, shim6 could benefit from these potential designs by *not* 
getting involved in complex path probing and SAS. ;)

Eg, the local address probing proposed for shim6 is /counter/ 
productive to other possible external mechanisms (worst of all, 
including "get a read-only BGP feed from your ISPs", which is the 
most practical way to do this.)

> If you are familiar with those, i am sure that your knowledge would 
> be very useful to help with the design of the path exploration 
> protocol of the shim

Drop the local source selection from shim6, things become easier, 
shim6 will interoperate better with other routing software, etc.

>> Note that it's the probing using every single local address which bothers 
>> the most. Simple heartbeats and monitoring which set of locators are 
>> reachable and just picking one and sticking to it till you needed to 
>> switch, I could agree with.
>
> AFAICT this is the approach being considered here or at least one 
> of them.

One of them yes, the n^2 probing is one of the options being 
considered - I hope to quash its further development :) - at least 
in terms of something that is mandated officially as part of shim6 
specs (can still be mentioned in some kind of implementors note).

> I mean, imho, we would only need to perform path exploration after 
> an outage

You can do it more cleverly than that.

> I fail to understand what you are missing. Failure detection and 
> Path exploration are key components of the shim, and they are 
> needed to preserve established communications through outages.

Sure, I agree. But why not do it in the simplest way possible?

>> on that. It's the n^2 probing I want to ensure is /not/ considered for 
>> inclusion in shim6 RFCs, other than as something mentioned as a possible 
>> implementation detail.
>> 
>
> Ok, i think i see now. Your problem is with probing with different 
> source locators, right?

Yup.

> Well, this is needed because the source address determines the exit 
> path from the multihomed site. I mean, because we are assuming PA 
> addressing, changing the source address results in using a 
> different ISP in the multihomed site. That is why different source 
> address need to be explored

Explored and then dismissed as a bad idea, I strongly hope.

The source to use for the shim should be determined by the 
/destination/ address in the shim6 packet you receive (eg) first from 
the other side.

Ie, given two shim6 stacks, A and B, that want to talk to each other 
via the global internet (eg to exchange info needed to create shim 
mapping between them, A initiates), each with 3 addresses say (A1, 
A2, so on). The communication required is:

A retrieves the required locator information, and gets a list "B1, 
B2, B3". It then sends a packet to each remote address, n packets at 
a time, in series, with whatever shim6 control message is required:

A(IN6_ADDRANY) -> B1
A(IN6_ADDRANY) -> B2
A(IN6_ADDRANY) -> B3

some time later a reply is received:

A(x)  <-  B3

The source address A should prefer to use (if it must prefer one over 
IN6_ADDRANY) is 'x', as determined by which destination address was 
in the packet of B's that got through.

See, that's simple and works - no probing required.

If "n^2 probing is simply not an option" is in your mind, then you'll 
start realising there are other, simpler, better ways of achieving 
the same end-goal - which don't require messing with local SAS policy 
either, but rather use it as intended (without modification).

>> There's many many years of existing deployment of IP using systems and 
>> applications that simply don't consider such complex probing worth it.

> right, because they are not assuming the usage of multiple PA 
> addresses in a single host

That's a fair point, and likely part of the answer.

Another possibility is simply that path failures in the 'middle' of 
internet are rare and hence users and apps have not had any 
compelling need to explore this possibility.

> When you include multiple PA addresses in hosts within a multihomed 
> site, then you find out that you need to try with different source 
> addresses.

No, you don't. Because you think it's an option (it isn't :) ), 
you've stopped trying to find better options (which there are).

Note that part of my definition of "better option" is one which 
includes "doesn't use n^2 probing and fiddle with SAS", so it may be 
a self-fulfilling definition.

> sort of... you still lacking DoS protection and locator security 
> but kind of what is being considered (with a couple of additional 
> messages)

Yes, no security in that.

> well, yes, but we are considering quite a few optimizations for 
> this, like ULP feedback and traffic monitoring also but yes, This 
> is in the lines of the failure detection mechanism being 
> considered,

Ok.

ULP feedback - don't rely on it :) (I checked, it exists in Solaris 
TCP, for use by NDP, as you pointed out. I can't find anything in 
Linux though).

> path exploration is more complex than that, because you need to 
> change the source address to change the exit ISP. remeber that we 
> are assuming PA addresses, and they are only routed through one of 
> the ISPs of the multihomed site.

I know this fine well, I've had operational experience of PA 
multihoming using tunneling with IPv4. :)

See above, given above, you *do not* need to do anything clever with 
source-address selection at all, imho.

> i think that most of this stuff is included in the current drafts, 
> just that additional complexity is considered, for instance 
> security stuff, unidirectional path support and cosniderations 
> about the constraints imposed by the usage of multiple PA addresses 
> in the multohomed site and ingress filtering

Sure.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
The mouse escaped.

Delivery-date: Thu, 18 Aug 2005 03:25:52 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Thu, 18 Aug 2005 05:24:55 +0200
To: Paul Jakma <[email-masked]>

El 17/08/2005, a las 21:42, Paul Jakma escribi=F3:

> On Wed, 17 Aug 2005, marcelo bagnulo braun wrote:
>
>> I guess that the only one that can identify this problem is the ULP=20=

>> itself.
>>
>> So i guess we have the following situation:
>> - The shim will only be able to detect (by itself, without additional=20=

>> information) failures when packets stop flowing
>
> How do you know if packets should or should not be flowing? What if=20
> it's a very quiet protocol? :)
>

There are two possibilities to deal with this, as explained in Ilijtsch=20=

draft

>> well, imho the shim needs only to define a default failure mode. I=20
>> mean, it is clear that different apps have different perception of=20
>> what a failure may be, but, since the shim is a generic layer, we=20
>> need to provide a generic definition of what a failure is. Clearly, i=20=

>> guess it would be very useful to allow ULPs to provide information to=20=

>> the shim about when they consider that a failure has occurred.
>
> Or define a very simple link^Wpath-state control protocol for shim6.
>

not sure what you mean by this...

regards, marcelo

> You could use it to monitor path-state to locators you're interested=20=

> in.
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Odets, where is thy sting?
> 		-- George S. Kaufman
>

Delivery-date: Thu, 18 Aug 2005 03:25:21 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Thu, 18 Aug 2005 05:23:32 +0200
To: Paul Jakma <[email-masked]>

El 17/08/2005, a las 21:38, Paul Jakma escribi=F3:

>
> IP is a best-effort protocol.
>
> Reliability, etc. is a concern of the upper-layers (eg TCP).
>
> Please explain why shim (which will, one hopes, look like fairly much=20=

> existing IP layers)) needs to reinvent functionality traditionally not=20=

> provided by IP?
>

SHIM wg is about providing multihoming support for IPv6. In particular=20=

a solution for IPv6 multihoming must be able to preserve communications=20=

through outages in the communicating path. Such functionality is=20
provided in IPv4 through BGP features but it requires the injection of=20=

site routes in the interdomain routing. In IPv6 PA addressing is used,=20=

so we need additional mechanisms (the shim) to provide equivalent=20
functionalities, in particular to preserve established communications=20
through outages.

> "Ah, but shim can make use of the fact that multiple locators could be=20=

> published for an endpoint!"
>
> Is the likely answer, explain:
>
> 1. Why this is a compelling argument given that it's been possible to=20=

> publish multiple addresses in DNS for a long long time, yet there has=20=

> been 0 demand for either applications to implement n^2 path-probing of=20=

> each local address to every remote address, or for OSes to implement=20=

> some kind of 'path-probe' shim to provide such functionality for all=20=

> applications?
>

I am afraid you are missing our goal here. this is not a matter of=20
oportunity but the way we can preserve established communication=20
through outages. see above

> 2. How this path-probing will interact with routing policy?
>
> The local administrator may have different cost local links. In order=20=

> to express policy he may do something like set the default route to go=20=

> via the low-cost link. (the default gets changes by some mechanism=20
> unknown to us, routing policy, script, whatever)
>
> Along comes shim6, sending packets with every possible source it can=20=

> find on the machine, as a consequence sending packets out of expensive=20=

> links (eg dial-on-demand links, or the $LOTS/Mbyte link..)
>

With the shim, path are closely related to addresses used, in=20
particular exit paths of the multihomed site are related to source=20
addresses used. So in order to provide this type of features, source=20
address selection has to be influenced, for instance using RFC 3484=20
policy table

> Ie: The best way to honour local policy is to use INADDR_ANY and let=20=

> the OS decide the source address by consulting local routing policy -=20=

> alternatively, an administratively specified address. Why exactly is=20=

> shim6 so different from everything else on the internet and special=20
> that this would not work for it?
>

this is exactly how the shim would support policing see above

> 3. The traditional way on the internet to guard against path failures=20=

> is to get a routing feed (and no, that does *not* imply you advertise=20=

> anything), why is shim6 so special that it can't defer to existing=20
> practice?
>

scalability. traditional IPv4 routing based multihoming lacks of it

> And reread 2 again :).
>
> 4. How will you decide which path is best?
>
> Some apps may prefer high-bandwidth/slightly lossy links over=20
> low-bandwidth/no-loss links. Other apps completely the opposite. Once=20=

> you start picking paths, how do you know what kind of path the=20
> application would prefer?
>

local policy can be expressed to some degree with the policy table=20
defined in RFC 3484. If more fine grained expression is needed (e.g.=20
per app) additional parameters need to be included in the policy table

> If you simply guess, how will your guess be any better than a very=20
> simple mechanism, eg using INADDR_ANY as source and just picking the=20=

> first locator that replies?
>
> (And again, the traditional way for administrators to set policy on=20
> what source address is the best to use is via routing policy. See 2=20
> yet again).
>
> 5. Given n^2 path-probing does not scale, and could be /very/=20
> expensive in some situations (and generally introducing complexity),=20=

> do you have statistics on the general reliability of path failures in=20=

> the internet to justify this expense and complexity?
>

we seem to be assuming that multihoming support is something useful and=20=

that it will be needed in IPv6. This multihoming support seems to=20
require communications to be preserved through outages.

> Are there any statistics as to how many path failures are due to=20
> /local/ link failures? (which does not require n^2 path-probing to=20
> detect).
>
> 6. If path-probing really is desired, explain why this is shim6=20
> specific?

path exploration is a fundamental part of the shim protocol-. maybe is=20=

not shim specific and ideas from other similar protocols can be used,=20
but it is imho a key part of the shim protocol and need to be part of=20
it.

>  Why could this not be done as part of a seperate programme or=20
> protocol?
>
> Some possibilities:
>
> - software that does local path-probing to determine reachability of
>   locally attached gateways (eg IPMP in Solaris for one)

this seems to be local, while shim is defined e2e

> - BFD and some other protocols in development

B means bidirectional, and we are not assuming bidirectional paths here

> - software that monitors some well-known paths and adjusts local
>   routing to suit based on administratively defined metrics
> - software that monitors the systems route-cache and does
>   path-probing for destinations that currently see flows
>

not sure what you mean by those but in any case, i am sure we can=20
benefit from these designs as well from the others you emntioned to=20
design the shim path exploration. If you are familiar with those, i am=20=

sure that your knowledge would be very useful to help with the design=20
of the path exploration protocol of the shim

> I could go on and on :).
>
> Note that it's the probing using every single local address which=20
> bothers the most. Simple heartbeats and monitoring which set of=20
> locators are reachable and just picking one and sticking to it till=20
> you needed to switch, I could agree with.

AFAICT this is the approach being considered here or at least one of=20
them. I mean, imho, we would only need to perform path exploration=20
after an outage

>
>> i mean, i see probing as a last resource to confirm that an outage=20
>> has occurred (and then a tool to explore alternative paths before=20
>> diverting the actual data packets)
>
> Why exactly must this be considered as a part of shim6? This does not=20=

> seem to be a shim6 specific thing at all, for a start off.
>

I fail to understand what you are missing. Failure detection and Path=20
exploration are key components of the shim, and they are needed to=20
preserve established communications through outages.

> As per above: picking the right /remote/ locator *is* a shim6 job - i=20=

> agree on that. It's the n^2 probing I want to ensure is /not/=20
> considered for inclusion in shim6 RFCs, other than as something=20
> mentioned as a possible implementation detail.
>

Ok, i think i see now.
Your problem is with probing with different source locators, right?
Well, this is needed because the source address determines the exit=20
path from the multihomed site. I mean, because we are assuming PA=20
addressing, changing the source address results in using a different=20
ISP in the multihomed site. That is why different source address need=20
to be explored

> It's the complexity of what's being proposed which I find wrong.
>
>> so, while in multiple occasions it may not be needed and can be=20
>> skipped, i see probing as a fundamental part of the shim
>
> If you mean probing every combination of local and remote addresses=20
> for reachability, I really don't see how you could come to that=20
> conclusion.
>
> There's many many years of existing deployment of IP using systems and=20=

> applications that simply don't consider such complex probing worth it.
>

right, because they are not assuming the usage of multiple PA addresses=20=

in a single host

When you include multiple PA addresses in hosts within a multihomed=20
site, then you find out that you need to try with different source=20
addresses.

>> may agree with this, but imho it need to be taken into account when=20=

>> discussing the present topics
>
> Sure.
>
> Well, I'd love to see discussion of the signalling formats for shim6=20=

> btw, rather than less immediately important talk of "how could we=20
> modify OS network stacks?" and "we could detect path failures and work=20=

> around them in ways nothing ever before has considered worth doing" :)=20=

> - and more importantly, I'd hate to see base shim6 specifications=20
> cluttered up with this kind of stuff (which likely wont be=20
> implemented, or wont be implemented soon in case of network stack=20
> signalling additions).
>
> I know several of you (marcelo, iljitsch, at least) have been thinking=20=

> about how to solve v6 multihoming for a /long/ time. I *know* you know=20=

> how to do it.
>
> The problem is, now that the end /is/ actually in sight (an actual=20
> IETF WG chartered to work on a /specific/ solution!), you've moved on=20=

> to considering problems /past/ shim6.
>
> So here's your endpoint locator algorithm:
>
> for every potential locator address for a ULID
>
> 	send a control message to probe the locator including
> 	 sufficient information for the other side to setup the shim
> 	 on their side
>
> wait for the minimum of PROBE_TIMEOUT seconds or until
>   you get a reply
>
> If you got a reply, it should have enough information to setup the=20
> shim, set it up and finish.
>

sort of... you still lacking DoS protection and locator security but=20
kind of what is being considered (with a couple of additional messages)

> Otherwise signal failure to the ULP (eg the system's equivalent of=20
> POSIX ENETUNREACH)
>
> That's it, very simple and implementable.
>
> Additionally, if you define shim6 to include a regular heartbeat, you=20=

> can monitor reachability. Include the locator's idea of its addresses=20=

> too, and two cookie fields (one for each side).
>

well, yes, but we are considering quite a few optimizations for this,=20
like ULP feedback and traffic monitoring also
but yes, This is in the lines of the  failure detection mechanism being=20=

considered,

> You can then detect:
>
> - which locator addresses work, if one doesn't mark it as unreachable.
> 	- if its the current locator, just pick the next one which
> 	  is not known to be unreachable (and so on).
>

path exploration is more complex than that, because you need to change=20=

the source address to change the exit ISP. remeber that we are assuming=20=

PA addresses, and they are only routed through one of the ISPs of the=20
multihomed site.

> - changes of locator on the /remote/ side
> 	- further, you can detect changes:
> 		- in advance (eg the locator can remove an address
> 	          in advance of it ceasing to accept packets on that
> 	          address, eg because of maintenance)
> 		- faster, eg the remote side may be monitoring its
> 	 	  local status, if it detects a change it can just
> 		  send a heartbeat immediately with the updated
> 		  locator addresses to use
>
> etc..
>
> <damn, i feel a draft coming on - are there better tools than opening=20=

> a text editor?>
>
> That's the kind of talk i want to see, about the actual nuts and bolts=20=

> of what is needed for shim6 to work - less "pie in the sky" stuff. :)
>

i think that most of this stuff is included in the current drafts, just=20=

that additional complexity is considered, for instance security stuff,=20=

unidirectional path support and cosniderations about the constraints=20
imposed by the usage of multiple PA addresses in the multohomed site=20
and ingress filtering

regards, marcelo

>> regards, marcelo
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Our POP server was kidnapped by a weasel.=

Delivery-date: Wed, 17 Aug 2005 19:58:28 +0000
Date: Wed, 17 Aug 2005 20:57:22 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: [email-masked], shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Wed, 17 Aug 2005, Paul Jakma wrote:

> Or define a very simple link^Wpath-state control protocol for shim6.
>
> You could use it to monitor path-state to locators you're interested in.

And yes, this wouldn't be as cool and as much fun as thrashing out a 
whole new set of signal between data-layer and IP and ULP's, but at 
least shim6 would be self-contained and wouldn't need any inside 
knowledge of ULP's and/or hints from ULPs.

And it needn't preclude optimising Shim6 if ULP hints /were/ 
available.

And yes, it involves additional packet overhead, proportional to the 
number of locators you want to monitor. But the overhead scales, at 
worst, linearly to the number of flows you see - in typical operation 
one would hope that some remote shim6 sites would be 'popular' and 
hence overhead amortised over multiple flows - a slight saving of 
overhead.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
HOW YOU CAN TELL THAT IT'S GOING TO BE A ROTTEN DAY:
 	#15 Your pet rock snaps at you.

Delivery-date: Wed, 17 Aug 2005 19:43:54 +0000
Date: Wed, 17 Aug 2005 20:42:49 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: [email-masked], shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Wed, 17 Aug 2005, marcelo bagnulo braun wrote:

> I guess that the only one that can identify this problem is the ULP itself.
>
> So i guess we have the following situation:
> - The shim will only be able to detect (by itself, without additional 
> information) failures when packets stop flowing

How do you know if packets should or should not be flowing? What 
if it's a very quiet protocol? :)

> well, imho the shim needs only to define a default failure mode. I 
> mean, it is clear that different apps have different perception of 
> what a failure may be, but, since the shim is a generic layer, we 
> need to provide a generic definition of what a failure is. Clearly, 
> i guess it would be very useful to allow ULPs to provide 
> information to the shim about when they consider that a failure has 
> occurred.

Or define a very simple link^Wpath-state control protocol for shim6.

You could use it to monitor path-state to locators you're interested 
in.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Odets, where is thy sting?
 		-- George S. Kaufman

Delivery-date: Wed, 17 Aug 2005 19:39:34 +0000
Date: Wed, 17 Aug 2005 20:38:18 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--299119927-2024522777-1124307498=:5353
Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8BIT

On Wed, 17 Aug 2005, marcelo bagnulo braun wrote:

> El 17/08/2005, a las 18:08, Paul Jakma escribió:

> I guess i agree, since we cannot be sure that all ULPs will provide 
> such feedback, we cannot base the shim failure detection on the 
> existence of such mechanisms, since it would result in limiting the 
> shim applicability to only those ULPs

Indeed.

>> The path-probing is, imho, mostly a complete waste of time. But an 
>> implementation can go wild if it wants.

> i am not sure what do you mean by this... i mean, if you don't have 
> a ULP that provides feedback, how can you be sure that the other 
> end is reachable?

IP is a best-effort protocol.

Reliability, etc. is a concern of the upper-layers (eg TCP).

Please explain why shim (which will, one hopes, look like fairly much 
existing IP layers)) needs to reinvent functionality traditionally 
not provided by IP?

"Ah, but shim can make use of the fact that multiple locators could 
be published for an endpoint!"

Is the likely answer, explain:

1. Why this is a compelling argument given that it's been possible to 
publish multiple addresses in DNS for a long long time, yet there has 
been 0 demand for either applications to implement n^2 path-probing 
of each local address to every remote address, or for OSes to 
implement some kind of 'path-probe' shim to provide such 
functionality for all applications?

2. How this path-probing will interact with routing policy?

The local administrator may have different cost local links. In order 
to express policy he may do something like set the default route to 
go via the low-cost link. (the default gets changes by some mechanism 
unknown to us, routing policy, script, whatever)

Along comes shim6, sending packets with every possible source it can 
find on the machine, as a consequence sending packets out of 
expensive links (eg dial-on-demand links, or the $LOTS/Mbyte link..)

Ie: The best way to honour local policy is to use INADDR_ANY and let 
the OS decide the source address by consulting local routing policy - 
alternatively, an administratively specified address. Why exactly is 
shim6 so different from everything else on the internet and special 
that this would not work for it?

3. The traditional way on the internet to guard against path failures 
is to get a routing feed (and no, that does *not* imply you advertise 
anything), why is shim6 so special that it can't defer to existing 
practice?

And reread 2 again :).

4. How will you decide which path is best?

Some apps may prefer high-bandwidth/slightly lossy links over 
low-bandwidth/no-loss links. Other apps completely the opposite. Once 
you start picking paths, how do you know what kind of path the 
application would prefer?

If you simply guess, how will your guess be any better than a very 
simple mechanism, eg using INADDR_ANY as source and just picking the 
first locator that replies?

(And again, the traditional way for administrators to set policy on 
what source address is the best to use is via routing policy. See 2 
yet again).

5. Given n^2 path-probing does not scale, and could be /very/ 
expensive in some situations (and generally introducing complexity), 
do you have statistics on the general reliability of path failures in 
the internet to justify this expense and complexity?

Are there any statistics as to how many path failures are due to 
/local/ link failures? (which does not require n^2 path-probing to 
detect).

6. If path-probing really is desired, explain why this is shim6 
specific? Why could this not be done as part of a seperate programme 
or protocol?

Some possibilities:

- software that does local path-probing to determine reachability of
   locally attached gateways (eg IPMP in Solaris for one)
- BFD and some other protocols in development
- software that monitors some well-known paths and adjusts local
   routing to suit based on administratively defined metrics
- software that monitors the systems route-cache and does
   path-probing for destinations that currently see flows

I could go on and on :).

Note that it's the probing using every single local address which 
bothers the most. Simple heartbeats and monitoring which set of 
locators are reachable and just picking one and sticking to it till 
you needed to switch, I could agree with.

> i mean, i see probing as a last resource to confirm that an outage 
> has occurred (and then a tool to explore alternative paths before 
> diverting the actual data packets)

Why exactly must this be considered as a part of shim6? This does not 
seem to be a shim6 specific thing at all, for a start off.

As per above: picking the right /remote/ locator *is* a shim6 job - i 
agree on that. It's the n^2 probing I want to ensure is /not/ 
considered for inclusion in shim6 RFCs, other than as something 
mentioned as a possible implementation detail.

It's the complexity of what's being proposed which I find wrong.

> so, while in multiple occasions it may not be needed and can be 
> skipped, i see probing as a fundamental part of the shim

If you mean probing every combination of local and remote addresses 
for reachability, I really don't see how you could come to that 
conclusion.

There's many many years of existing deployment of IP using systems 
and applications that simply don't consider such complex probing 
worth it.

> may agree with this, but imho it need to be taken into account when 
> discussing the present topics

Sure.

Well, I'd love to see discussion of the signalling formats for shim6 
btw, rather than less immediately important talk of "how could we 
modify OS network stacks?" and "we could detect path failures and 
work around them in ways nothing ever before has considered worth 
doing" :) - and more importantly, I'd hate to see base shim6 
specifications cluttered up with this kind of stuff (which likely 
wont be implemented, or wont be implemented soon in case of network 
stack signalling additions).

I know several of you (marcelo, iljitsch, at least) have been 
thinking about how to solve v6 multihoming for a /long/ time. I 
*know* you know how to do it.

The problem is, now that the end /is/ actually in sight (an actual 
IETF WG chartered to work on a /specific/ solution!), you've moved on 
to considering problems /past/ shim6.

So here's your endpoint locator algorithm:

for every potential locator address for a ULID

 	send a control message to probe the locator including
 	 sufficient information for the other side to setup the shim
 	 on their side

wait for the minimum of PROBE_TIMEOUT seconds or until
   you get a reply

If you got a reply, it should have enough information to setup the 
shim, set it up and finish.

Otherwise signal failure to the ULP (eg the system's equivalent of 
POSIX ENETUNREACH)

That's it, very simple and implementable.

Additionally, if you define shim6 to include a regular heartbeat, you 
can monitor reachability. Include the locator's idea of its addresses 
too, and two cookie fields (one for each side).

You can then detect:

- which locator addresses work, if one doesn't mark it as 
unreachable.
 	- if its the current locator, just pick the next one which
 	  is not known to be unreachable (and so on).

- changes of locator on the /remote/ side
 	- further, you can detect changes:
 		- in advance (eg the locator can remove an address
 	          in advance of it ceasing to accept packets on that
 	          address, eg because of maintenance)
 		- faster, eg the remote side may be monitoring its
 	 	  local status, if it detects a change it can just
 		  send a heartbeat immediately with the updated
 		  locator addresses to use

etc..

<damn, i feel a draft coming on - are there better tools than opening 
a text editor?>

That's the kind of talk i want to see, about the actual nuts and 
bolts of what is needed for shim6 to work - less "pie in the sky" 
stuff. :)

> regards, marcelo

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Our POP server was kidnapped by a weasel.
--299119927-2024522777-1124307498=:5353--

Delivery-date: Wed, 17 Aug 2005 18:18:27 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Wed, 17 Aug 2005 20:17:18 +0200
To: <[email-masked]>

Hi John,

some comments below...

El 15/08/2005, a las 13:36, <[email-masked]> escribi=F3:

> Iljitsch,
>
>
>> Somewhat to my disappointment, there were no opinions about
>> which of the two failure detection mechanisms is better,
>> either during the sessions or afterward on the list.
>
> At least for me, I'm still puzzeling over how to capture failure ...
> Between 2 hosts, tcp traffic may work but not udp traffic (mostly due
> to stupid middleboxes). Is this a failure from the shim point of view?
>

i guess we agree that the shim won't be able to track each different=20
ULP communication, so the shim won't be able to know if each=20
communication is progressing adequately by its own, right?

so, the shim can only see the exchange of packets with the other end,=20
and determine if packets are flowing with a given frequency. If packets=20=

stop flowing, the shim can guess a potential failure.
Now if packets are flowing (even if packets are only flowing to a given=20=

port and not to others) the shim by itself won't be able to detect this=20=

problem AFAICT.

I guess that the only one that can identify this problem is the ULP=20
itself.

So i guess we have the following situation:
- The shim will only be able to detect (by itself, without additional=20
information) failures when packets stop flowing
So, when there is not additional information, the case that you are=20
considering would not be detected by the shim

- This case can be detected when there is additional information such=20
as ULP feedback and perhaps some ICMP meesage
In this case, we have the following scenarios:
a) different ULP provide contradictory feedback
b) only the UDP app provides negative feedback but the shim can see=20
that packets are still flowing (and maybe reachability tests are=20
successful)
c) the SHIM receives ICMP errors but packets are still flowing.

I guess that in those scenarios thee shim can detect that something=20
strange in the lines that you have considered is going on, and maybe=20
some smart decision could be taken to deal with this... but before=20
that, do you agree with this description of the scenarios?

> Also, different transports have different concepts of what 'failure'
> is. When is a path considered failed? Classical TCP failure
> (retransmission
> of the the same packet x times)?  Too high bit-error rate?  I think we
> might
> run into circumstances where if we let the shim layer decide
> conclusively
> that a path has failed, it might decide much later that the path is =
bad
> than what the transport layer knows.
>

well, imho the shim needs only to define a default failure mode. I=20
mean, it is clear that different apps have different perception of what=20=

a failure may be, but, since the shim is a generic layer, we need to=20
provide a generic definition of what a failure is. Clearly, i guess it=20=

would be very useful to allow ULPs to provide information to the shim=20
about when they consider that a failure has occurred.

So, i would say that we will have:
- A generic definition of failure that is used when ULPs don't provide=20=

additional information
- each ULP can provide feedback about when they think that a failure=20
has occurred.

For the generic failure definition, i think that the first hint that a=20=

failure may have occurred is that packets are flowing outwards but no=20
incoming packets are received (for a given period T)
At this point i guess that an explicit reachability test needs to be=20
performed.

> I think we should avoid any transport layer functions in the shim =
layer
> (or at least as much as possible).  I'm leaning more towards the
> rich-api
> type of functionality, where the shim provides as much info as it can
> to the transport layers, and let the transport layers make decisions.
>

agree, but do you agree that we also need to support those ULPs that=20
don't have this rich api? i mean, that we need to support existent ULP=20=

without requiring modifications to benefit from the shim?

> I'm also wondering if the failure detection mechanism should be a
> pluggable
> option; I've been thinking how the shim layer would work in wireless
> environments,
> and I don't think I'd like to have fast heartbeating, as that would
> drain
> batteries unnesscesarily.  Could it be possible to have this
> functionality
> as optional?

Well, i guess that many folks have considered that when positive=20
feedback from the ULP is provided, then heartbeats would be omited, is=20=

this what you had in mind?

regards, marcelo

>
> John
>
>

Delivery-date: Wed, 17 Aug 2005 18:18:27 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Wed, 17 Aug 2005 20:17:58 +0200
To: Paul Jakma <[email-masked]>

El 17/08/2005, a las 18:08, Paul Jakma escribi=F3:

> On Wed, 17 Aug 2005, marcelo bagnulo braun wrote:
>
>> FWIW, TCP for instance already provide some form of such signaling=20
>> see Appendix E.1 of RFC 2461 Of course other ULP don't provide any=20
>> kind of feedback.
>>
>> So i guess it would be a better not to fully rely on ULP feedback,=20
>> but otoh not using the information that some ULP provide seems=20
>> suboptimal to me.
>
> 1. I said "extra signaling"
>
> 2. Note the title of appendix E: "IMPLEMENTATION ISSUES" and
>    describes a possible optimisation that can be made if ULP's can be
>    modified - it's wholly at the implementation's discretion.
>
> Exploring stuff like path-reachability probing or implementation=20
> optimisations via shim<->ULP signalling (which ULP could be IP quite=20=

> likely btw, not TCP - implementations possibly will have /two/ layers=20=

> to pass hints through) is fine, but do it as an appendix or an=20
> informational document or wait for implementors to experiment.
>

I guess i agree, since we cannot be sure that all ULPs will provide=20
such feedback, we cannot base the shim failure detection on the=20
existence of such mechanisms, since it would result in limiting the=20
shim applicability to only those ULPs

> The path-probing is, imho, mostly a complete waste of time. But an=20
> implementation can go wild if it wants.
>

i am not sure what do you mean by this... i mean, if you don't have a=20
ULP that provides feedback, how can you be sure that the other end is=20
reachable? i mean, i see probing as a last resource to confirm that an=20=

outage has occurred (and then a tool to explore alternative paths=20
before diverting the actual data packets)

so, while in multiple occasions it may not be needed and can be=20
skipped, i see probing as a fundamental part of the shim

> The potentials of TCP<->Shim signaling are interesting I'll admit.=20
> However, again it's an implementation detail

may agree with this, but imho it need to be taken into account when=20
discussing the present topics

regards, marcelo

> (and only reason its being discussed is cause of the crazy=20
> path-probing stuff).
>
> I note from the archives that I'm not the first person to have=20
> cautioned against going down the path-probing route (Joe Abley too).
>
> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> Manly's Maxim:
> 	Logic is a systematic method of coming to the wrong conclusion
> 	with confidence.
>

Delivery-date: Wed, 17 Aug 2005 16:09:27 +0000
Date: Wed, 17 Aug 2005 17:08:23 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Wed, 17 Aug 2005, marcelo bagnulo braun wrote:

> FWIW, TCP for instance already provide some form of such signaling 
> see Appendix E.1 of RFC 2461 Of course other ULP don't provide any 
> kind of feedback.
>
> So i guess it would be a better not to fully rely on ULP feedback, 
> but otoh not using the information that some ULP provide seems 
> suboptimal to me.

1. I said "extra signaling"

2. Note the title of appendix E: "IMPLEMENTATION ISSUES" and
    describes a possible optimisation that can be made if ULP's can be
    modified - it's wholly at the implementation's discretion.

Exploring stuff like path-reachability probing or implementation 
optimisations via shim<->ULP signalling (which ULP could be IP quite 
likely btw, not TCP - implementations possibly will have /two/ layers 
to pass hints through) is fine, but do it as an appendix or an 
informational document or wait for implementors to experiment.

The path-probing is, imho, mostly a complete waste of time. But an 
implementation can go wild if it wants.

The potentials of TCP<->Shim signaling are interesting I'll admit. 
However, again it's an implementation detail (and only reason its 
being discussed is cause of the crazy path-probing stuff).

I note from the archives that I'm not the first person to have 
cautioned against going down the path-probing route (Joe Abley too).

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Manly's Maxim:
 	Logic is a systematic method of coming to the wrong conclusion
 	with confidence.

Delivery-date: Wed, 17 Aug 2005 15:32:37 +0000
Date: Wed, 17 Aug 2005 08:33:24 -0700
From: Erik Nordmark <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: shim6 <[email-masked]>
Subject: Re: failure detection

Iljitsch van Beijnum wrote:
> Hi,
> 
> Somewhat to my disappointment, there were no opinions about which of  
> the two failure detection mechanisms is better, either during the  
> sessions or afterward on the list.

There wasn't any discussion after your presentation :-(
What I would have said is that I think FBD is sufficient for verifying 
reachability and simple. All we need is reference the text in the l3shim 
draft which says that the shim isn't applied to multicast.

Below you seem to assume that there is an option to do probing instead 
of (or in addition to) FBD, which complicates things and introduces the 
desire to supress such probing.

I don't think we need that, but I haven't fully digested the issues of 
which end discovers the lack of reachability yet, so I reserve the right 
to change my mind :-)

    Erik

> Since then, I've been thinking about ways to unify the two  approaches. 
> What I've been able to come up with is this:
> 
> 1. when host A is uncertain about the reachability status of the  
> current address pair, it sends host B a request
> 2. host B now sends back an answer, so host A knows that in the  absence 
> of this answer that something is wrong
> 3. host A may optionally request that host B send X packets at Y  
> intervals to allow A to determine that there is reachability in the B  
> -> A direction
> 4. with this request, A may also request that B suppress the  additional 
> packets when there is other traffic in the B -> A direction
> 
> Capabilities 1 and 2 are mandatory, capabilities 3 and 4 are optional.
> 
> A brain-dead implementation would simply send a probe and expect an  
> answer every 10 seconds or so. (I'm not sure we should allow this as  
> the entire failure detection mechanism as it adds unnecessary extra  
> packets for all associations.)
> 
> An implementation that uses upper layer advice would only initiate a  
> probe/reply sequence when the upper layer is unhappy.
> 
> An implementation that knows about bidirectional data flow would only  
> initiate a probe/reply sequence when there is outgoing traffic but no  
> incoming traffic. It would probably also ask the other side to keep  
> sending packets when it has no payload data to transmit. The other  side 
> may or may not implement this. If it does, so much the better.  If it 
> doesn't, there will be more frequent unidirectional traffic so  host A 
> will do a probe/reply more often.
> 
> Although this increases the number of possible ways the failure  
> detection can play out and thus makes debugging more difficult, it  has 
> the advantage that a basic impledmentation is simple, but it  allows for 
> implementers to add additional capabilities and improve  failure 
> detection algorithms without having to go through an IETF  
> standardization cycle.
> 
> Thoughts?
> 

Delivery-date: Wed, 17 Aug 2005 15:31:35 +0000
Date: Wed, 17 Aug 2005 10:31:01 -0500
From: Spencer Dawkins <[email-masked]>
Reply-To:  [email-masked]
To: shim6 <[email-masked]>
Subject: Re: shim-aware transports

Pasi Sarolahti wrote:

> Hi,
> 
> I think it is good that the TCP (and other transport protocols)
> interworking was brought up, and it would be good to write the issues
> down in some document. The question is whether it should be part of some
> existing document, or a separate document. To me the latter seems good
> choice, because many of the issues would be common also to other
> mechanisms than shim6 (HIP came to my mind first, and why not even
> Mobile IP).

Yes, exactly. Even layer-two mobility events can change path 
characteristics.

> I have some comments about the TCP interworking, below..
> 
> On Sun, 2005-08-14 at 14:54 +0200, ext Iljitsch van Beijnum wrote:
> 
> 
>>This would allow for the following succession in TCP/shim developments:
>>
>>- Initially, TCP would be unmodified, so the shim would use the  
>>adaption layer to do address magic whenever the basic shim sublayer  
>>determines reachability has changed.
>>
>>- Next step: TCP is modified slightly to provide reachability hints  
>>to the shim layer. The basic shim sublayer can optimize some of its  
>>failure/reachability detection mechanism based on this.
> 
> 
> I think it would be good also to have some kind of signal from shim to
> TCP to indicate when path (address) has changed from quite early on
> (somewhere between "initially" and "next step"). In response TCP should
> re-initialize its congestion control and RTT parameters, and probe the
> new path capacity in slow-start. The new path likely has different
> properties, so slow-start is the correct way of re-probing the current
> path capacity.

I'm not saying that the TRIGTRAN BoF was the canonical answer, but when 
we were discussing TCP in TRIGTRAN, we pretty quickly converged on NOT 
notifying transports when bandwidth available changed, because TCPs have 
to respond to losses, and adding additional hints just adds stack 
complexity beyond the complexity that is unavoidable.

Yeah, in a perfect world, we might slow-start, but in a perfect world, 
users wouldn't hit "reload" on their browsers while we were 
slow-starting, either.

So, the TRIGTRAN answer was

- ignore apparent path changes.
- if you lose something, respond the same way you would have responded 
to loss in the absence of a path change - even congestion avoidance 
drops your congestion window in half, so if you have enough packets in 
flight, you are already responding dramatically in something like one RTT.

Since ("how to lie with statistics") you have a 50-50 chance of landing 
on a BETTER path, and you will have to respond to any losses using 
existing mechanisms anyway, forcing slow start without encountering a 
loss seemed like overkill, and even forcing congestion avoidance just 
added state machine complexity without a lot of obvious benefit.

If people want to pursue this, let's pursue it in TSVWG, OK?

>>- Finally: TCP becomes aware of multiple addresses, and starts  
>>keeping per-address pair RTT statistics. TCP moves to another address  
>>pair either when the shim tells it to (because the shim detected a  
>>problem) or when it determines another pair is "better" based on its  
>>own statistics.
> 
> 
> (TCP would need to maintain also per address-path cwnd and ssthresh, and
> possibly other internal congestion control-related parameters, in
> addition to RTT and friends)

Joe Touch can speak more definitively on this than I can, but my 
understanding is that proposals that touch this much of TCP haven't been 
widely deployed (for example, 
ftp://ftp.rfc-editor.org/in-notes/rfc2140.txt) - at least, that was the 
case a couple of years ago, when I was still doing TSV stuff.

> This seems a pretty significant change to TCP. Alternatively, if the
> path changes are not expected to be very frequent, it might be

We're in one of two situations here - either path changes are rare 
enough for TCP to be able to appropriately adjust to bandwidth changes 
using only end-to-end (multiple-RTT) mechanisms, or they are common 
enough that TCP is going to have horrible throughput, anyway.

My suggestion is to make the first assumption.

> sufficient just to simply reset the congestion control and RTT
> parameters upon path change, and proceed in slow-start. TCP should
> anyway apply congestion window validation [RFC2861] for each of the per
> path cwnd and ssthresh parameters, so after an idle period of few RTOs
> (generally less than one minute) the effect would not be much different
> for those paths that have not been used for a while.

Yeah, congestion window validation would be a good thing to do, and 
modern TCPs actually do handle this correctly (older TCPs reset their 
timers when they received a packet, and THEN checked the timer to see if 
they needed to validate the congestion window. Amazingly, they never 
needed to :-)

Spencer

Delivery-date: Wed, 17 Aug 2005 14:46:19 +0000
Cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Wed, 17 Aug 2005 16:41:45 +0200
To: Paul Jakma <[email-masked]>

El 14/08/2005, a las 16:49, Paul Jakma escribi=F3:

>
> (and again, I hope this WG will disabuse itself of the notion that=20
> extra signals /down/ from transports to shim6 can even be considered=20=

> at this early stage in shim6's life).
>

FWIW, TCP for instance already provide some form of such signaling see=20=

Appendix E.1 of RFC 2461
Of course other ULP don't provide any kind of feedback.

So i guess it would be a better not to fully rely on ULP feedback, but=20=

otoh not using the information that some ULP provide seems suboptimal=20
to me.

Regards, marcelo

> regards,
> --=20
> Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
> Fortune:
> You will be married within a year, and divorced within two.
>

Delivery-date: Wed, 17 Aug 2005 14:45:49 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: about reachability detection draft
Date: Wed, 17 Aug 2005 16:24:45 +0200
To: Iljitsch van Beijnum <[email-masked]>

El 14/08/2005, a las 21:57, Iljitsch van Beijnum escribi=F3:

> Ok, the long awaited reply:
>
> On 16-jul-2005, at 17:49, marcelo bagnulo braun wrote:
>
>> In section 2 it is stated that:
>
>> - In the second model, a host can only detect problems in the=20
>> receiving
>>   direction so it must depend on the correspondent to detect problems
>>   in the other direction
>
> [this is what I called forced bidirectional or FBD in the slides]
>
>> I think it is important to consider one step further then. I mean,=20
>> what can a host possibly do when he detects an outage in the incoming=20=

>> path? If a host detects an outage in the outgoing path, he can change=20=

>> the address pair that he is using to send packets and see if it=20
>> solves the problem, but what can he do if he detects an outage in the=20=

>> incoming path? I guess that the only option would be to notify the=20
>> correspondent node about the failure so that the correspondent uses=20=

>> an alternative path (Note that we are asuming the the case of=20
>> unidirectional connectivity is possible)
>> So i guess that this mode needs an additional message informing the=20=

>> failure, which needs to be taken into account when comparing the=20
>> options.
>
> Well, the idea is that when there is actually a failure in the active=20=

> path, it's likely that there will be failures in one or more of the=20
> backup paths too. So in my opinion, it doesn't make sense to switch to=20=

> a new path blindly. So we must first test any path we may want to=20
> switch to.

i fully agree with this... but i dont understand how this is related=20
with my comment above though

>
> Now one of two things can happen:
>
> 1. the candidate path doesn't work either -> timeout, try another
> 2. the candidate path works -> we can communicate with the=20
> correspondent
>
> If there is any connectivity left, eventually we'll end up in=20
> situation 2, and the fact that we're sending the correspondent these=20=

> test packets (which are different from the regular ones that we send=20=

> when we thing there is still connectivity, see below) tells the=20
> correspondent that something is wrong, so the correspondent starts=20
> sending test packets in our direction too.
>

let me see if i get this
Suppose that A is communicating with B
Suppose that we are using FBD so that a given frequency of packets=20
(data or just signaling) is guaranteed by the shim in each direction

Suppose now that A stops receiving packets. This implies a failure in=20
the B->A path.

Now are you assuming that when A stops receiving packets A should try=20
with alternative paths?
I am not sure this is good approach, since the path A->B may be working=20=

properly.. moreover, maybe this path A->B is the only one working.

I mean, if A tries alternative paths when the B->A path fails,=20
basically we are assuming bidirectional connectivity, and we wouldn't=20
be considering the unidirectional connectivity case, right?

As i see the FBD mechanism would be the following:

A and B are communicating
they are using FBD
A stops receiving packets

So, A informs B that the B->A path has failed (this implies some form=20
of signaling from A to B, which is likely to be required to be somehow=20=

reliable, which is why i see a potential difficulty here)

A continues using the path from A->B while B starts exploring=20
alternative paths

> The reason I think we need two types of test packets is because in the=20=

> situation where there is a large number of address pairs with unknown=20=

> reachability and/or RTT, we need to do extra work to make sure that=20
> when A sends (for instance) a probe A2 -> B3, which makes it to B, but=20=

> there is only connectivity from B to A over B1 -> A4, it's unlikely=20
> that B1 -> A4 is the first pair B tries, so B needs to include reports=20=

> about what it got from A in all of its packets.

So you think that we need one packet type for periodic keepalives for=20
failure detection and another type of packet for exploring alternative=20=

paths?

I am not sure about this since when two different unidirectional paths=20=

are being used for a communication then the keepalive packet will need=20=

to carry information about the locator pair used for incoming packets.

I mean, consider that a and Bare communicating, but they are using two=20=

different unidirectional paths i.e. A->B is using locators A1 and B2,=20
while B->A is using locators A2 and B2

In this case when A sends a reachabilitity test request to B it will=20
use A1 and B1. However, when B replies, it will use B2 and A2, so i=20
guess it would a good idea to include in this reply packet information=20=

about the address used for the request packet i.e. A1 and B1

>  And we need a reasonable level of authentication too, because we=20
> haven't previously established that these addresses indeed belong to=20=

> the correspondent.

I am not sure what is the level of security we need here yet, but in=20
any case imho it will be very related to the security used during the=20
shim context establishment.

I mean, imho the critical part of the security if the addition of new=20
locators to the locator set. Once the locators have been validated,=20
using them shouldn't require to tight security requirements imho

>
> On the other hand, when A1 <-> B1 is working happily, there is no need=20=

> to use such a complex protocol: we are only testing one pair in each=20=

> direction, and the correspondent has been authenticated earlier. If we=20=

> wanted we could even use pings to determine whether this still works.=20=

> (Well, sort of...)
>

But as i see it, we are only going to try locators that are included in=20=

the locator set available for that shim context, which means that they=20=

have already been validated....

>> I guess it would also make sense to state how the those mechanisms=20
>> behave when there are no outgoing packets? i mean, i guess that in=20
>> any of the modes signaling is suppressed right?, however, i guess=20
>> that the hosts assume that the address pair is reachable, right?
>
> With correspondent unreachability detection (first mechanism in the=20
> draft) the transport hints would accompany the packets, so packets =3D=20=

> no, hints =3D no -> no action. Only packets =3D yes and hints =3D no=20=

> requires reachability probes.
>
> In the forced bidirectional communication we only send probes when we=20=

> sent data recently but didn't receive data recently, so no probes in=20=

> this case either.
>
> Note that some transports and some applications explicitly keep the=20
> session alive with periodic traffic.
>

right, i guess it would be good to include this explicitly this in the=20=

draft

>> Additionally, i guess that there are other information that needs to=20=

>> be taken into account when detecting reachability, such as ICMP error=20=

>> messages, address deprecation, lower layers information (i know that=20=

>> you state that you assume that addresses are available, but what=20
>> happens if an address of the currently used address pair is=20
>> deprecated? or if the associated interface goes down?)
>
> It makes sense to send a probe when there is an ICMP error (have to=20
> rate limit this, though).

ok

>  Deprecation is irrelevant, as we can continue to use deprecated=20
> addresses.

Well, deprecation means that somewhere in the near future, the address=20=

won't be available anymore, right?
So i guess it make sense to keep on using it as ULID, but i wonder if=20
it wouldn't be a good strategy to try to rehome the ongoing=20
communications to an alternative locator...?

> Lower layer events are also a reason to do a reachability test, I=20
> think.

ok

>  A more interesting case is when an address is removed from the=20
> system. I don't remember which session it was, but in Paris someone=20
> was talking about how systems keep using addresses they no longer have=20=

> because upper layers still use those addresses.

well, in the shim this address should be kept as a possible ULID but=20
not as a valid locator i guess

>  IMO this is a feature: if I unplug my ethernet from my powerbook and=20=

> turn on my wifi, I get the same address on a different interface and=20=

> my sessions are still alive. Under windows, things like this kill your=20=

> sessions immediately.
>

in this case i guess that this address should be:
- kept as a ULID during all the time
- should be removed from the locator set during the period which the=20
address was not available (i.e. during the time it took to remove it=20
from the ethernet and it is back in the wifi)

>> Another issue that may be of interest is what happens with an address=20=

>> (pair) after it becomes unreachable? i mean, is it used in followings=20=

>> address pair explorations? is it putted in quarantine?
>
> The way I see it there is an ordered list of address pairs. The more=20=

> probes fail to make it to the other side the lower the address pair=20
> ends up on the list, I imagine.  :-)

but what happens when you only have 2 address paris for instance? you=20
may end up trying with the same two address pairs forever... I mean, i=20=

guess we need a mechanism to give up trying, right?

Putting address pairs in quarantine would be one of such mechanisms,=20
but there may be others, like a maximum number of attempts...

>
> We need to think about the situation where a fast primary link fails=20=

> and we switch to a slow backup, though. Presumably, we'll want to=20
> switch back to the fast primary address pair when possible.
>

Right, the same arguments may apply for other reasons like cost of the=20=

link, security/privacy features of the path... But the question is how=20=

does the shim is aware of such information? i guess that these are=20
policy issues and we could include some means to express this type of=20
considerations in the shim

Regards, marcelo

Delivery-date: Tue, 16 Aug 2005 09:31:06 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: shim-aware transports
Date: Tue, 16 Aug 2005 11:30:09 +0200
To: Pasi Sarolahti <[email-masked]>

On 16-aug-2005, at 9:43, Pasi Sarolahti wrote:

> I think it would be good also to have some kind of signal from shim to
> TCP to indicate when path (address) has changed from quite early on
> (somewhere between "initially" and "next step").

Yes, early on for the shim. Whether implementers want to change TCP  
immediately is up to them.

> In response TCP should
> re-initialize its congestion control and RTT parameters, and probe the
> new path capacity in slow-start. The new path likely has different
> properties, so slow-start is the correct way of re-probing the current
> path capacity.

I agree. However, at least one person argued that slow start was  
unnecessary/undesirable here.

> (TCP would need to maintain also per address-path cwnd and  
> ssthresh, and
> possibly other internal congestion control-related parameters, in
> addition to RTT and friends)

Yes.

> This seems a pretty significant change to TCP.

Right. But it would allow TCP to use the best path, or even load  
balance across multiple paths. Whether this is worth it is hard to  
tell at this point.

Iljitsch

Delivery-date: Tue, 16 Aug 2005 07:44:02 +0000
Subject: Re: shim-aware transports
From: Pasi Sarolahti <[email-masked]>
To: ext Iljitsch van Beijnum <[email-masked]>
Cc: shim6 <[email-masked]>
Organization: Nokia Research Center
Date: Tue, 16 Aug 2005 10:43:07 +0300

--=-o4k6w+Sx1INCEXc/ksDI
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

Hi,

I think it is good that the TCP (and other transport protocols)
interworking was brought up, and it would be good to write the issues
down in some document. The question is whether it should be part of some
existing document, or a separate document. To me the latter seems good
choice, because many of the issues would be common also to other
mechanisms than shim6 (HIP came to my mind first, and why not even
Mobile IP).

I have some comments about the TCP interworking, below..

On Sun, 2005-08-14 at 14:54 +0200, ext Iljitsch van Beijnum wrote:

> This would allow for the following succession in TCP/shim developments:
>=20
> - Initially, TCP would be unmodified, so the shim would use the =20
> adaption layer to do address magic whenever the basic shim sublayer =20
> determines reachability has changed.
>=20
> - Next step: TCP is modified slightly to provide reachability hints =20
> to the shim layer. The basic shim sublayer can optimize some of its =20
> failure/reachability detection mechanism based on this.

I think it would be good also to have some kind of signal from shim to
TCP to indicate when path (address) has changed from quite early on
(somewhere between "initially" and "next step"). In response TCP should
re-initialize its congestion control and RTT parameters, and probe the
new path capacity in slow-start. The new path likely has different
properties, so slow-start is the correct way of re-probing the current
path capacity.

> - Finally: TCP becomes aware of multiple addresses, and starts =20
> keeping per-address pair RTT statistics. TCP moves to another address =20
> pair either when the shim tells it to (because the shim detected a =20
> problem) or when it determines another pair is "better" based on its =20
> own statistics.

(TCP would need to maintain also per address-path cwnd and ssthresh, and
possibly other internal congestion control-related parameters, in
addition to RTT and friends)

This seems a pretty significant change to TCP. Alternatively, if the
path changes are not expected to be very frequent, it might be
sufficient just to simply reset the congestion control and RTT
parameters upon path change, and proceed in slow-start. TCP should
anyway apply congestion window validation [RFC2861] for each of the per
path cwnd and ssthresh parameters, so after an idle period of few RTOs
(generally less than one minute) the effect would not be much different
for those paths that have not been used for a while.

- Pasi

--=-o4k6w+Sx1INCEXc/ksDI
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-o4k6w+Sx1INCEXc/ksDI--

Delivery-date: Mon, 15 Aug 2005 20:05:02 +0000
Date: Mon, 15 Aug 2005 21:03:57 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Mon, 15 Aug 2005, Paul Jakma wrote:

> 	- some kind of independent path-probing software that adjusts
> 	  routing as required
> 	  (and some OSes already have this functionality)

> - Shim6 does not need this complexity, it would be redundant and
>  counter-productive (particularly for sites that got route
>  availability information from their ISP)

Note that the latter does not imply that the former functionality 
could not be included as part of (or alongside) a shim6 
implementation - just that I don't think it's at all useful to go 
into all the complexity of specifying the former as part of shim6. 
Let it be an implementation detail (or not).

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
if (user_specified)
     /* Didn't work, but the user is convinced this is the
      * place. */
         linux-2.4.0-test2/drivers/parport/parport_pc.c

Delivery-date: Mon, 15 Aug 2005 19:04:51 +0000
Date: Mon, 15 Aug 2005 20:03:46 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Mon, 15 Aug 2005, Iljitsch van Beijnum wrote:

> The thing is that for TCP, the source address is only selected at 
> the beginning of the session. After that, it can't be changed 
> because the other side wouldn't recognize the session anymore, 
> i.e., it's needed for demultiplexing.

Uh, so shim6 on the receiving side is going to use details of TCP to 
demux shim6 packets before passing them up the stack (to IP layer 
again). I'm guesing the answer is "No" :).

I think we're likely agreed details of TCP, or whatever happens to be 
flowing, should be invisble to the shim layer.

And again, if you don't try tie shim6 demux to the /specific/ locator 
address, this all gets easier. Don't make shim6 stateful, things get 
easier. Demux can just can be the IPv6 equivalent of SNAT.

> The shim MUST be prepared to set the source address because there 
> may be ingress filtering at some point, and the wrong source 
> address will make communication impossible.

??

Well, tough luck, then that local 'locator' address should never have 
been specified as valid for shim to use. Can you give a more concrete 
example? Do you mean RPF filtering by ISPs? Yes, for the multi-PA 
approach you'll need ISP visible source to match.

But you don't per-se need address selection in shim6 for this - 
however, I'm unsure now of what is being proposed exactly for the 
mechanism.

> If the shim is going to rewrite the source address at the 
> destination, we're no longer limited by TCP demux, but of course 
> the shim itself must also demux. It's possible to do this without 
> considering the source address, or at least the first 64 bits of 
> the source address, but it's not entirely non-trivial.

Why exactly? Demux can look like (security aside):

- take source address (the locator source) and do a lookup in a
   locator -> endpoint table
- replace address

Hey presto. All that needs to be done is populate the 
locator->endpoint table during setup with all the possible locators 
the other-side advertises.

> If you follow your line of thinking about the source address to its 
> logical conclusion, then obviously the site exit router must select 
> the source address.

Forgive me, I don't seem to be aware of what shim6 actually /is/ 
proposing for its mechanism then.

If the source is not selected by the exit router/host, then what is 
the source to be set to exactly? If the proposal is for packets on 
exit to contain the multihomed-prefix, that's fine. No locator 
selection needed at all (cause you're not rewritting the source). 
(But see the RPF problem, right?).

If the talk about locator address selection and probing is to do with 
shim setup? My point stands: Don't bother, just try reach one of the 
remote locator addresses, as appropriate.

>> I'd suggest it would be more fruitful to ensure state between two 
>> shim6 speakers /not/ depend on the 'outside' (ie globally 
>> routable) addresses, and let shim6 be able to handle messages 
>> between two endpoints arrive with any 'outside' source address.
>
> Not sure what you mean...

Wrt specifics: sorry, if the rewrite is to be of destination on 
/demux/ at ingress site, above doesn't apply.

In general, try not make shim6 state depend on any /specific/ locator 
state, so as to allow shim6 to work with any /valid/ locator.

> There is no getting around it, except for giving up before you reach n^2...

Yes there is. Just dont bother.

Nothing else in common usage tries to work around interner-cloud-path 
failures in such a fashion, nor should shim6. This specific proposal 
(from what I saw of the slides at IETF) is a *really poor* 
reinvention of internet routing. Don't go there.

> (You can of course move this problem to another layer.)

Tada.

>>    - A may load-balance outbound packets between the two ISPs
>
> Not really. Suppose A sends packets to router R which connects to 
> both ISPs. How is A going to influence to which ISP R sends any 
> given packet?

It doesn't. That's how internet routing works.

>> However, I feel it is a *really* bad idea to even consider trying to solve 
>> the problem of "internet cloud" path failures,
>
> I'm afraid you've come to the wrong wg.  :-)

No, I havn't.

The charter (thankfully) says nothing about coping with /any/ path 
failure. Just that shim6 be able to cope with changes at the 
endpoints.

>>    - Existing protocols already live with this today, and
>>      generally we're happy
>
> Are you saying there is no requirement to multihome, even for people who go 
> out of business when their link to the net fails for a significant amount of 
> time?

I have no idea how you got that from the point above.

Multihomed or not is irrelevant to "internet cloud" path failures 
that you can do nothing about. And you should know I'm *very* eager 
to see IPv6 multihoming made possible ;).

>>    - It can be solved/mostly-mitigated by running a routing
>>      protocol (ie BGP)
>
> Not without an information explosion. If you have address 
> dada::dead:beef, and I see in BGP that dada::/16 is up, I'm going 
> to send the packet as per this route. If it then turns out that 
> your link to this ISP is down the packet isn't going to make it. 
> And sending out a route for dada::dead:0/112 over another ISP won't 
> scale.

Did I say anything about *advertising* anything over BGP?

Getting read-only BGP feeds is quite useful if you have multiple 
links, and is a far *far* more sane way to detect the right path than 
the n*2 path-probing idea.

> Ever looked inside the cockpit of a modern airliner?

> Not simple. And these people care A LOT more about their 
> "redundancy protocols" that we do.

Firstly I don't actually see the relevancy, further discussing 
analogies is usually a waste of time. FWIW, yes, I've seen lots of 
cockpits and they are designed to be as simple as they can be - 
modern flight automation and glass cockpits make this even more 
possible. Go compare a modern airliner cockpit to that of to an /old/ 
one.

But not at all relevant. ;)

> Keeping it simple is good advice when the choice is gaining 
> advantage X in a simple way or gaining advantage X in a complex 
> way. If keeping it simple means you only get X/2 rather than X, 
> then simplicity isn't necessarily the right choice.

The most common cases for IPv6 "multihoming" will be one or more of:

a) Single homed, but want "PI" addresses for portability
 	- Think of this as multihoming but with spread over time ;)

b) Cheap multihoming (eg two cheap end-user/SOHO DSL links)
 	- ISP wont do dynamic routing (be it advertising a default
 	  via RIP or somesuch, or BGP)

c) Enterprise multihoming
 	- ISP /will/ advertise availibility via routing

For a, no path-probing locator-selection is needed in shim6 at all. 
For b, no path-probing locator-selection is needed in shim6 at all.

So the only reason to want path-selection (by doing path-probes with 
different locators) is for case b. My argument would be it's not 
worth adding lots of complexity to shim6 for this case, when you 
could do just as good a job /outside/ of shim6, with eg:

 	- a simple daemon to monitor link-status (if available)
 	- some kind of independent path-probing software that adjusts
 	  routing as required
 	  (and some OSes already have this functionality)

To reiterate:

- We already have internet-cloud failures today, and we cope

   (eg, cause we're singly-homed, or we get route availibity
   information from our ISP, or with some other path-probing
   software, or we just trust our ISPs to do what they're supposed to
   and get our packets to where they want to go, and we live with the
   X nines availability we get from them or we get a different ISP)

- Those /same/ mechanisms can be used to cope with
   internet-cloud paths failures in a shim6 world

Ergo:

- Shim6 does not need this complexity, it would be redundant and
   counter-productive (particularly for sites that got route
   availability information from their ISP)

> What "crap" are you referring to?

The n^2 probing. Don't make shim6 specify that, please.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Lonesome?

Like a change?
Like a new job?
Like excitement?
Like to meet new and interesting people?

JUST SCREW-UP ONE MORE TIME!!!!!!!

Delivery-date: Mon, 15 Aug 2005 17:54:32 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: failure detection
Date: Mon, 15 Aug 2005 19:53:33 +0200
To: Paul Jakma <[email-masked]>

On 15-aug-2005, at 18:08, Paul Jakma wrote:

> I'd also like to potentially add source-address-selection (for the  
> globally routable shim address) to this list.

[...]

> Again, SAS is simply *not* something shim6 should do, or something  
> which it has remotely near enough information to do. Let normal IP  
> routing and SAS handle this task.

The thing is that for TCP, the source address is only selected at the  
beginning of the session. After that, it can't be changed because the  
other side wouldn't recognize the session anymore, i.e., it's needed  
for demultiplexing.

The shim MUST be prepared to set the source address because there may  
be ingress filtering at some point, and the wrong source address will  
make communication impossible. If the shim is going to rewrite the  
source address at the destination, we're no longer limited by TCP  
demux, but of course the shim itself must also demux. It's possible  
to do this without considering the source address, or at least the  
first 64 bits of the source address, but it's not entirely non-trivial.

If you follow your line of thinking about the source address to its  
logical conclusion, then obviously the site exit router must select  
the source address. We've talked about this A LOT in multi6 and its  
various design teams, and (others jump in if you don't agree) the  
conclusion was that this is certainly a capability we would like to  
have, but since it has its own difficulties (routers must know which  
packets they can and can't touch) and it will take a long time before  
we can depend on routers to do this, we have to make the shim work  
without this capability.

> I'd suggest it would be more fruitful to ensure state between two  
> shim6 speakers /not/ depend on the 'outside' (ie globally routable)  
> addresses, and let shim6 be able to handle messages between two  
> endpoints arrive with any 'outside' source address.

Not sure what you mean...

> It also eliminates the need for hideous n^2 address-to-address  
> probing which is being considered.

There is no getting around it, except for giving up before you reach  
n^2...

(You can of course move this problem to another layer.)

>     - A may load-balance outbound packets between the two ISPs

Not really. Suppose A sends packets to router R which connects to  
both ISPs. How is A going to influence to which ISP R sends any given  
packet?

> However, I feel it is a *really* bad idea to even consider trying  
> to solve the problem of "internet cloud" path failures,

I'm afraid you've come to the wrong wg.  :-)

>     - Existing protocols already live with this today, and
>       generally we're happy

Are you saying there is no requirement to multihome, even for people  
who go out of business when their link to the net fails for a  
significant amount of time?

>     - It can be solved/mostly-mitigated by running a routing
>       protocol (ie BGP)

Not without an information explosion. If you have address  
dada::dead:beef, and I see in BGP that dada::/16 is up, I'm going to  
send the packet as per this route. If it then turns out that your  
link to this ISP is down the packet isn't going to make it. And  
sending out a route for dada::dead:0/112 over another ISP won't scale.

>     - If you don't need to specify complexity, don't -> KISS

Ever looked inside the cockpit of a modern airliner? Not simple. And  
these people care A LOT more about their "redundancy protocols" that  
we do.

Keeping it simple is good advice when the choice is gaining advantage  
X in a simple way or gaining advantage X in a complex way. If keeping  
it simple means you only get X/2 rather than X, then simplicity isn't  
necessarily the right choice.

>     - In trying to specify such a thing in shim6 you'll have to
>       consider interaction with underlying IP routing, yet
>       another layering violation.

Layers may interact, no violation per se.

> Please drop the locator-selection crap for now.

What "crap" are you referring to?

Iljitsch

Delivery-date: Mon, 15 Aug 2005 16:18:57 +0000
Date: Mon, 15 Aug 2005 17:18:08 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: [email-masked]
cc: [email-masked], [email-masked]
Subject: RE: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Mon, 15 Aug 2005, Paul Jakma wrote:

> - INADDR_ANY per default for its local source and the OS

Or even IN6ADDR_ANY, or whatever is appropriate for the version of IP 
'substrate' concerned.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Isn't it strange that the same people that laugh at gypsy fortune
tellers take economists seriously?

Delivery-date: Mon, 15 Aug 2005 16:10:29 +0000
Date: Mon, 15 Aug 2005 17:08:42 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: [email-masked]
cc: [email-masked], [email-masked]
Subject: RE: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Mon, 15 Aug 2005, [email-masked] wrote:

> I think we should avoid any transport layer functions in the shim 
> layer (or at least as much as possible).  I'm leaning more towards 
> the rich-api type of functionality, where the shim provides as much 
> info as it can to the transport layers, and let the transport 
> layers make decisions.

Hear hear.

I'd also like to potentially add source-address-selection (for the 
globally routable shim address) to this list.

That's something which (my gut feeling right now tells me) belongs 
*under* shim, in the final output stage of IP. Let shim specify use 
of either:

- INADDR_ANY per default for its local source and the OS
   + administratively specified or dynamically specified routes do
   their job
- an admistratively specified address (or even a set of addresses,
   but that shouldn't be required)

for the endpoint locator/outside/globally-routable address.

Again, SAS is simply *not* something shim6 should do, or something 
which it has remotely near enough information to do. Let normal IP 
routing and SAS handle this task.

I'd suggest it would be more fruitful to ensure state between two 
shim6 speakers /not/ depend on the 'outside' (ie globally routable) 
addresses, and let shim6 be able to handle messages between two 
endpoints arrive with any 'outside' source address.

It also eliminates the need for hideous n^2 address-to-address 
probing which is being considered.

Case to consider in support for this:

Shim6 endpoint, A, with 2 links to 2 ISPs, keeping some kind of 
shim6 mapping up to a remote endpoint, B:

 	    B
 	   /|\
     [ye olde internet cloud]
 	ISP1  ISP2
            \ /
 	    A

A has two 'locator' addresses, one for each ISP.

If shim6 cares not a jot which locator address A uses to send 
packets, then shim6:

- can leave local routing policy to IP routing, as $DEITY intended
   eg:
 	- A may prefer to use ISP1 for outbound packets
    	- A may run some kind of reachability-determining protocol
 	  on its ISP links and update routing to suit
 	- A may load-balance outbound packets between the two ISPs

   If shim6 assumes *nothing* about the 'outside' IP state, then shim6
   could deal very nicefully with the above.

- has no need to specify complex and/or n^2 algorithms for source
   address selection

 	- let IP output decide
 	- automatically take advantage of any local reachability
 	  information (eg routing protocols, or even a script that
 	  pings links)
 	- can cope with locator addresses changing (eg link to ISP1
 	  goes down and hence the ISP1 related locator address is
 	  unreachable to B. A's messages, if stateless from IP
 	  routing layer POV, start being sent with locator of ISP1, B
 	  gets messages from A still. If shim6 has heartbeat control
 	  messages, then state can be updated between the two
 	  reasonably quickly)

Note that this does *not* preclude A doing more sophisticated address 
selection, if it desires (eg, to work around path failures somewhere 
in "ye olde internet cloud"). However, I would suspect most 
implementations would /not/ bother trying hard to solve this.

However, I feel it is a *really* bad idea to even consider trying to 
solve the problem of "internet cloud" path failures, if it's going to 
involve horridly unscaleable algorithms, cause:

 	- This WG isn't here to solve problems of internet routing
 	- Existing protocols already live with this today, and
 	  generally we're happy
 	- Even in a shim6 world, internet-cloud path failures will
 	  still affect non-shim6 destinations.
 	- It can be solved/mostly-mitigated by running a routing
 	  protocol (ie BGP)
 	- If you don't need to specify complexity, don't -> KISS
 	- In trying to specify such a thing in shim6 you'll have to
 	  consider interaction with underlying IP routing, yet
 	  another layering violation.

Please drop the locator-selection crap for now. It's just not needed, 
certainly not for an initial shim6 specification, and if shim6 is 
designed right (no assumptions made of outside IP state), it can be 
an implementation detail, if desired.

- Keep it simple.
- Don't try to reinvent transport layers
- Don't try to tackle the problems of internet routing
- Defer the problem of solving world hunger to another day

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
printk(KERN_WARNING "Hey who turned the DMA off?\n");
         linux-2.6.6/drivers/net/wan/z85230.c

Delivery-date: Mon, 15 Aug 2005 11:37:57 +0000
Content-class: urn:content-classes:message
Subject: RE: failure detection
Date: Mon, 15 Aug 2005 14:36:57 +0300
From: <[email-masked]>
To: <[email-masked]>, <[email-masked]>

Iljitsch,

>Somewhat to my disappointment, there were no opinions about=20
>which of the two failure detection mechanisms is better,=20
>either during the sessions or afterward on the list.

At least for me, I'm still puzzeling over how to capture failure ...
Between 2 hosts, tcp traffic may work but not udp traffic (mostly due
to stupid middleboxes). Is this a failure from the shim point of view?

Also, different transports have different concepts of what 'failure'
is. When is a path considered failed? Classical TCP failure
(retransmission
of the the same packet x times)?  Too high bit-error rate?  I think we
might=20
run into circumstances where if we let the shim layer decide
conclusively
that a path has failed, it might decide much later that the path is bad
than what the transport layer knows.

I think we should avoid any transport layer functions in the shim layer
(or at least as much as possible).  I'm leaning more towards the
rich-api
type of functionality, where the shim provides as much info as it can
to the transport layers, and let the transport layers make decisions.

I'm also wondering if the failure detection mechanism should be a
pluggable
option; I've been thinking how the shim layer would work in wireless
environments,
and I don't think I'd like to have fast heartbeating, as that would
drain
batteries unnesscesarily.  Could it be possible to have this
functionality
as optional? =20

John

Delivery-date: Mon, 15 Aug 2005 06:52:56 +0000
Date: Mon, 15 Aug 2005 08:52:11 +0200
From: Brian E Carpenter <[email-masked]>
Organization: IBM
To: Iljitsch van Beijnum <[email-masked]>
CC: shim6 <[email-masked]>
Subject: Re: shim-aware transports

Iljitsch,

Iljitsch van Beijnum wrote:
> As a result of the discussion in the shim6 session in Paris it  occurred 
> to me that we've been focussing too much on making the shim  work with 
> unmodified IP and unmodified upper layers.
> 
> This is of course extremely important, no question about that.

And needs to get shipped first, to resolve the ISPs'concerns about
IPv6 deployability.

> 
> But if we're successful, it won't be long before existing transports  
> are modified or even new ones are created that know about the shim  and 
> may be able to make the shim work better. Other transports may  already 
> be doing work that the shim must also be able to do, like SCTP.
> 
> I think this implies that our efforts basically consists of two  parts, 
> or should consist of two parts:
> 
> 1. The basic shim functionality that for some reason or another  should 
> NOT be implemented in transports. For instance: reachability  detection, 
> state management and security. Having each transport do  these on its 
> own is a waste of resources and my lead to suboptimal  results if one or 
> more transports doesn't get it right.
> 
> 2. Multi-address handling and transport-specific (potential) failure  
> detection. For instance, TCP has a good idea about when its sessions  
> aren't progressing (hence the talk about hints), SCTP can also handle  
> multiple addresses.

Yes, but you need 2a: the stateful handling of all this in the shim
layer to support the unmodified ULPs. And that is more urgent than 2.

> 
> (note the difference between failure and reachability detection.)
> 
> The idea would be that the basic shim sublayer provides a set of  
> required building blocks. I'll call the other sublayer the "adaption  
> layer" because it adapts existing transport behavior to the  
> requirements of the shim. Transports or even applications that know  how 
> to work with the shim themselves may bypass the entire adaption  layer, 
> or specific parts.

Sure, that's 2a.

> 
> This would allow for the following succession in TCP/shim developments:
> 
> - Initially, TCP would be unmodified, so the shim would use the  
> adaption layer to do address magic whenever the basic shim sublayer  
> determines reachability has changed.
> 
> - Next step: TCP is modified slightly to provide reachability hints  to 
> the shim layer. The basic shim sublayer can optimize some of its  
> failure/reachability detection mechanism based on this.
> 
> - Finally: TCP becomes aware of multiple addresses, and starts  keeping 
> per-address pair RTT statistics. TCP moves to another address  pair 
> either when the shim tells it to (because the shim detected a  problem) 
> or when it determines another pair is "better" based on its  own 
> statistics.

Don't forget that SCTP already contains some of this logic, and maybe
DCCP needs to as well.

Thanks for articulating this. It's been in my mind for a while but I
never managed to say it clearly. It's the correct response to those who
say the problem should be solved in layer 4 - actually it has to be solved
by cooperation between layers 3 and 4.

BTW it implies a two-way API between the shim and the ULP, in which the
necessary events are transmitted up or down.

      Brian

Delivery-date: Sun, 14 Aug 2005 19:58:47 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: about reachability detection draft
Date: Sun, 14 Aug 2005 21:57:43 +0200
To: marcelo bagnulo braun <[email-masked]>

Ok, the long awaited reply:

On 16-jul-2005, at 17:49, marcelo bagnulo braun wrote:

> In section 2 it is stated that:

> - In the second model, a host can only detect problems in the  
> receiving
>   direction so it must depend on the correspondent to detect problems
>   in the other direction

[this is what I called forced bidirectional or FBD in the slides]

> I think it is important to consider one step further then. I mean,  
> what can a host possibly do when he detects an outage in the  
> incoming path? If a host detects an outage in the outgoing path, he  
> can change the address pair that he is using to send packets and  
> see if it solves the problem, but what can he do if he detects an  
> outage in the incoming path? I guess that the only option would be  
> to notify the correspondent node about the failure so that the  
> correspondent uses an alternative path (Note that we are asuming  
> the the case of unidirectional connectivity is possible)
> So i guess that this mode needs an additional message informing the  
> failure, which needs to be taken into account when comparing the  
> options.

Well, the idea is that when there is actually a failure in the active  
path, it's likely that there will be failures in one or more of the  
backup paths too. So in my opinion, it doesn't make sense to switch  
to a new path blindly. So we must first test any path we may want to  
switch to.

Now one of two things can happen:

1. the candidate path doesn't work either -> timeout, try another
2. the candidate path works -> we can communicate with the correspondent

If there is any connectivity left, eventually we'll end up in  
situation 2, and the fact that we're sending the correspondent these  
test packets (which are different from the regular ones that we send  
when we thing there is still connectivity, see below) tells the  
correspondent that something is wrong, so the correspondent starts  
sending test packets in our direction too.

The reason I think we need two types of test packets is because in  
the situation where there is a large number of address pairs with  
unknown reachability and/or RTT, we need to do extra work to make  
sure that when A sends (for instance) a probe A2 -> B3, which makes  
it to B, but there is only connectivity from B to A over B1 -> A4,  
it's unlikely that B1 -> A4 is the first pair B tries, so B needs to  
include reports about what it got from A in all of its packets. And  
we need a reasonable level of authentication too, because we haven't  
previously established that these addresses indeed belong to the  
correspondent.

On the other hand, when A1 <-> B1 is working happily, there is no  
need to use such a complex protocol: we are only testing one pair in  
each direction, and the correspondent has been authenticated earlier.  
If we wanted we could even use pings to determine whether this still  
works. (Well, sort of...)

> I guess it would also make sense to state how the those mechanisms  
> behave when there are no outgoing packets? i mean, i guess that in  
> any of the modes signaling is suppressed right?, however, i guess  
> that the hosts assume that the address pair is reachable, right?

With correspondent unreachability detection (first mechanism in the  
draft) the transport hints would accompany the packets, so packets =  
no, hints = no -> no action. Only packets = yes and hints = no  
requires reachability probes.

In the forced bidirectional communication we only send probes when we  
sent data recently but didn't receive data recently, so no probes in  
this case either.

Note that some transports and some applications explicitly keep the  
session alive with periodic traffic.

> Additionally, i guess that there are other information that needs  
> to be taken into account when detecting reachability, such as ICMP  
> error messages, address deprecation, lower layers information (i  
> know that you state that you assume that addresses are available,  
> but what happens if an address of the currently used address pair  
> is deprecated? or if the associated interface goes down?)

It makes sense to send a probe when there is an ICMP error (have to  
rate limit this, though). Deprecation is irrelevant, as we can  
continue to use deprecated addresses. Lower layer events are also a  
reason to do a reachability test, I think. A more interesting case is  
when an address is removed from the system. I don't remember which  
session it was, but in Paris someone was talking about how systems  
keep using addresses they no longer have because upper layers still  
use those addresses. IMO this is a feature: if I unplug my ethernet  
from my powerbook and turn on my wifi, I get the same address on a  
different interface and my sessions are still alive. Under windows,  
things like this kill your sessions immediately.

> Another issue that may be of interest is what happens with an  
> address (pair) after it becomes unreachable? i mean, is it used in  
> followings address pair explorations? is it putted in quarantine?

The way I see it there is an ordered list of address pairs. The more  
probes fail to make it to the other side the lower the address pair  
ends up on the list, I imagine.  :-)

We need to think about the situation where a fast primary link fails  
and we switch to a slow backup, though. Presumably, we'll want to  
switch back to the fast primary address pair when possible.

>   In its essence, address pair exploration is very simple: just send
>   probes using every possible address pair, wait for something to come
>   back and possibly consider the round trip time.

> I guess we agree that you are oversimplifying the issue here :-)

Well, you know me.  (-:

> I mean, the complexity is not only due to the amount of probe  
> packets that are needed, but also becuase of unidirectional  
> connectivity. I think it is very important to express such  
> difficulty. I mean, even with two address pairs, the problem can be  
> quite complex, because replies need to carry information, not only  
> about the particular incoming packet, but also from other  
> previously received probes, in order to allow the transmitter to  
> determine if previous probes have succesfully arrived. I think it  
> is important to describe this problem and possible approaches to  
> deal with it.

You're absolutely right. I think I said something about these  
subjects but it has to be fleshed out in detail.

> Finally, in the security considerations section, i think that there  
> is closely related problem that perhaps needs to be presented here  
> that is flooding protection. I mean, the path exploration exchange  
> can be used for identifying working address pairs but also for  
> preventing that the shim can be used for flooding attacks. In order  
> to enable the path exploration exchange to be used for this, you  
> need to include some additional information in the exchange, some  
> information that identifies the shim context, so that the receiver  
> of a packet of the address pair exploration process can determine  
> if this is one of its own established sessions that are being  
> genuinely rehomed or if this is a flooding attack.

Yes.

Iljitsch

Delivery-date: Sun, 14 Aug 2005 18:43:17 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: failure detection
Date: Sun, 14 Aug 2005 20:43:11 +0200
To: Iljitsch van Beijnum <[email-masked]>

Hi Iljitsch,

El 14/08/2005, a las 13:01, Iljitsch van Beijnum escribi=F3:

> Hi,
>
> Somewhat to my disappointment, there were no opinions about which of=20=

> the two failure detection mechanisms is better, either during the=20
> sessions or afterward on the list.
>

I am still trying to see the whole picture...

perhaps you could address the comments that i made a while ago in the=20
list (i know you had these in the todo list :-), especially the first=20
question about which end of the communication needs to discover the=20
outage. imho this is relevant to understand which mechanism is more=20
suitable, especially because conveying outage information from the ed=20
that discover the outae to the end that is able to recover in a=20
reliable fashion may be an issue (i a not sure but...)

(i comment on your new hybrid mechanism when i am more clear about this=20=

point then)

Inicio mensaje reenviado:

> De: marcelo bagnulo braun <[email-masked]>
> Fecha: 16 de julio de 2005 17:49:02 GMT+02:00
> Para: Iljitsch van Beijnum <[email-masked]>
> Cc: shim6 <[email-masked]>
> Asunto: about reachability detection draft
>
> Hi Iljitsch,
>
> thanks for the draft, it is very clear in its presentation
>
> i know this is 00 version, but some comments anyway of things i=20
> consider relevant.
>
>
> In section 2 it is stated that:
>
> - In the second model, a host can only detect problems in the =
receiving
>   direction so it must depend on the correspondent to detect problems
>   in the other direction
>
> I think it is important to consider one step further then. I mean,=20
> what can a host possibly do when he detects an outage in the incoming=20=

> path? If a host detects an outage in the outgoing path, he can change=20=

> the address pair that he is using to send packets and see if it solves=20=

> the problem, but what can he do if he detects an outage in the=20
> incoming path? I guess that the only option would be to notify the=20
> correspondent node about the failure so that the correspondent uses an=20=

> alternative path (Note that we are asuming the the case of=20
> unidirectional connectivity is possible)
> So i guess that this mode needs an additional message informing the=20
> failure, which needs to be taken into account when comparing the=20
> options.
>
> I guess it would also make sense to state how the those mechanisms=20
> behave when there are no outgoing packets? i mean, i guess that in any=20=

> of the modes signaling is suppressed right?, however, i guess that the=20=

> hosts assume that the address pair is reachable, right?
>
> Additionally, i guess that there are other information that needs to=20=

> be taken into account when detecting reachability, such as ICMP error=20=

> messages, address deprecation, lower layers information (i know that=20=

> you state that you assume that addresses are available, but what=20
> happens if an address of the currently used address pair is=20
> deprecated? or if the associated interface goes down?) i guess it=20
> would be important to deal with this cases also
>
> Another issue that may be of interest is what happens with an address=20=

> (pair) after it becomes unreachable? i mean, is it used in followings=20=

> address pair explorations? is it putted in quarantine?
>
> Later on in section 3 it is stated that:
>
>   In its essence, address pair exploration is very simple: just send
>   probes using every possible address pair, wait for something to come
>   back and possibly consider the round trip time.
>
> I guess we agree that you are oversimplifying the issue here :-)
> I mean, the complexity is not only due to the amount of probe packets=20=

> that are needed, but also becuase of unidirectional connectivity. I=20
> think it is very important to express such difficulty. I mean, even=20
> with two address pairs, the problem can be quite complex, because=20
> replies need to carry information, not only about the particular=20
> incoming packet, but also from other previously received probes, in=20
> order to allow the transmitter to determine if previous probes have=20
> succesfully arrived. I think it is important to describe this problem=20=

> and possible approaches to deal with it.
>
> Finally, in the security considerations section, i think that there is=20=

> closely related problem that perhaps needs to be presented here that=20=

> is flooding protection. I mean, the path exploration exchange can be=20=

> used for identifying working address pairs but also for preventing=20
> that the shim can be used for flooding attacks. In order to enable the=20=

> path exploration exchange to be used for this, you need to include=20
> some additional information in the exchange, some information that=20
> identifies the shim context, so that the receiver of a packet of the=20=

> address pair exploration process can determine if this is one of its=20=

> own established sessions that are being genuinely rehomed or if this=20=

> is a flooding attack.
>
> regards, marcelo
>
>

> Since then, I've been thinking about ways to unify the two approaches.=20=

> What I've been able to come up with is this:
>
> 1. when host A is uncertain about the reachability status of the=20
> current address pair, it sends host B a request
> 2. host B now sends back an answer, so host A knows that in the=20
> absence of this answer that something is wrong
> 3. host A may optionally request that host B send X packets at Y=20
> intervals to allow A to determine that there is reachability in the B=20=

> -> A direction
> 4. with this request, A may also request that B suppress the=20
> additional packets when there is other traffic in the B -> A direction
>
> Capabilities 1 and 2 are mandatory, capabilities 3 and 4 are optional.
>
> A brain-dead implementation would simply send a probe and expect an=20
> answer every 10 seconds or so. (I'm not sure we should allow this as=20=

> the entire failure detection mechanism as it adds unnecessary extra=20
> packets for all associations.)
>
> An implementation that uses upper layer advice would only initiate a=20=

> probe/reply sequence when the upper layer is unhappy.
>
> An implementation that knows about bidirectional data flow would only=20=

> initiate a probe/reply sequence when there is outgoing traffic but no=20=

> incoming traffic. It would probably also ask the other side to keep=20
> sending packets when it has no payload data to transmit. The other=20
> side may or may not implement this. If it does, so much the better. If=20=

> it doesn't, there will be more frequent unidirectional traffic so host=20=

> A will do a probe/reply more often.
>
> Although this increases the number of possible ways the failure=20
> detection can play out and thus makes debugging more difficult, it has=20=

> the advantage that a basic impledmentation is simple, but it allows=20
> for implementers to add additional capabilities and improve failure=20
> detection algorithms without having to go through an IETF=20
> standardization cycle.
>
> Thoughts?
>

Delivery-date: Sun, 14 Aug 2005 16:05:40 +0000
Date: Sun, 14 Aug 2005 17:04:42 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Sun, 14 Aug 2005, Iljitsch van Beijnum wrote:

>> - OS maintainers will be prepared fundamentally change their network
>>  stacks to support shim6
>
> Well, obviously they have to implement it or it won't work, no pain no gain. 
> :-)

I'd rather the bar for implementing shim6 were not that high.

>> Don't try get clever, cause the upper layer knows /more/ than 
>> shim6 does:
>
> Just now you questioned whether the upper layer knows enough to be unhappy!

I said it might not have a notion of unhappy/happy, different thing.

Anyway, such feedback typically goes /up/ the stack, not down. 
Ultimately such feedback is presented to the *user* (eg in case of a 
user trying to download something and their throughput progress meter 
is low, the user may want to try a different mirror site).

>> - It might have a list of addresses it wants to try, don't delay it
>
> It doesn't. The application may have a list of addresses, but TCP 
> or UDP only see a single one.

Uhmm.. SCTP? How do you even know what the transport has or doesn't 
have?

Further, when you're at/near the bottom of the pile, then looking up 
the only thing you will see of the application and user is the 
transport.

Stop even thinking about TCP. Forget TCP. You're specifying a 
data-layer essentially, it's job should be best-effort delivery with 
a minimal, bounded time in which to determine failure (if it's even 
possible to do so).

Leave the transport layer stuff to the "experts", ie TCP or whatever 
the hell is up the stack. There might even be some kind of 
QoS-measuring software or protocol running to make decisions about 
latency/throughput between different links and change routing between 
them.

Don't presuppose, don't add in stuff to Shim6 that doesn't need to be 
there.

> So basically you're saying that applications can multihome better 
> than the network stack? I disagree.

I didn't say that at all.

I'm raising my concerns about the scope of the problem shim6 is 
trying to solve, particularly the talk of creeping up into the 
transport. This working group is not:

- nominally an expert group on OS networking design
- chartered to investigate data-layer/transport interaction

Shim6 is here to provide a shim layer for IPv6 multihoming soltuion 
that can operate (from the charter) "... without upsetting transport 
protocols or applications."

I'd suggest the WG follow this advice. ;)

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
... The prejudices people feel about each other disappear when then get
to know each other.
 		-- Kirk, "Elaan of Troyius", stardate 4372.5

Delivery-date: Sun, 14 Aug 2005 15:40:55 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: failure detection
Date: Sun, 14 Aug 2005 17:40:33 +0200
To: Paul Jakma <[email-masked]>

On 14-aug-2005, at 16:49, Paul Jakma wrote:

>> An implementation that uses upper layer advice would only initiate  
>> a probe/reply sequence when the upper layer is unhappy.

> This kind of thing worries me.

Apparently... I don't really get why, though.

> Assuming it will be possible to have the upper layer 'direct' shim6  
> is assuming:

> - OS maintainers will be prepared fundamentally change their network
>   stacks to support shim6

Well, obviously they have to implement it or it won't work, no pain  
no gain.  :-)

> - Worse, assuming upper layers even /have/ a notion of being
>   unhappy or happy.

I'm sure there will be ones that have such a notion and equally sure  
there are ones that don't.

> Don't try get clever, cause the upper layer knows /more/ than shim6  
> does:

Just now you questioned whether the upper layer knows enough to be  
unhappy!

> - It might have a list of addresses it wants to try, don't delay it

It doesn't. The application may have a list of addresses, but TCP or  
UDP only see a single one.

> It is *much* better for the data-layer to signal *up*:

>     "sorry, it's unreachable"

> as soon as it can, and let the transport or the application or  
> maybe even the *user* get on with trying potential alternatives  
> than for the data-layer to try aimlessly continue on for a  
> significant period of time with complicated reachability tests.

So basically you're saying that applications can multihome better  
than the network stack? I disagree.

Delivery-date: Sun, 14 Aug 2005 14:51:08 +0000
Date: Sun, 14 Aug 2005 15:49:04 +0100 (IST)
From: Paul Jakma <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: failure detection
Mail-Copies-To: [email-masked]
Mail-Followup-To: [email-masked]

On Sun, 14 Aug 2005, Iljitsch van Beijnum wrote:

> An implementation that uses upper layer advice would only initiate 
> a probe/reply sequence when the upper layer is unhappy.

This kind of thing worries me. Assuming it will be possible to have 
the upper layer 'direct' shim6 is assuming:

- OS maintainers will be prepared fundamentally change their network
   stacks to support shim6

- Worse, assuming upper layers even /have/ a notion of being
   unhappy or happy.

I don't think designing shim6 to accomodate either of those 
assumptions is a good idea at this time, or anywhere in the near to 
mid term future.

It's a data-layer: Provide best-effort packet delivery in a low-order 
*bounded* magnitude of time. Bounded time is possibly more important 
than best-effort.

If there are multiple paths available, let shim6 do heartbeats of 
them all at startup and X interval thereafter, pick whichever one 
seems best for sending packets down; Keep a note of which other paths 
seem to be available; if a path starts failing, pick another within 
some bounded time.

Don't try get clever, cause the upper layer knows /more/ than shim6 
does:

- It might have a list of addresses it wants to try, don't delay it

- At the /application/ or even *user* layer, there might be other
   alternatives, eg:
 	- VoIP: if the users call to foo@multi-site doesnt initiate
 	  the user might know they should try joe@other-site

It is *much* better for the data-layer to signal *up*:

 	"sorry, it's unreachable"

as soon as it can, and let the transport or the application or maybe 
even the *user* get on with trying potential alternatives than for 
the data-layer to try aimlessly continue on for a significant period 
of time with complicated reachability tests.

Shim6 will have the *least* amount of information about where the 
user is trying to get packets to.

Adding overcomplicated multi-pathing reachability/failover to shim6, 
when it will be the *least* informed part of the stack, is daft. 
Failure *IS* an option, and it's preferable to signal failure up 
*sooner* rather than later.

(and again, I hope this WG will disabuse itself of the notion that 
extra signals /down/ from transports to shim6 can even be considered 
at this early stage in shim6's life).

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
You will be married within a year, and divorced within two.

Delivery-date: Sun, 14 Aug 2005 12:55:22 +0000
To: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: shim-aware transports
Date: Sun, 14 Aug 2005 14:54:49 +0200

As a result of the discussion in the shim6 session in Paris it  
occurred to me that we've been focussing too much on making the shim  
work with unmodified IP and unmodified upper layers.

This is of course extremely important, no question about that.

But if we're successful, it won't be long before existing transports  
are modified or even new ones are created that know about the shim  
and may be able to make the shim work better. Other transports may  
already be doing work that the shim must also be able to do, like SCTP.

I think this implies that our efforts basically consists of two  
parts, or should consist of two parts:

1. The basic shim functionality that for some reason or another  
should NOT be implemented in transports. For instance: reachability  
detection, state management and security. Having each transport do  
these on its own is a waste of resources and my lead to suboptimal  
results if one or more transports doesn't get it right.

2. Multi-address handling and transport-specific (potential) failure  
detection. For instance, TCP has a good idea about when its sessions  
aren't progressing (hence the talk about hints), SCTP can also handle  
multiple addresses.

(note the difference between failure and reachability detection.)

The idea would be that the basic shim sublayer provides a set of  
required building blocks. I'll call the other sublayer the "adaption  
layer" because it adapts existing transport behavior to the  
requirements of the shim. Transports or even applications that know  
how to work with the shim themselves may bypass the entire adaption  
layer, or specific parts.

This would allow for the following succession in TCP/shim developments:

- Initially, TCP would be unmodified, so the shim would use the  
adaption layer to do address magic whenever the basic shim sublayer  
determines reachability has changed.

- Next step: TCP is modified slightly to provide reachability hints  
to the shim layer. The basic shim sublayer can optimize some of its  
failure/reachability detection mechanism based on this.

- Finally: TCP becomes aware of multiple addresses, and starts  
keeping per-address pair RTT statistics. TCP moves to another address  
pair either when the shim tells it to (because the shim detected a  
problem) or when it determines another pair is "better" based on its  
own statistics.

Delivery-date: Sun, 14 Aug 2005 11:02:48 +0000
To: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: failure detection
Date: Sun, 14 Aug 2005 13:01:03 +0200

Hi,

Somewhat to my disappointment, there were no opinions about which of  
the two failure detection mechanisms is better, either during the  
sessions or afterward on the list.

Since then, I've been thinking about ways to unify the two  
approaches. What I've been able to come up with is this:

1. when host A is uncertain about the reachability status of the  
current address pair, it sends host B a request
2. host B now sends back an answer, so host A knows that in the  
absence of this answer that something is wrong
3. host A may optionally request that host B send X packets at Y  
intervals to allow A to determine that there is reachability in the B  
-> A direction
4. with this request, A may also request that B suppress the  
additional packets when there is other traffic in the B -> A direction

Capabilities 1 and 2 are mandatory, capabilities 3 and 4 are optional.

A brain-dead implementation would simply send a probe and expect an  
answer every 10 seconds or so. (I'm not sure we should allow this as  
the entire failure detection mechanism as it adds unnecessary extra  
packets for all associations.)

An implementation that uses upper layer advice would only initiate a  
probe/reply sequence when the upper layer is unhappy.

An implementation that knows about bidirectional data flow would only  
initiate a probe/reply sequence when there is outgoing traffic but no  
incoming traffic. It would probably also ask the other side to keep  
sending packets when it has no payload data to transmit. The other  
side may or may not implement this. If it does, so much the better.  
If it doesn't, there will be more frequent unidirectional traffic so  
host A will do a probe/reply more often.

Although this increases the number of possible ways the failure  
detection can play out and thus makes debugging more difficult, it  
has the advantage that a basic impledmentation is simple, but it  
allows for implementers to add additional capabilities and improve  
failure detection algorithms without having to go through an IETF  
standardization cycle.

Thoughts?

Delivery-date: Tue, 09 Aug 2005 05:11:25 +0000
Date: Tue, 09 Aug 2005 15:10:27 +1000
From: Greg Daley <[email-masked]>
Subject: Re: policy information in dhcp
To: Francis Dupont <[email-masked]>
Cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Reply-to: [email-masked]
Organization: Monash University

Hi,

Francis Dupont wrote:
>  In your previous mail you wrote:
> 
>    We may want to formulate a position as a wg on this.
>    
> => when I did some experiments on multi-homing the first thing I did
> was to specialize the policy table on all involved hosts. So I strongly
> support you!

There's a discussion about methods (DHC or ND) going on in IPv6 WG,
but I think that the idea is valid (providing information to hosts
about the usefulness or stability of addresses).

I'd prefer if the discussion concentrated on support for the idea,
rather than the delivery method, until the discussion in IPv6 concludes.

Greg

Envelope-to: [email-masked]
Delivery-date: Thu, 04 Aug 2005 13:18:21 +0000
Message-Id: <[email-masked]>
From: Francis Dupont <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
cc: shim6 <[email-masked]>
Subject: Re: policy information in dhcp 
Date: Thu, 04 Aug 2005 15:17:43 +0200

 In your previous mail you wrote:

   We may want to formulate a position as a wg on this.
   
=> when I did some experiments on multi-homing the first thing I did
was to specialize the policy table on all involved hosts. So I strongly
support you!

Thanks

[email-masked]

Delivery-date: Thu, 04 Aug 2005 12:30:08 +0000
From: Arifumi Matsumoto <[email-masked]>
Subject: Re: policy information in dhcp
Date: Thu, 4 Aug 2005 21:29:38 +0900
To: shim6 <[email-masked]>

Iljitsch,
thank you for your comments at dhc session.

On 2005/08/04, at 18:49, Iljitsch van Beijnum wrote:

> The dhc wg just discussed taking on http://www1.tools.ietf.org/tools/rfcmarkup/rfcmarkup.cgi?repository=http://internet-drafts.org&draft=draft-fujisaki-dhc-addr-select-opt-00.txt as a working group item. The idea is to configure the IPv6 policy table from DHCPv6.
>
> Margaret said that it's not like the ipv6 wg is asking for this. However, I think this, or something like it, would be very useful when deploying the shim, for instance to avoid hitting ingress filtering needlessly or to do traffic engineering.
>
> We may want to formulate a position as a wg on this.
>

Actually, I presented the former version of our draft at multi6 session in IETF61st meeting and directed to goto ipv6 wg. At that time, my proposal was mainly for a multihoming site and also a kind of multihoming called "global and closed mixed connectivity".

Now, thanks to some peoples' comments, it is modified as a more generic mechanism for delivering address selection policy table and now serves as a solution for some more problems that have roots in multi-address environment such as IPv4/6 prioritization and renumbering transition.

So, what I want to say is that this proposal was intended to solve multihoming problems at first. If anybody has interest in this kind of mechanism, I'm glad to co-work on this.

--
Arifumi Matsumoto <[email-masked]>

Delivery-date: Thu, 04 Aug 2005 09:50:08 +0000
To: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: policy information in dhcp
Date: Thu, 4 Aug 2005 11:49:28 +0200

The dhc wg just discussed taking on http://www1.tools.ietf.org/tools/ 
rfcmarkup/rfcmarkup.cgi?repository=http://internet- 
drafts.org&draft=draft-fujisaki-dhc-addr-select-opt-00.txt as a  
working group item. The idea is to configure the IPv6 policy table  
from DHCPv6.

Margaret said that it's not like the ipv6 wg is asking for this.  
However, I think this, or something like it, would be very useful  
when deploying the shim, for instance to avoid hitting ingress  
filtering needlessly or to do traffic engineering.

We may want to formulate a position as a wg on this.

Iljitsch

Delivery-date: Thu, 04 Aug 2005 06:39:11 +0000
Date: Thu, 04 Aug 2005 16:37:49 +1000
To: shim6 <[email-masked]>
From: Geoff Huston <[email-masked]>
Subject: SHIM6 WG meeting notes from IETF 63
Cc: kurt Erik Lindqvist <[email-masked]>

This is the draft of the notes of the SHIM6 meeting at IETF 63.

Thanks to Spencer for preparing these notes.

Are there any comments on these notes as a record of our meeting?

Thanks,

    Kurtis & Geoff

-------------------------------------------------------------------------

Site multi-homing by IPv6 Intermediation WG (shim6)

Tuesday, August 2 at 0900-1000
Wednesday, August 3 at 0900-1000
================================

CHAIRS: Geoff Huston <[email-masked]>
         Kurtis Lindqvist <[email-masked]>

Notes: Spencer Dawkins <[email-masked]>

SUMMARY

   Initial WG meeting. There are five areas of design activity namely
   protocol, triggers, crypto-locator use, architecture and applicability.
   Initial drafts in these areas are based on the concluding design team
   work in the Multi6 Working Group

   The current status on each of these areas was presented to the Working
   Group for review.

   Next steps will be based on further refinement of five working group
   documents, using lead authors / editors assigned to each draft.

ACTIONS

   * Pekka Nikkander, John Loughney and Christian Huitema
     to prepare some initial ideas (slide pack / draft) on a bi-directional
     SHIM / ULP API that would include the ability for the ULP to
     obtain/share locator pair path info and then express a locator pair
     preference to the SHIM (i.e. keep the notion of binding sessions at the
     transport level but allow ULP signaling to include Loc Pair
     preferences to be expressed) in addition to sending the SHIM the
     trigger / heartbeat information associated with the current locator
     pair.

NOTES

1. Agenda Bashing

    - A very tight agenda.
    - no changes proposed.
    - Noted that the SHIM6 Charter discussion has already happened...

2. Review of status and proposed work areas

    Overview
       Kurtis Lindqvist

    MULTI6 drafts moving to SHIM6 (with very few changes).

      - Charter is very aggressive.

      - A lot of questions are going to pop up as we work on a protocol,
        including consideration of state management, identifier
        characteristics, locator pair detection, etc.

    Architecture
      document: draft-ietf-shim6-arch-00.txt
      Geoff Huston

      - MULTI6 architecture draft is in RFC Editor Queue now, this is trying
        to be a taxonomy as well (what layer, where identifiers are
        rewritten, etc.)

      - Includes a discussion of identities (ephemeral, persistent, etc.)
        and implications.

      - Includes a list of things that need to happen in a protocol that
        does SHIM6 - how do you set up session state? how do you know to
        re-home? what is Identity Equivalence? How do you select locators?
        How do you remove state?

          Aaron Falk: is this restoration and repair, or multi-homing? Open
          question, but this is unicast, not multicast, so assuming
          alternate paths are idle at session layer.

          Jason Shiller: has to take into account more than just restoration
          and repair - we can do load balancing in IPv4 today, for example.
          Need to get the basics right first, though. There is discussion
          about traffic engineering, but we'll need to do more.

      - Initial draft, is incomplete - add equivalent state and design
        tradeoffs, at a minimum.

      - SHIM6 is in IP layer, and is actually below things like
        fragmentation/reassembly, AH/ESP, etc.

      - SHIM6 uses "the first" locator as the identifier, but can map other
        locators to the same identifier - identifiers should be referable.

      - Need more text on maintaining state in this draft. At IP level, we
        don't have all the information that would be available at transport
        or at application levels.

      - Routing information isn't in hosts today - do site exit routers need
        to signal hosts?

      - Lots of possible triggers - what triggers are valid?

      - Vertical signaling so you know when you can release state
        information?

      - Need to make decisions about dynamic locator sets - this would
        affect a lot.

      - Are there "distinguished locators" you always use to get a session
        up?

      - Is this purely IP datagrams?

      - Next steps - review SHIM6 contributions, solicit explicit answers
        from document editors, but this draft will be active for a while.

    Protocol
      document: draft-ietf-shim6-l3shim-00.txt
                draft-ietf-shim6-functional-dec-00.txt
      Erik Nordmark

      - We will not have a distinct namespace for identifiers.

      - We will not assume that a single FQDN is a single host (could be a
        multi-host service) - we'll try multiple initial locators until one
        works. Telnet does this today, so it's at least a plausible starting
        point.

      - Make sure we minimize failure impact during session initialization.

      - Don't force every connection to set up SHIM6 state.

      - Context establishment is a four-packet exchange - looks a lot like
        HIP exchange.

      - Include DOS protection on first packet received.

      - Since these were MULTI6 drafts, changes have been editorial.

      - Several open issues  - packet formats (including demultiplexing),
        state management, packet formats for control protocol, APIs for
        advice, path maintenance and exploration protocol), etc.

      - Pick one possible starting point and work out the details.
        Alternative would be using 8-byte extension header for data packets
        after failover.

          Pekka Savola - Do we need functional decomposition document? Seems
          redundant...

          Jim Bound - If we change the upper layer identifier, we break TCP.
          A lot of this stuff is already in SCTP now. Are we going to stay
          compatible with the old APIs? SCTP is in most of our stacks now.
          Are we going to dynamically change the ULID? Erik - no, locator
          changes won't be visible to TCP (for example). It is interesting
          to wonder about what SCTP over SHIM6 looks like... We're talking
          about adding a lot of code to production stacks.

          Pekka Nikander - If we are doing something at the IP layer, let
          the session-level state still be at the transport layer. We need
          to figure out SCTP over SHIM6. Why do we need anything in the IP
          header at all? After you have a failure, you want to be able to
          undo the address-swaps at the receiver correctly.

          Christian Huitema - there is a LOT of interaction with transport
          protocols. If we are assuming there's no impact, that's probably a
          mistake. Look at what should change in transport protocol if it's
          to take advantage of SHIM6. Need to interact with TSV working
          groups.

          Iljitsch van Beijnum - there are a handful of transports and
          millions of applications, and we are already concerned about
          changing transports...

          ??? If you're hiding multiple addresses, this may be sub-optimal.

          Pekka Nikander - as an implementation option, maybe you don't re-
          write the IP addresses, you pass them unchanged to SCTP? SCTP and
          TCP interfaces may look different in the future. May also need to
          think about path congestion versus session congestion - but this
          is research.

           Christian Huitema - there was running code several years ago for
           TCP address changing - the issue was making sure you weren't
           being spoofed when the addresses change.

    Crytpo Locators
       document: draft-ietf-shim6-hba-00.txt
                 refers to draft-ietf-send-cga-06.txt
       Marcelo Bagnulo, Jari Arkko

       - Concerns are preventing hijacking and flooding attacks in
         multihomed environments - generate a set of securely established
         prefixes

       - Idea is to contain information about the set in the identifiers
         themselves

       - Using the IID bits (like CGAs) for HBAs - HBA would be a CGA
         extension

       - Includes a mechanism to add prefixes dynamically using hashes and
         PK operations.

          Dave Thaler - How does this work with temporary addresses/private
          addresses?
          response: HVA set is fixed for a given communication, but you can
          have multiple HVAs.

          Francis Dupont - IPR status of HVA?
          response: No KNOWN IPR on this :-)

    Triggers
       document: draft-ietf-shim6-reach-detect-00.txt
       Iljitsch van Beijnum

       - goals - detect failures, and route around them ...

       - note that "address pair" is a unidirectional path.

       - Ping all source/dest pairs and pick best RTT or some other metric?
         Doesn't detect unidirectional paths, doesn't scale well (M x N).

       - Need better answer, so need more complexity - detect unidirectional
         paths, stop probing when you find a reasonable path.

       - Need heuristics (this is hard work), need a lightweight protocol.

       - "CUD", similar to Neighbor Unreachability Detection, or "FBD",
         requiring bidirectional communication (if you don't see
         bidirectional communication, do full reachability exploration).

       - CUD needs ULP feedback, FBD adds packets in an otherwise idle
         direction.

       - CUD detects failure in either direction, FBD in inbound direction.

       - Several choices on granularity (host to host, locator to locator,
         ULID to ULID, etc.).

       - NATs and Firewalls - protocol may be reused for IPv4, so ... would
         like IP options for fate-sharing, but this doesn't work with
         NATs/firewalls. How important is sneaking past firewalls?

       - Security - don't want too much reachability detection, but don't
         want too little :-) Must be lightweight!

       - Combine with draft on failure detection

          John Loughney - need to clarify (on the list) how to go forward
          based on architecture draft (transport layer? etc.) -
          response: this is in the absence of session information coming
          through transports, etc. - not either/or. Transports use ULIDs,
          not locators, so the point where mapping from ULID to locator is
          at the SHIM.

          Hesham - Can you assume that all the addresses on an interface are
          reachable?
          response: This is site multi-homing, not host multi-homing, so ...
          not in the SHIM6 context.

          Spencer Dawkins - can people send info on unidrectional
          applications to the list?

          Jason Shiller - is this just reachability, or degradation? Don't
          know the answer at this time.

          Tony Hain - assuming all links are equal? Path selection may
          depend on application (short T1 or long OC-48?) - use different
          ULIDs for different applications?

    Applicability
       documents: draft-ietf-shim6-applicability-00.txt
                  draft-ietf-shim6-app-refer-00.txt
       Joe Abley (reported by Geoff Huston)

       - very short draft, mostly placeholders at this time!

3. Discussion about approach

      - Geoff looking for co-author assistance with architecture draft

       - At IP level, all paths are the same - do we need a richer set of
         choices? Need to do design work here, especially for triggers.

       - Kurtis - Please put ideas into existing drafts, not into new
         drafts!

       - Christian Huitema - transport protocols determine whether a path is
         "good enough". Do we want to do this in IP, or expose information
         to transports that do it? What if we don't have a translation layer
         at all, and transports start doing this normally?

       - Iljitsch - need to do this at some point, but don't do it now -
         it's too early.

       - Hesham - what about applications?

       - Christian - two ways to solve this problem - API to push decision
         mechanisms, or keep IP the way it is. Not the same thing at all.

       - Hesham - in addition, there's also the factor of who makes the
         decision.

       - Pekka Nikander - yes, but most of this is implementation issue, not
         protocol issue. Simulation would be good!

       - Spencer - this is like the IAB Addressing workshop in Vienna -
         every layer does everything, this is not what we really want. Need
         to be really crisp here.

       - John Loughney - NS2 is good. We're talking about policy here - need
         to work through all of this from an application point of view.

4. Additional Items:

    Interactions between Shim6 and MIPv6
      document: draft-bagnulo-shim6-mip-00.txt
      Marcelo Bagnulo

      - This draft is informational for the working group at this point.

      - Need to understand interactions (SHIM6 over MIP6).

      - BT mode forces all traffic through home agent, if there is a
        failure, so SHIM6 detects failure and uses alternative home address.
        Or change care-of addresses?

      - RO mode give you two sets of paths (optimized and un optimized
        paths). One of the paths may fail, but not the other. SHIM6 uses
        alternative locator, same issue.

      - Should SHIM6 know about HoA/CoA?

          Spencer - OK, I was hoping that when I said each layer was in
          business for itself, I was hoping that was a bounded set of
          problems...

      - Charter says, "Don't break MIP6"...

          John Loughney - which one is on top? there is no assumption at
          this point. Mailing list to resolve this.

      - Do an API before November IETF?

    L3Shim state management using Vertical layered Communication
      document: draft-you-shim6-L3Shim-state-meagement-00.txt
      You Taewan

      - TCP Slow start is needed after path switch?

      - Will propose state management in more detail - if other protocol
        layer MAY be modified, will propose specific method.

          Spencer was going to say  - TRIGTRAN answer was that new path
          could be better or worse, simpler to let TCP figure it out rather
          than try to listen to hints. Mark Allman and Joe Touch were good
          sources on this. Answer may have changed, but that's what it was
          18 months ago.

Delivery-date: Tue, 02 Aug 2005 14:32:58 +0000
To: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: my slides for tomorrow
Date: Tue, 2 Aug 2005 16:32:13 +0200

For those of you who don't like taking your eyes off your laptop  
screens...

http://www.xs4all.nl/~iljitsch/shim6-reach-detect-00.pdf

Delivery-date: Tue, 02 Aug 2005 07:39:37 +0000
Content-class: urn:content-classes:message
Subject: RE: I-D ACTION:draft-you-shim6-l3shim-state-management-00.txt
Date: Tue, 2 Aug 2005 00:38:36 -0700
From: "Christian Huitema" <[email-masked]>
To: "Aaron Falk" <[email-masked]>
Cc: "shim6" <[email-masked]>

During the shim6 question, Aaron asked whether it was common practice
for hosts to acquire multiple addresses. Well, yes. Here is the output
of "ipconfig" on my laptop:

   Wireless Network Connection:

   Connection-specific DNS Suffix  . : ietf63.ietf.org
   IPv6 Address. . . . . . . . . . . :
2001:688:ffff:24:20e:35ff:fe78:76f3
   Temporary IPv6 Address. . . . . . :
2001:688:ffff:24:804a:a17d:386:b038
   IPv6 Address. . . . . . . . . . . :
2002:56ff:1c1c:8:20e:35ff:fe78:76f3
   Temporary IPv6 Address. . . . . . :
2002:56ff:1c1c:8:45cf:f22c:8119:db6
   IPv6 Address. . . . . . . . . . . :
2002:56ff:1f49:6:20e:35ff:fe78:76f3
   Temporary IPv6 Address. . . . . . :
2002:56ff:1f49:6:d5b6:4684:26e0:8ea
   Link-local IPv6 Address . . . . . : fe80::20e:35ff:fe78:76f3%47
   Site-local IPv6 Address . . . . . : fec0::6:20e:35ff:fe78:76f3%1
   Site-local IPv6 Address . . . . . : fec0::8:20e:35ff:fe78:76f3%1
   IPv4 Address. . . . . . . . . . . : 86.255.28.34
   Subnet Mask . . . . . . . . . . . : 255.255.248.0
   Default Gateway . . . . . . . . . : fe80::20f:8fff:fedf:e647%47
                                       fe80::204:23ff:fe7a:fb3e%47
                                       fe80::204:23ff:fe78:827e%47
                                       86.255.24.1

-- Christian Huitema