Subject: Re: fwd: WG Action: Site Multihoming by IPv6 Intermediation (shim6)
From: Geoff Huston <[email-masked]>
To: [email-masked]
Date: Fri, 01 Jul 2005 06:51:06 +1000

A minor correction:

The Secretariat's  announcement omitted Thomas Narten as the other 
technical reviewer
for this working group

thanks,

   Geoff

At 10:56 AM 30/06/2005, Geoff Huston wrote:
>The following Secretariat message bounced when posted to the shim6 mailing 
>list
>
>   Geoff
>
>------------------
>
>
>A new IETF working group has been formed in the Internet Area. For additional
>information, please contact the Area Directors or the WG Chairs.
>
>+++
>
>Site Multihoming by IPv6 Intermediation (shim6)
>================================================
>
>Current Status: Active Working Group
>
>Chair(s):
>Kurt Lindqvist <[email-masked]>
>Geoff Huston <[email-masked]>
>
>Internet Area Director(s):
>Mark Townsley <[email-masked]>
>Margaret Wasserman <[email-masked]>
>
>Internet Area Advisor:
>Margaret Wasserman <[email-masked]>
>
>Technical Advisor(s):
>Dave Meyer <[email-masked]>
>
>Mailing Lists:
>General Discussion: [email-masked]
>To Subscribe: [email-masked]
>Archive:
>
>Description of Working Group:
>For the purposes of redundancy, load sharing, operational policy or
>cost, a site may be multi-homed, with the site's network having
>connections to multiple IP service providers. The current Internet
>routing infrastructure permits multi-homing using provider independent
>addressing, and adapts to changes in the availability of these
>connections. However if the site uses multiple provider-assigned
>address
>prefixes for every host within the site, host application associations
>cannot use alternate paths, such as for surviving the changes or for
>creating new associations, when one or more of the site's address
>prefixes becomes unreachable. This working group will produce
>specifications for an IPv6-based site multi-homing solution that
>inserts a new sub-layer (shim) into the IP stack of end-system hosts.
>It
>will enable hosts on multi-homed sites to use a set of provider-
>assigned
>IP address prefixes and switch between them without upsetting transport
>protocols or applications.
>
>The work will be based on the architecture developed by the IETF multi6
>working group. The shim6 working group is to complete the required
>protocol developments and the architecture and security analysis of the
>required protocols.
>
>Requirements for the solution are:
>
>o The approach must handle re-homing both existing communication and
>being able to establish new communication when one or more of the
>addresses is unreachable.
>
>o IPv6 NAT devices are assumed not to exist, or not to present an
>obstacle about which the shim6 solution needs to be concerned.
>
>o Only IPv6 is considered.
>
>o Changes in the addresses that are used below the shim will be
>invisible to the upper layers, which will see a fixed address (termed
>Upper Layer Identifier or ULID).
>
>o ULIDs will be actual IP addresses, permitting existing applications
>to
>continue to work unchanged, and permitting application referrals to
>work, as long as the IP Addresses are available.
>
>o The solution should assume ingress filtering may be applied at
>network
>boundaries.
>
>o The solution must allow the global routing system to scale even if
>there is a very large number of multi-homed sites. This implies that
>re-homing not be visible to the routing system.
>
>o Compatibility will remain for existing mobility mechanisms. It will
>be possible to use Mobile IPv6 on a node that also supports Shim6.
>However, any optimizations or advanced configurations are out of
>scope for shim6.
>
>o The approach is to provide an optimized way to handle a static set of
>addresses, while also providing a way to securely handle dynamic
>changes in the set of addresses. The dynamic changes might be useful
>for future combinations of multi-homing and IP mobility, but the
>working
>group will not take on such mobility capabilities directly.
>
>o The specifications must specifically refer to all applicable threats
>and describe how they are handled, with the requirement being that the
>resulting solution not introduce any threats that make the security any
>less than in today's Internet.
>
>The background documents to be considered by the WG include:
>
>RFC 3582
>draft-ietf-multi6-architecture-04.txt
>draft-ietf-multi6-things-to-think-about-01.txt
>draft-ietf-multi6-multihoming-threats-03.txt
>
>The input documents that the WG will use as the basis for its design
>are:
>
>draft-huston-l3shim-arch-00.txt
>draft-ietf-multi6-functional-dec-00.txt
>draft-ietf-multi6-l3shim-00.txt
>draft-ietf-multi6-failure-detection-00.txt
>draft-ietf-multi6-hba-00.txt
>draft-ietf-multi6-app-refer-00.txt
>
>In addition to the network layer shim solution, the shim6 WG is
>specifically chartered to work on:
>
>o Solutions for site exit router selection that work when each ISP
>uses ingress filtering, i.e. when the chosen site exit needs to
>be related to the source address chosen by the host. This site
>exit router selection and the associated address selection
>process should work whether or not the peer site supports
>the shim6 protocol.
>
>o Solutions to establish new communications after an outage has
>occurred that do not require shim support from the
>non-multihomed end of the communication. The Working Group will
>explore whether such solutions are also useful when both ends
>support the shim.
>
>o The possible impact of the use of multiple locators at both ends
>on congestion control, traffic engineering, and QoS will be analysed
>in conjunction with the Transport Area.
>
>o The relationships between Upper Layer Identifiers (ULIDs)
>and unique local addresses.
>
>o ICMP error demuxing for locator failure discovery.
>
>o If necessary, develop and specify formats and structure for:
>
>- Cryptographically protected locators
>
>- Carrying the flow label across the shim layer
>defined in the multi6 architecture.
>
>The shim6 WG is to publish, as standards track RFC's, specifications
>with enough details to allow fully interoperable implementations.
>
>Goals and Milestones:
>Aug 05    First draft of architectural document
>Aug 05    First draft of protocol document
>Aug 05    First draft on cryptographic locators, if required
>Aug 05    First draft on multi-homing triggers description
>Aug 05    First draft on applicability statement document
>Oct 05    WG last-call on architectural document
>Oct 05    WG last-call on applicability statement document
>Feb 06    WG last-call on protocol document
>Feb 06    WG last-call on cryptographic locators, if required
>Feb 06    Submit completed architectural document to IESG
>Feb 06    Submit applicability statement document to IESG
>Apr 06    WG last-call on multihoming triggers description
>Apr 06    Submit document on cryptographic locators to the IESG, if required
>Apr 06    Submit protocol document to the IESG
>Jun 06    Submit draft on multihoming triggers description to the IESG
>
>
>

Delivery-date: Thu, 30 Jun 2005 00:57:35 +0000
Date: Thu, 30 Jun 2005 10:56:59 +1000
To: [email-masked]
From: Geoff Huston <[email-masked]>
Subject: fwd: WG Action: Site Multihoming by IPv6 Intermediation (shim6)

The following Secretariat message bounced when posted to the shim6 mailing list

   Geoff

------------------

A new IETF working group has been formed in the Internet Area. For additional
information, please contact the Area Directors or the WG Chairs.

+++

Site Multihoming by IPv6 Intermediation (shim6)
================================================

Current Status: Active Working Group

Chair(s):
Kurt Lindqvist <[email-masked]>
Geoff Huston <[email-masked]>

Internet Area Director(s):
Mark Townsley <[email-masked]>
Margaret Wasserman <[email-masked]>

Internet Area Advisor:
Margaret Wasserman <[email-masked]>

Technical Advisor(s):
Dave Meyer <[email-masked]>

Mailing Lists:
General Discussion: [email-masked]
To Subscribe: [email-masked]
Archive:

Description of Working Group:
For the purposes of redundancy, load sharing, operational policy or
cost, a site may be multi-homed, with the site's network having
connections to multiple IP service providers. The current Internet
routing infrastructure permits multi-homing using provider independent
addressing, and adapts to changes in the availability of these
connections. However if the site uses multiple provider-assigned
address
prefixes for every host within the site, host application associations
cannot use alternate paths, such as for surviving the changes or for
creating new associations, when one or more of the site's address
prefixes becomes unreachable. This working group will produce
specifications for an IPv6-based site multi-homing solution that
inserts a new sub-layer (shim) into the IP stack of end-system hosts.
It
will enable hosts on multi-homed sites to use a set of provider-
assigned
IP address prefixes and switch between them without upsetting transport
protocols or applications.

The work will be based on the architecture developed by the IETF multi6
working group. The shim6 working group is to complete the required
protocol developments and the architecture and security analysis of the
required protocols.

Requirements for the solution are:

o The approach must handle re-homing both existing communication and
being able to establish new communication when one or more of the
addresses is unreachable.

o IPv6 NAT devices are assumed not to exist, or not to present an
obstacle about which the shim6 solution needs to be concerned.

o Only IPv6 is considered.

o Changes in the addresses that are used below the shim will be
invisible to the upper layers, which will see a fixed address (termed
Upper Layer Identifier or ULID).

o ULIDs will be actual IP addresses, permitting existing applications
to
continue to work unchanged, and permitting application referrals to
work, as long as the IP Addresses are available.

o The solution should assume ingress filtering may be applied at
network
boundaries.

o The solution must allow the global routing system to scale even if
there is a very large number of multi-homed sites. This implies that
re-homing not be visible to the routing system.

o Compatibility will remain for existing mobility mechanisms. It will
be possible to use Mobile IPv6 on a node that also supports Shim6.
However, any optimizations or advanced configurations are out of
scope for shim6.

o The approach is to provide an optimized way to handle a static set of
addresses, while also providing a way to securely handle dynamic
changes in the set of addresses. The dynamic changes might be useful
for future combinations of multi-homing and IP mobility, but the
working
group will not take on such mobility capabilities directly.

o The specifications must specifically refer to all applicable threats
and describe how they are handled, with the requirement being that the
resulting solution not introduce any threats that make the security any
less than in today's Internet.

The background documents to be considered by the WG include:

RFC 3582
draft-ietf-multi6-architecture-04.txt
draft-ietf-multi6-things-to-think-about-01.txt
draft-ietf-multi6-multihoming-threats-03.txt

The input documents that the WG will use as the basis for its design
are:

draft-huston-l3shim-arch-00.txt
draft-ietf-multi6-functional-dec-00.txt
draft-ietf-multi6-l3shim-00.txt
draft-ietf-multi6-failure-detection-00.txt
draft-ietf-multi6-hba-00.txt
draft-ietf-multi6-app-refer-00.txt

In addition to the network layer shim solution, the shim6 WG is
specifically chartered to work on:

o Solutions for site exit router selection that work when each ISP
uses ingress filtering, i.e. when the chosen site exit needs to
be related to the source address chosen by the host. This site
exit router selection and the associated address selection
process should work whether or not the peer site supports
the shim6 protocol.

o Solutions to establish new communications after an outage has
occurred that do not require shim support from the
non-multihomed end of the communication. The Working Group will
explore whether such solutions are also useful when both ends
support the shim.

o The possible impact of the use of multiple locators at both ends
on congestion control, traffic engineering, and QoS will be analysed
in conjunction with the Transport Area.

o The relationships between Upper Layer Identifiers (ULIDs)
and unique local addresses.

o ICMP error demuxing for locator failure discovery.

o If necessary, develop and specify formats and structure for:

- Cryptographically protected locators

- Carrying the flow label across the shim layer
defined in the multi6 architecture.

The shim6 WG is to publish, as standards track RFC's, specifications
with enough details to allow fully interoperable implementations.

Goals and Milestones:
Aug 05    First draft of architectural document
Aug 05    First draft of protocol document
Aug 05    First draft on cryptographic locators, if required
Aug 05    First draft on multi-homing triggers description
Aug 05    First draft on applicability statement document
Oct 05    WG last-call on architectural document
Oct 05    WG last-call on applicability statement document
Feb 06    WG last-call on protocol document
Feb 06    WG last-call on cryptographic locators, if required
Feb 06    Submit completed architectural document to IESG
Feb 06    Submit applicability statement document to IESG
Apr 06    WG last-call on multihoming triggers description
Apr 06    Submit document on cryptographic locators to the IESG, if required
Apr 06    Submit protocol document to the IESG
Jun 06    Submit draft on multihoming triggers description to the IESG

Delivery-date: Mon, 27 Jun 2005 22:56:32 +0000
Date: Mon, 27 Jun 2005 15:55:41 -0700
From: David Meyer <[email-masked]>
To: Geoff Huston <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Call for agenda items for the first Shim6 WG Meeting

--tKW2IUtsqtDRztdT
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Jun 27, 2005 at 10:15:04AM +1000, Geoff Huston wrote:
> Hi,
>=20
> Kurtis and I have been informed that the IESG has approved the charter fo=
r=20
> the SHIM6 WG.  IN addition to Kurtis and myself as WG chairs, we'll have=
=20
> the benefit of Thomas Narten and Dave Meyer as technical advisors for the=
=20
> working group.
>=20
> Firstly, thanks to Dave and Thomas for being able to assist in this way.

	Thanks Geoff, glad to be part of it.

	Dave

--tKW2IUtsqtDRztdT
Content-Type: application/pgp-signature
Content-Disposition: inline

--tKW2IUtsqtDRztdT--

Delivery-date: Mon, 27 Jun 2005 11:56:35 +0000
Subject: Re: Call for agenda items for the first Shim6 WG Meeting
From: Jeroen Massar <[email-masked]>
To: Geoff Huston <[email-masked]>
Cc: [email-masked]
Organization: Unfix
Date: Mon, 27 Jun 2005 13:55:10 +0200

--=-ZEEM9yT83iIcs7l/8X2K
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, 2005-06-27 at 10:15 +1000, Geoff Huston wrote:
> Hi,
>=20
> Kurtis and I have been informed that the IESG has approved the charter fo=
r=20
> the SHIM6 WG.  IN addition to Kurtis and myself as WG chairs, we'll have=20
> the benefit of Thomas Narten and Dave Meyer as technical advisors for the=
=20
> working group.

This is great news, now hope we can things on the road.
But we need a protocol spec to get on with it, as....

I have the main infrastructure code done, that is kernelspace editions
for both FreeBSD + Linux, which can easily insert/add/remove/pass
certain parts of headers to a userspace.
BSD/Linux kernelcode is different of course, also license style, the
userspace daemon, BSD-license, is the same and should be quite
extensible to fit the needs of shim6. I hope to clean this stuff up in
the coming weeks so that I can shove it onto the mine + sourceforge
servers for public scrutiny. I hope that with this step, and the
license, it will be very easy for implementers to later on just copy the
code into their products allowing a speedy process of implementations
and hopefully deployment.

Jusqu'=E0 Paris,
 Jeroen

--=-ZEEM9yT83iIcs7l/8X2K
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

--=-ZEEM9yT83iIcs7l/8X2K--

Delivery-date: Mon, 27 Jun 2005 00:16:25 +0000
Date: Mon, 27 Jun 2005 10:15:04 +1000
To: [email-masked]
From: Geoff Huston <[email-masked]>
Subject: Call for agenda items for the first Shim6 WG Meeting
Cc: [email-masked], [email-masked], [email-masked]

Hi,

Kurtis and I have been informed that the IESG has approved the charter for 
the SHIM6 WG.  IN addition to Kurtis and myself as WG chairs, we'll have 
the benefit of Thomas Narten and Dave Meyer as technical advisors for the 
working group.

Firstly, thanks to Dave and Thomas for being able to assist in this way.

We would now like to get moving on the arrangements for the first Working 
Group at the next IETF meeting, so at this stage we'd like to hear from any 
working group member who has material that they are interested in 
presenting to the working group. Don't forget that we have moved past the 
more general survey of possible approaches that was undertaken in the 
previous multihoming working group and are looking quite specifically at 
the approach as documented in the following drafts:

draft-huston-l3shim-arch-00.txt
draft-ietf-multi6-functional-dec-00.txt
draft-ietf-multi6-l3shim-00.txt
draft-ietf-multi6-failure-detection-00.txt
draft-ietf-multi6-hba-00.txt
draft-ietf-multi6-app-refer-00.txt

regards,

    Kurtis & Geoff

Delivery-date: Mon, 20 Jun 2005 13:19:46 +0000
To: IETF Announcement list <[email-masked]>
Cc: [email-masked]
From: IESG <[email-masked]>
Subject: WG Review: Site Multihoming by IPv6 Intermediation (shim6)
Date: Fri, 17 Jun 2005 15:47:34 -0400

A new IETF working group has been proposed in the Internet Area. The IESG has
not made any determination as yet. The following draft charter was submitted,
and is provided for informational purposes only. Please send your comments to
the IESG mailing list ([email-masked]) by Friday, June 24th.

+++

Site Multihoming by IPv6 Intermediation (shim6)
==============================================

Current Status: Proposed Working Group

Internet Area Director(s):
Mark Townsley <townsley at cisco.com>
Margaret Wasserman <margaret at thingmagic.com>

Internet Area Advisor:
TBD

WG Chair(s):
TBD

Technical Advisor(s):
TBD

Secretary(ies):
TBD

Mailing List: shim6 at psg.com
To Subscribe: shim6-request at psg.com
Archive: TBD

Description:

For the purposes of redundancy, load sharing, operational policy or cost, a
site may be multi-homed, with the site's network having connections to
multiple IP service providers. The current Internet routing infrastructure
permits multi-homing using provider independent addressing, and adapts to
changes in the availability of these connections. However if the site uses
multiple provider-assigned address prefixes for every host within the site,
host application associations cannot use alternate paths, such as for
surviving the changes or for creating new associations, when one or more of
the site's address prefixes becomes unreachable. This working group will
produce specifications for an IPv6-based site multi-homing solution that
inserts a new sub-layer (shim) into the IP stack of end-system hosts. It
will enable hosts on multi-homed sites to use a set of provider-assigned IP
address prefixes and switch between them without upsetting transport
protocols or applications.

The work will be based on the architecture developed by the IETF multi6
working group. The shim6 working group is to complete the required protocol
developments and the architecture and security analysis of the required
protocols.

Requirements for the solution are:

o The approach must handle re-homing both existing communication and
being able to establish new communication when one or more of the
addresses is unreachable.

o IPv6 NAT devices are assumed not to exist, or not to present an
obstacle about which the shim6 solution needs to be concerned.

o Only IPv6 is considered.

o Changes in the addresses that are used below the shim will be invisible
to the upper layers, which will see a fixed address (termed Upper Layer
Identifier or ULID).

o ULIDs will be actual IP addresses, permitting existing applications to
continue to work unchanged, and permitting application referrals to
work, as long as the IP Addresses are available.

o The solution should assume ingress filtering may be applied at network
boundaries.

o The solution must allow the global routing system to scale even if there
is a very large number of multi-homed sites. This implies that re-homing
not be visible to the routing system.

o Compatibility will remain for existing mobility mechanisms. It will
be possible to use Mobile IPv6 on a node that also supports Shim6.
However, any optimizations or advanced configurations are out of
scope for shim6.

o The approach is to provide an optimized way to handle a static set of
addresses, while also providing a way to securely handle dynamic
changes in the set of addresses. The dynamic changes might be useful
for future combinations of multi-homing and IP mobility, but the working
group will not take on such mobility capabilities directly.

o The specifications must specifically refer to all applicable threats and
describe how they are handled, with the requirement being that the
resulting solution not introduce any threats that make the security any
less than in today's Internet.

The background documents to be considered by the WG include:

RFC 3582
draft-ietf-multi6-architecture-04.txt
draft-ietf-multi6-things-to-think-about-01.txt
draft-ietf-multi6-multihoming-threats-03.txt

The input documents that the WG will use as the basis for its design are:

draft-huston-l3shim-arch-00.txt
draft-ietf-multi6-functional-dec-00.txt
draft-ietf-multi6-l3shim-00.txt
draft-ietf-multi6-failure-detection-00.txt
draft-ietf-multi6-hba-00.txt
draft-ietf-multi6-app-refer-00.txt

In addition to the network layer shim solution, the shim6 WG is
specifically chartered to work on:

o Solutions for site exit router selection that work when each ISP
uses ingress filtering, i.e. when the chosen site exit needs to
be related to the source address chosen by the host. This site
exit router selection and the associated address selection
process should work whether or not the peer site supports
the shim6 protocol.

o Solutions to establish new communications after an outage has
occurred that do not require shim support from the
non-multihomed end of the communication. The Working Group will
explore whether such solutions are also useful when both ends
support the shim.

o The possible impact of the use of multiple locators at both ends
on congestion control, traffic engineering, and QoS will be analysed
in conjunction with the Transport Area.

o The relationships between Upper Layer Identifiers (ULIDs)
and unique local addresses.

o ICMP error demuxing for locator failure discovery.

o If necessary, develop and specify formats and structure for:

- Cryptographically protected locators
- Carrying the flow label across the shim layer
defined in the multi6 architecture.

The shim6 WG is to publish, as standards track RFC's, specifications with
enough details to allow fully interoperable implementations.

Milestones

AUG 05 First draft of architectural document
AUG 05 First draft of protocol document
AUG 05 First draft on cryptographic locators, if required
AUG 05 First draft on multi-homing triggers description
AUG 05 First draft on applicability statement document
OCT 05 WG last-call on architectural document
OCT 05 WG last-call on applicability statement document
FEB 06 WG last-call on protocol document
FEB 06 WG last-call on cryptographic locators, if required
FEB 06 Submit completed architectural document to IESG
FEB 06 Submit applicability statement document to IESG
APR 06 WG last-call on multihoming triggers description
APR 06 Submit document on cryptographic locators to the IESG, if
required
APR 06 Submit protocol document to the IESG
JUN 06 Submit draft on multihoming triggers description to the IESG

Delivery-date: Wed, 08 Jun 2005 09:36:21 +0000
Cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Wed, 8 Jun 2005 11:35:41 +0200
To: Erik Nordmark <[email-masked]>

Hi Erik,

El 06/06/2005, a las 20:12, Erik Nordmark escribi=F3:

> Marcelo,
>
> you seemed to have concluded that we need negative advice from the=20
> ULPs (or a combination of negative and positive advice), but I must=20
> have missed how you came to that conclusion.
>

well, i haven't reach to any conclusion so far, i was just exploring=20
how the ULP feedback would look like (sorry if my writing expressed=20
otherwise)

> In my mind, given that some ULPs might not provide any advice and we=20=

> want to provide failover on that case, we need the default behavior of=20=

> the shim to be to probe at some frequency (once every 10 seconds or=20
> so?) when the ULP is sending packets. Perhaps it makes sense to use Tx=20=

> and Rx counters to be able to suppress such probes, and I think it=20
> makes sense to use positive ULP advice (when given) to suppress the=20
> probes.
>
> Do we agree so far?
>

agree

> Once we have the above, then I don't see what negative ULP advice=20
> would add.

well, i guess that it would result in a faster detection of the=20
problem, since if the failure detection is made through lack of=20
possitive feedback, then the shim would need to timeout. Of course, the=20=

ULP would also need to timeout, but probably the timeout of the ULP is=20=

more adjusted than the deafult value of the shim. In any case, as you=20
point out below, this can be achieved also by letting the ulp to inform=20=

the shim about the timeout. So i don't have a strong opinion w.r.t. to=20=

this positive or negative feedback issue so far

regards, marcelo

>  We clearly can't always send probes on negative advice, since that=20
> could mean that a congestive loss in the network would result in=20
> negative ULP advice, causing the host to send more packets, which=20
> could make the congestion worse. While one could rate limit the probes=20=

> triggered by negative advice, the negative advice just seems like=20
> added complexity. (A rate limit of one every 10 seconds means that the=20=

> negative advice would be completely ignored, with a default of probing=20=

> once every 10 seconds.)
>
>> that brings out an interesting issue: what if we have multiple ULPs=20=

>> using the same session and they provide different feedback?
>> For instance, a simple case would be that some apps are more=20
>> sensitive than others, so they will complain sooner. More complex=20
>> cases could be that one app complains and the other one provides=20
>> positive feedback (suppose that the failure is on the app level and=20=

>> not in the path for instance) how do we deal with this?
>
> If we only use positive advice then I think we can avoid this=20
> complexity. In any case, I don't think we should worry much about it=20=

> up front. Later it might make sense to allow the ULPs to express a=20
> "desired failure detection time" to the shim, and the shim can use=20
> that to determine how often to probe. If ULPs with different desires=20=

> use the same ULID pair, then presumably the shim would have to operate=20=

> on the minimum of the requested times. And the shim presumably needs=20=

> to ensure that it doesn't try to probe more frequently than a=20
> conservative RTT estimate no matter what the ULPs desire.
>
>> IMHO, ULP feedback should result in an explicity reachabililty test=20=

>> on the current locator pair i.e. ULP feedback does not directly=20
>> implies rehoming, but in a verification though a reachability test=20
>> exchange of the current locator pair.
>
> Here you are equation "ULP feedback" with "negative ULP feedback".=20
> Such verification can lead to congestion (and as I stated above, I=20
> don't see how handling negative ULP feedback fits into the big=20
> picture.)
>
>    Erik
>
>
>

Delivery-date: Tue, 07 Jun 2005 11:29:52 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Tue, 7 Jun 2005 13:29:10 +0200
To: Erik Nordmark <[email-masked]>

On 6-jun-2005, at 20:12, Erik Nordmark wrote:

> In my mind, given that some ULPs might not provide any advice and  
> we want to provide failover on that case, we need the default  
> behavior of the shim to be to probe at some frequency (once every  
> 10 seconds or so?) when the ULP is sending packets.

Probing when you're sending is rather superflous: you're already  
sending in that direction so no need to send more. Having the other  
end send keepalives accomplishes the same thing with lower overhead:  
you get feedback that the other end is still there, but packets only  
flow in the otherwise idle direction.

> Once we have the above, then I don't see what negative ULP advice  
> would add. We clearly can't always send probes on negative advice,  
> since that could mean that a congestive loss in the network would  
> result in negative ULP advice, causing the host to send more  
> packets, which could make the congestion worse.

:-)

Well I would hope it would result in a rehoming so a non-congested  
path would be selected.

> If ULPs with different desires use the same ULID pair, then  
> presumably the shim would have to operate on the minimum of the  
> requested times. And the shim presumably needs to ensure that it  
> doesn't try to probe more frequently than a conservative RTT  
> estimate no matter what the ULPs desire.

Conservative RTT could be 1 ms on fast ethernet...

Delivery-date: Mon, 06 Jun 2005 18:13:05 +0000
Date: Mon, 06 Jun 2005 11:12:16 -0700
From: Erik Nordmark <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers

Marcelo,

you seemed to have concluded that we need negative advice from the ULPs 
(or a combination of negative and positive advice), but I must have 
missed how you came to that conclusion.

In my mind, given that some ULPs might not provide any advice and we 
want to provide failover on that case, we need the default behavior of 
the shim to be to probe at some frequency (once every 10 seconds or so?) 
when the ULP is sending packets. Perhaps it makes sense to use Tx and Rx 
counters to be able to suppress such probes, and I think it makes sense 
to use positive ULP advice (when given) to suppress the probes.

Do we agree so far?

Once we have the above, then I don't see what negative ULP advice would 
add. We clearly can't always send probes on negative advice, since that 
could mean that a congestive loss in the network would result in 
negative ULP advice, causing the host to send more packets, which could 
make the congestion worse. While one could rate limit the probes 
triggered by negative advice, the negative advice just seems like added 
complexity. (A rate limit of one every 10 seconds means that the 
negative advice would be completely ignored, with a default of probing 
once every 10 seconds.)

> that brings out an interesting issue: what if we have multiple ULPs 
> using the same session and they provide different feedback?
> 
> For instance, a simple case would be that some apps are more sensitive 
> than others, so they will complain sooner. More complex cases could be 
> that one app complains and the other one provides positive feedback 
> (suppose that the failure is on the app level and not in the path for 
> instance) how do we deal with this?

If we only use positive advice then I think we can avoid this 
complexity. In any case, I don't think we should worry much about it up 
front. Later it might make sense to allow the ULPs to express a "desired 
failure detection time" to the shim, and the shim can use that to 
determine how often to probe. If ULPs with different desires use the 
same ULID pair, then presumably the shim would have to operate on the 
minimum of the requested times. And the shim presumably needs to ensure 
that it doesn't try to probe more frequently than a conservative RTT 
estimate no matter what the ULPs desire.

> IMHO, ULP feedback should result in an explicity reachabililty test on 
> the current locator pair i.e. ULP feedback does not directly implies 
> rehoming, but in a verification though a reachability test exchange of 
> the current locator pair.

Here you are equation "ULP feedback" with "negative ULP feedback". Such 
verification can lead to congestion (and as I stated above, I don't see 
how handling negative ULP feedback fits into the big picture.)

    Erik

Delivery-date: Sun, 05 Jun 2005 10:22:35 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Sun, 5 Jun 2005 12:21:48 +0200
To: marcelo bagnulo braun <[email-masked]>

On 4-jun-2005, at 18:11, marcelo bagnulo braun wrote:

>> On the other hand, I wouldn't necessarily put too much trust in =20
>> what weird ULPs have to say. But as long as what they have to say =20
>> can only help or hurt themselves I don't really care, of course.

> that brings out an interesting issue: what if we have multiple ULPs =20=

> using the same session and they provide different feedback?

> For instance, a simple case would be that some apps are more =20
> sensitive than others, so they will complain sooner. More complex =20
> cases could be that one app complains and the other one provides =20
> positive feedback (suppose that the failure is on the app level and =20=

> not in the path for instance) how do we deal with this?

I think we have to be prepared for the situation where an ULP =20
provides incorrect feedback.

> IMHO, ULP feedback should result in an explicity reachabililty test =20=

> on the current locator pair i.e. ULP feedback does not directly =20
> implies rehoming, but in a verification though a reachability test =20
> exchange of the current locator pair.

Yes. But we also rate limit these tests to avoid excessive probing. =20
For instance, use exponential backoff for negative feedback. So if an =20=

ULP complains, first time we do a reachability test immediately. If =20
there is nothing wrong that time and it then complains again within X =20=

seconds (60 or so) we schedule a reachability test for 10 seconds =20
from now. If it then complains again within 60 seconds after that =20
test (ie 70 seconds) the next one is scheduled in 20 seconds. And so =20
on until we pretty much don't listen to that ULP any more.

I'm not sure how positive feedback enters the equation, except maybe =20
to cancel pending scheduled reachability tests.

> Failure detection hints including:
>     - ULP negative feedback
>     - Tx>0 and Rx=3D0
>     - Receive a reachability test exchange from the peer (do we =20
> still need this?)

Yes, if the ULP doesn't provide hints we can't discover a =20
unidirectional reachability failure from here to the other end, so we =20=

need to depend on the other end to detect those.

And if the other end started the exchange already with one extra =20
packet we know what's going on too. Even if we don't really need to =20
know what's up with the connectivity it might still be good to =20
measure the RTT.

>     - ICMP error
>     - SHIM error =BF?

> As the result of any of these hints, a reachability test exchange =20
> is performed using the current locator pair

Yes. Only exception to this could be when the other side starts a =20
reachability test because they don't hear from us anymore and we know =20=

we sent them stuff recently. In that case it's almost certain that =20
the current pair in that direction doesn't work anymore.

> If reachability test succeed then keep on using the current locator =20=

> pair
> If reachability test fails, the start alternative locator pair =20
> exploration process

> When one alternative locator pair that is working is found, then =20
> rehome the communication

> Did i miss something?

We need to avoid bad paths. So just selecting the first one that =20
works could be dangerous. Obviously it helps to be smart about what =20
to try first. For instance, if host A notices that for 10 sessions to =20=

different destinations with source or dest A1 there are (potential) =20
problems while A2 is doing fine, try with A2 first of course. (And =20
include information in the probe that tells the other side to try A2 =20
first as well.)

In addition, we could send out probes at an interval that's only =20
slightly higher than the RTT. This means that if the RTT for the =20
address pair we're testing is lower than the current one, we have an =20
answer before we send out the second probe, and it's likely we found =20
a good path. If the answer takes longer than (say) 80% probability =20
for the RTT for the old path considering the stddev, it's likely =20
we're trying a bad path so sending out a second probe is good. Even =20
if we get an answer for the first one after sending the second one, =20
if the second one uses a better path we'll get that answer shortly =20
after than and we can immediately rehome to that path if it looks =20
better.

Iljitsch=

Delivery-date: Sat, 04 Jun 2005 16:15:32 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Sat, 4 Jun 2005 18:11:44 +0200
To: Iljitsch van Beijnum <[email-masked]>

...
>

>> If we have positive ACK from the ULP, we should be fine and do=20
>> nothing, even if the TX and RX are detecting a failure)
>
> If we only receive and don't send but the ULP is happy we still need=20=

> to send keepalives because we can't know that the ULP at the other=20
> side is also providing positive feedback.
>

right, we still have to send keepalives

> If we only send and don't receive but the ULP is happy (how does it=20
> know it can be happy??) I guess we can go either way without much=20
> harm.
>
> So basically positive feedback wouldn't (have to) change our behavior.
>
>> Rx and Tx are to be used when there is no info from ULP
>> The reasons for this is that ULP really know for sure if=20
>> communication is progressing adequately. Tx and Rx are just guessing.=20=

>> For instance, Rx could be fooled by an attacker injecting packets=20
>> with spoofed source address, in order to keep the communication in a=20=

>> given path.
>
> That's true.
>
> On the other hand, I wouldn't necessarily put too much trust in what=20=

> weird ULPs have to say. But as long as what they have to say can only=20=

> help or hurt themselves I don't really care, of course.
>

that brings out an interesting issue: what if we have multiple ULPs=20
using the same session and they provide different feedback?

For instance, a simple case would be that some apps are more sensitive=20=

than others, so they will complain sooner. More complex cases could be=20=

that one app complains and the other one provides positive feedback=20
(suppose that the failure is on the app level and not in the path for=20
instance) how do we deal with this?

IMHO, ULP feedback should result in an explicity reachabililty test on=20=

the current locator pair i.e. ULP feedback does not directly implies=20
rehoming, but in a verification though a reachability test exchange of=20=

the current locator pair.

Actually, imho all the info about failure should be considered as=20
hints, that imply a reachability test exchange to verify that the apth=20=

is working or not.

So, as i understand it we would have:

Failure detection hints including:
     - ULP negative feedback
     - Tx>0 and Rx=3D0
     - Receive a reachability test exchange from the peer (do we still=20=

need this?)
     - ICMP error
     - SHIM error =BF?

As the result of any of these hints, a reachability test exchange is=20
performed using the current locator pair

If reachability test succeed then keep on using the current locator pair
If reachability test fails, the start alternative locator pair=20
exploration process

When one alternative locator pair that is working is found, then rehome=20=

the communication

Did i miss something?

Regards, marcelo

> Iljitsch
>

Delivery-date: Sat, 04 Jun 2005 14:44:47 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Sat, 4 Jun 2005 16:44:17 +0200
To: marcelo bagnulo braun <[email-masked]>

Hi Marcelo,

On 4-jun-2005, at 12:33, marcelo bagnulo braun wrote:

>> - if recent rx and recent tx: be happy, everything works
>> - if no recent rx and no recent tx: session is idle, do nothing
>> - if recent rx, but not recent tx: send keepalive
>> - if recent tx, but not recent rx: failure likely, start full  
>> reachability testing procedure

> i asume that receiving a keepalive does not increment Rx, right?  
> because if it does, then i think you could end up exchanging only  
> keepalives, and i guess we don't want that...

Right, when no recent tx and recent rx just keepalives, don't send  
keepalives. The other side will then stop sending keepalives as well  
so the state goes to idle.

> full reachability testing procedure does include verifying if  
> currently used locator pair is working through a reachability test,  
> right?

Right, and exploring other locator pairs until a working one is found  
or we know that none of them work.

>> The improvement over what you propose and/or NUD is that when  
>> there is no outgoing traffic, we just send keepalives, which could  
>> theoretically be just IPv6 headers without a payload. Since these  
>> are unidirectional they don't take up bandwidth in the other  
>> direction, as there is traffic flowing in that direction anyway.

> right, this would reduce 50% of the signalling in this situations i  
> guess

Yes, 50% of the packets and probably a bit more of the data as the  
keepalives can be very small, they don't have to carry any data or  
signalling.

>> There wouldn't be any advantage in combining this with positive  
>> feedback from ULPs, except maybe some small implementation  
>> optimization. However, listening for negative feedback could be  
>> useful to start reachability tests faster than the regular shim  
>> timeout.

> imho, ULP feedback should supersede this mechanism, since it it the  
> most reliable feedback imho.

:-)  I hope so.

Obviously TCP does really well here, but UDP-based stuff is a  
different story.

> I mean, if we have negative ACK from the ULP, then we must perform  
> a reahcbaility test to verify current locator pair (even if the Rx  
> ANd Tx are not detecting any problem)

Right: the ULP can have detected the problem faster, and, better safe  
than sorry.

> If we have positive ACK from the ULP, we should be fine and do  
> nothing, even if the TX and RX are detecting a failure)

If we only receive and don't send but the ULP is happy we still need  
to send keepalives because we can't know that the ULP at the other  
side is also providing positive feedback.

If we only send and don't receive but the ULP is happy (how does it  
know it can be happy??) I guess we can go either way without much harm.

So basically positive feedback wouldn't (have to) change our behavior.

> Rx and Tx are to be used when there is no info from ULP
> The reasons for this is that ULP really know for sure if  
> communication is progressing adequately. Tx and Rx are just  
> guessing. For instance, Rx could be fooled by an attacker injecting  
> packets with spoofed source address, in order to keep the  
> communication in a given path.

That's true.

On the other hand, I wouldn't necessarily put too much trust in what  
weird ULPs have to say. But as long as what they have to say can only  
help or hurt themselves I don't really care, of course.

Iljitsch

Delivery-date: Sat, 04 Jun 2005 10:36:28 +0000
Cc: shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Sat, 4 Jun 2005 12:33:47 +0200
To: Iljitsch van Beijnum <[email-masked]>

Hi Iljitsch,

El 02/06/2005, a las 17:49, Iljitsch van Beijnum escribi=F3:

> On 2-jun-2005, at 16:39, marcelo bagnulo braun wrote:
>
>> However, my understanding is that a mechanism in the lines of what=20
>> Iljitsch suggests (based on the observation of the existence of a=20
>> bidirectional flow of packets) could be an interesting optimization=20=

>> in the bidirectional case.
>
>> My understanding of how such mechanism would work is the following:
>
>> The shim layer observes the amount of traffic exchanged during the=20
>> last T seconds, being Tx the number of packets transmitted and Rx the=20=

>> number of packets received during the last T seconds.
>> If Tx>0 and Rx>0, then no problem
>> If Tx=3D0, then no problem
>> If Tx>0 and Rx=3D0, then perform a reachability test to verify the=20
>> current locator pair, and eventually a path exploration exchange
>
>> Upon the reception of a path exploration by the peer, the node must=20=

>> perform a reachability test to verify the current locator pair.
>
> I think if we combine all of this, a good mechanism would be:
>
> - if recent rx and recent tx: be happy, everything works
> - if no recent rx and no recent tx: session is idle, do nothing
> - if recent rx, but not recent tx: send keepalive
> - if recent tx, but not recent rx: failure likely, start full=20
> reachability testing procedure
>

i asume that receiving a keepalive does not increment Rx, right?=20
because if it does, then i think you could end up exchanging only=20
keepalives, and i guess we don't want that...

full reachability testing procedure does include verifying if currently=20=

used locator pair is working through a reachability test, right?

> (These tests would have to be performed X seconds after=20
> sending/receiving a packet, not at the time of packet=20
> transmission/reception.)
>
> The improvement over what you propose and/or NUD is that when there is=20=

> no outgoing traffic, we just send keepalives, which could=20
> theoretically be just IPv6 headers without a payload. Since these are=20=

> unidirectional they don't take up bandwidth in the other direction, as=20=

> there is traffic flowing in that direction anyway.
>

right, this would reduce 50% of the signalling in this situations i=20
guess

> Doing things this way guarantuees that traffic can never flow in just=20=

> one direction, so when a host is sending and not receiving, the only=20=

> explanation can be a failure, so this is the point where we start a=20
> full reachability exploration procedure, with the advantage that we=20
> know the current path is almost certainly in trouble.
>
> There wouldn't be any advantage in combining this with positive=20
> feedback from ULPs, except maybe some small implementation=20
> optimization. However, listening for negative feedback could be useful=20=

> to start reachability tests faster than the regular shim timeout.
>

imho, ULP feedback should supersede this mechanism, since it it the=20
most reliable feedback imho.

I mean, if we have negative ACK from the ULP, then we must perform a=20
reahcbaility test to verify current locator pair (even if the Rx ANd Tx=20=

are not detecting any problem)
If we have positive ACK from the ULP, we should be fine and do nothing,=20=

even if the TX and RX are detecting a failure)

Rx and Tx are to be used when there is no info from ULP
The reasons for this is that ULP really know for sure if communication=20=

is progressing adequately. Tx and Rx are just guessing. For instance,=20
Rx could be fooled by an attacker injecting packets with spoofed source=20=

address, in order to keep the communication in a given path.

>> So, imho there are at least two questions to answer in order to see=20=

>> if this is interesting:
>> 1- is this really an optimization? i.e. does it provides some form of=20=

>> improvement (and is it worthy?)
>> 2- How frequent are we expecting this case to be, so that it is=20
>> worthy to optimize it?
>
>> My opinion about these two are the following:
>
>> About 1: i guess this mechanism would improve the efficiency of the=20=

>> solution because it would reduce the number of reachability test=20
>> exchanged to verify the current locator pair in the case of and UDP=20=

>> bidirectional flow (or any other bidirectional flow where the ULP=20
>> does not provide positive feedback)
>
> Indeed.
>
>> About 2: this mechanism optimizes the case of bidirectional flow that=20=

>> belongs to ULPs that not provide positive feedback (e.g. UDP)
>> However, this mechanism only work in the case where at least one path=20=

>> with bidirectional connectivity is available.
>
> ??? Isn't that true of any interactive communication?
>
>> This mechanism fails in the case where the only paths available are=20=

>> two different unidirectional paths.
>
> No, not necessarily. The question is what we test for:
>
> locator <-> locator
> ULID <-> ULID
> host <-> host
>
> I would prefer the last one, but we'll probably have to settle for the=20=

> middle one because the constraints for the last one would probably be=20=

> too severe. (I.e., if any packet from A to B means there is=20
> reachability from A to B, ALL packets from A to B would have to use=20
> the same locators or we could miss some failures. The constraint that=20=

> all packets from ULID Ax to ULID Bx use the same locator set is=20
> probably more reasonable.)
>
>

right, as i said, i was thinking in the locator pair case, but as you=20
say this can be extended to the ULID case, which is superior.

Regards, marcelo

Delivery-date: Sat, 04 Jun 2005 10:35:58 +0000
Cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Sat, 4 Jun 2005 12:20:13 +0200
To: Erik Nordmark <[email-masked]>

Hi Erik,

El 04/06/2005, a las 0:50, Erik Nordmark escribi=F3:

> marcelo bagnulo braun wrote:
>
>> My understanding of how such mechanism would work is the following:
>> The shim layer observes the amount of traffic exchanged during the=20
>> last T seconds, being Tx the number of packets transmitted and Rx the=20=

>> number of packets received during the last T seconds.
>> If Tx>0 and Rx>0, then no problem
>> If Tx=3D0, then no problem
>
> In those two cases it is really "not my problem", since there might be=20=

> a problem, but the peer will have to detect it.
>
>> If Tx>0 and Rx=3D0, then perform a reachability test to verify the=20
>> current locator pair, and eventually a path exploration exchange
>> Upon the reception of a path exploration by the peer, the node must=20=

>> perform a reachability test to verify the current locator pair.
>
> ok
>
>> My opinion about these two are the following:
>> About 1: i guess this mechanism would improve the efficiency of the=20=

>> solution because it would reduce the number of reachability test=20
>> exchanged to verify the current locator pair in the case of and UDP=20=

>> bidirectional flow (or any other bidirectional flow where the ULP=20
>> does not provide positive feedback)
>
> Yes, I think it will reduce the number of probes in this case.
>
>> About 2: this mechanism optimizes the case of bidirectional flow that=20=

>> belongs to ULPs that not provide positive feedback (e.g. UDP)
>> However, this mechanism only work in the case where at least one path=20=

>> with bidirectional connectivity is available. This mechanism fails in=20=

>> the case where the only paths available are two different=20
>> unidirectional paths. This means, that if we want to support such=20
>> case (i.e. if we want to be able to preserve the communication in=20
>> this case), then we need to be able to detect and overrule such case.
>
> I don't understand why you think this is limited to the bidirectional=20=

> locator pair case. If A is using <A1, B1> and B is using <B2, A1>,=20
> then the optimization to suppress probes works just as well as when B=20=

> is using <B2, A1>. The probes just serve to verify the current locator=20=

> pair, hence the response to a probe would presumably use the current=20=

> locator pair in the reverse direction.
>

right, i was thinking that these applies to the locator pair i.e.=20
before making the demux,
If you do it after the demux, based on ULIDs as Iljitsch put it, then,=20=

i guess you are both right, and that this scheme works also for=20
unidirectional paths, which is great! :-)

> One other factor to take into account is that for the optimization to=20=

> work the receiver has to count the Rx packets for a ULID pair. But,=20
> before there is any failure i.e. when the locators are the ULIDs, the=20=

> receiver wouldn't otherwise have a need to lookup the locator pair; it=20=

> would just like today's IP code
>  - look at IP destination to determine if "packet for me"
>  - look at IP nexthdr to see what to do (e.g., pass to UDP code)
>
> Rx counting would add to that code path
>  - lookup IP <src, dst> to find counter; increment Rx counter
>
> [Note that once the locators are no longer the ULIDs the receiver have=20=

> to find the shim context, thus there wouldn't be any additional lookup=20=

> to count Rx packets. But assuming most packets arrive using=20
> ULIDs=3Dlocators, we need to worry about the performance of an=20
> implementation in that case.]
>

agree
this should be put into the tradeoff consideration, but i am not sure=20
how to quantify this effect though

regards, marcelo

>   Erik
>

Delivery-date: Fri, 03 Jun 2005 22:50:58 +0000
Date: Fri, 03 Jun 2005 15:50:03 -0700
From: Erik Nordmark <[email-masked]>
To: marcelo bagnulo braun <[email-masked]>
CC: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers

marcelo bagnulo braun wrote:

> My understanding of how such mechanism would work is the following:
> 
> The shim layer observes the amount of traffic exchanged during the last 
> T seconds, being Tx the number of packets transmitted and Rx the number 
> of packets received during the last T seconds.
> If Tx>0 and Rx>0, then no problem
> If Tx=0, then no problem

In those two cases it is really "not my problem", since there might be a 
problem, but the peer will have to detect it.

> If Tx>0 and Rx=0, then perform a reachability test to verify the current 
> locator pair, and eventually a path exploration exchange
> 
> Upon the reception of a path exploration by the peer, the node must 
> perform a reachability test to verify the current locator pair.

ok

> My opinion about these two are the following:
> 
> About 1: i guess this mechanism would improve the efficiency of the 
> solution because it would reduce the number of reachability test 
> exchanged to verify the current locator pair in the case of and UDP 
> bidirectional flow (or any other bidirectional flow where the ULP does 
> not provide positive feedback)

Yes, I think it will reduce the number of probes in this case.

> About 2: this mechanism optimizes the case of bidirectional flow that 
> belongs to ULPs that not provide positive feedback (e.g. UDP)
> However, this mechanism only work in the case where at least one path 
> with bidirectional connectivity is available. This mechanism fails in 
> the case where the only paths available are two different unidirectional 
> paths. This means, that if we want to support such case (i.e. if we want 
> to be able to preserve the communication in this case), then we need to 
> be able to detect and overrule such case.

I don't understand why you think this is limited to the bidirectional 
locator pair case. If A is using <A1, B1> and B is using <B2, A1>, then 
the optimization to suppress probes works just as well as when B is 
using <B2, A1>. The probes just serve to verify the current locator 
pair, hence the response to a probe would presumably use the current 
locator pair in the reverse direction.

One other factor to take into account is that for the optimization to 
work the receiver has to count the Rx packets for a ULID pair. But, 
before there is any failure i.e. when the locators are the ULIDs, the 
receiver wouldn't otherwise have a need to lookup the locator pair; it 
would just like today's IP code
  - look at IP destination to determine if "packet for me"
  - look at IP nexthdr to see what to do (e.g., pass to UDP code)

Rx counting would add to that code path
  - lookup IP <src, dst> to find counter; increment Rx counter

[Note that once the locators are no longer the ULIDs the receiver have 
to find the shim context, thus there wouldn't be any additional lookup 
to count Rx packets. But assuming most packets arrive using 
ULIDs=locators, we need to worry about the performance of an 
implementation in that case.]

   Erik

Delivery-date: Thu, 02 Jun 2005 15:50:07 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Thu, 2 Jun 2005 17:49:40 +0200
To: marcelo bagnulo braun <[email-masked]>

On 2-jun-2005, at 16:39, marcelo bagnulo braun wrote:

> However, my understanding is that a mechanism in the lines of what  
> Iljitsch suggests (based on the observation of the existence of a  
> bidirectional flow of packets) could be an interesting optimization  
> in the bidirectional case.

> My understanding of how such mechanism would work is the following:

> The shim layer observes the amount of traffic exchanged during the  
> last T seconds, being Tx the number of packets transmitted and Rx  
> the number of packets received during the last T seconds.
> If Tx>0 and Rx>0, then no problem
> If Tx=0, then no problem
> If Tx>0 and Rx=0, then perform a reachability test to verify the  
> current locator pair, and eventually a path exploration exchange

> Upon the reception of a path exploration by the peer, the node must  
> perform a reachability test to verify the current locator pair.

I think if we combine all of this, a good mechanism would be:

- if recent rx and recent tx: be happy, everything works
- if no recent rx and no recent tx: session is idle, do nothing
- if recent rx, but not recent tx: send keepalive
- if recent tx, but not recent rx: failure likely, start full  
reachability testing procedure

(These tests would have to be performed X seconds after sending/ 
receiving a packet, not at the time of packet transmission/reception.)

The improvement over what you propose and/or NUD is that when there  
is no outgoing traffic, we just send keepalives, which could  
theoretically be just IPv6 headers without a payload. Since these are  
unidirectional they don't take up bandwidth in the other direction,  
as there is traffic flowing in that direction anyway.

Doing things this way guarantuees that traffic can never flow in just  
one direction, so when a host is sending and not receiving, the only  
explanation can be a failure, so this is the point where we start a  
full reachability exploration procedure, with the advantage that we  
know the current path is almost certainly in trouble.

There wouldn't be any advantage in combining this with positive  
feedback from ULPs, except maybe some small implementation  
optimization. However, listening for negative feedback could be  
useful to start reachability tests faster than the regular shim timeout.

> So, imho there are at least two questions to answer in order to see  
> if this is interesting:
> 1- is this really an optimization? i.e. does it provides some form  
> of improvement (and is it worthy?)
> 2- How frequent are we expecting this case to be, so that it is  
> worthy to optimize it?

> My opinion about these two are the following:

> About 1: i guess this mechanism would improve the efficiency of the  
> solution because it would reduce the number of reachability test  
> exchanged to verify the current locator pair in the case of and UDP  
> bidirectional flow (or any other bidirectional flow where the ULP  
> does not provide positive feedback)

Indeed.

> About 2: this mechanism optimizes the case of bidirectional flow  
> that belongs to ULPs that not provide positive feedback (e.g. UDP)
> However, this mechanism only work in the case where at least one  
> path with bidirectional connectivity is available.

??? Isn't that true of any interactive communication?

> This mechanism fails in the case where the only paths available are  
> two different unidirectional paths.

No, not necessarily. The question is what we test for:

locator <-> locator
ULID <-> ULID
host <-> host

I would prefer the last one, but we'll probably have to settle for  
the middle one because the constraints for the last one would  
probably be too severe. (I.e., if any packet from A to B means there  
is reachability from A to B, ALL packets from A to B would have to  
use the same locators or we could miss some failures. The constraint  
that all packets from ULID Ax to ULID Bx use the same locator set is  
probably more reasonable.)

Delivery-date: Thu, 02 Jun 2005 14:40:20 +0000
Cc: Iljitsch van Beijnum <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Failure Detection (was Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Thu, 2 Jun 2005 16:39:54 +0200
To: Erik Nordmark <[email-masked]>

Hi Erik, Iljitsch,

my understanding is that the two mechanisms that Erik proposes seems to 
be the base mechanims fro detecting that a pair of addresses is not 
working any longer, that is:

> No it isn't hard to avoid. What we need is more or less exactly what 
> we have in NUD in RFC 2462 but applies end to end (and as a result 
> using a different timeout behavior etc):
>  - A probe using the current address pair which is sent when a ULP 
> packet is sent, but only if there is no e2e reachability confirmation 
> in the last N seconds.
>  - Optional ULP positive advice which suppresses the need to probe 
> since it provides an e2e reachability confirmation
>

imho this is enough, as Erik says, and it should be the building blocks 
of the failure detection

However, my understanding is that a mechanism in the lines of what 
Iljitsch suggests (based on the observation of the existence of a 
bidirectional flow of packets) could be an interesting optimization in 
the bidirectional case.

My understanding of how such mechanism would work is the following:

The shim layer observes the amount of traffic exchanged during the last 
T seconds, being Tx the number of packets transmitted and Rx the number 
of packets received during the last T seconds.
If Tx>0 and Rx>0, then no problem
If Tx=0, then no problem
If Tx>0 and Rx=0, then perform a reachability test to verify the 
current locator pair, and eventually a path exploration exchange

Upon the reception of a path exploration by the peer, the node must 
perform a reachability test to verify the current locator pair.

So, imho there are at least two questions to answer in order to see if 
this is interesting:
1- is this really an optimization? i.e. does it provides some form of 
improvement (and is it worthy?)
2- How frequent are we expecting this case to be, so that it is worthy 
to optimize it?

My opinion about these two are the following:

About 1: i guess this mechanism would improve the efficiency of the 
solution because it would reduce the number of reachability test 
exchanged to verify the current locator pair in the case of and UDP 
bidirectional flow (or any other bidirectional flow where the ULP does 
not provide positive feedback)

About 2: this mechanism optimizes the case of bidirectional flow that 
belongs to ULPs that not provide positive feedback (e.g. UDP)
However, this mechanism only work in the case where at least one path 
with bidirectional connectivity is available. This mechanism fails in 
the case where the only paths available are two different 
unidirectional paths. This means, that if we want to support such case 
(i.e. if we want to be able to preserve the communication in this 
case), then we need to be able to detect and overrule such case.

Regards, marcelo

Delivery-date: Thu, 02 Jun 2005 10:38:11 +0000
Cc: Erik Nordmark <[email-masked]>, shim6 <[email-masked]>
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: address pair exploration, flooding and state loss
Date: Thu, 2 Jun 2005 12:37:29 +0200
To: Iljitsch van Beijnum <[email-masked]>

Hi Iljitsch,

El 01/06/2005, a las 12:31, Iljitsch van Beijnum escribi=F3:

> On 31-mei-2005, at 23:03, Erik Nordmark wrote:
>
>>> Another way to do this would be the "ICMP bad  checksum" message=20
>>> that I suggested a few days ago.
>

would this be a mandatory change (i.e. required to make the SHIm work)=20=

or just an optimization?

I mean, i don't think that it would be a good idea to require mandatory=20=

changes on transport layers, even if they only involve TCP, UDP and=20
SCTP...

Regards, marcelo

>> But as I responded, such an approach only makes sense if we are going=20=

>> to modify all the protocols which run over IP to generate such=20
>> errors. And assuming that an implementation can do this is=20
>> problematic due to "raw socket" APIs which allow there to be any=20
>> number of IP protocols implemented unbeknown to the provider of the=20=

>> IP and shim6 layers.
>
> AFAIK, applications that use raw sockets for actual user traffic are=20=

> extremely rare. (I can't think of any, not counting IP tunnels.)=20
> Typically, applications use UDP for "strange" protocols. So if we=20
> modify TCP, UDP and SCTP we should be out of the woods here. And if=20
> there really are applications that don't use these transports but are=20=

> interested in recovering quickly from shim state loss, it shouldn't be=20=

> too hard for them to implement the "icmp bad transport checksum".
>

Delivery-date: Wed, 01 Jun 2005 23:47:26 +0000
Date: Wed, 01 Jun 2005 16:47:19 -0700
From: Erik Nordmark <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: shim6 <[email-masked]>
Subject: Re: soft state (was Re: shim6 and bit errors in data packet headers

Iljitsch van Beijnum wrote:

>> The path from B->A fails.
>> The shim layer on A observes this because it stops receiving packets.
>> (But TCP on A also starts retransmitting.)
>> The shim layer on B thinks everything is fine because it sees the  
>> packets from A, and it sees the ACKs that B is sending back to A.
> 
> 
> What would happen here is that A doesn't see any return traffic so it  
> starts sending reachability probes. In this case B knows about the  
> problem very soon because the first probe presumably uses address  pair 
> A1, B1 which still works in the A -> B direction.

Yes, so having the shim track whether packets are sent and received 
isn't useful in this case.

> Yes. But as long as we switch from having a short timeout when there  is 
> traffic to having a long or infinite timout when there is no  traffic 
> this shouldn't be problematic.

But you'd need to tell the peer what timeout you are using, right?
So you'd still be adding packets?

> That's true, but it's hard to avoid unless you want to rely on ULP  
> timeouts or behavior very similar to ULP timeouts.

No it isn't hard to avoid. What we need is more or less exactly what we 
have in NUD in RFC 2462 but applies end to end (and as a result using a 
different timeout behavior etc):
  - A probe using the current address pair which is sent when a ULP 
packet is sent, but only if there is no e2e reachability confirmation in 
the last N seconds.
  - Optional ULP positive advice which suppresses the need to probe 
since it provides an e2e reachability confirmation

That is sufficient to tell when there is a problem with the current 
address pair without any extra packets when the ULP provides positive 
advise, and at most one probe exchange every N seconds when the ULP is 
sending packets but doesn't provide positive advice.

In addition we'd need to trigger exploring alternative locator pairs, so 
we would need to decide when that starts (at the same time as the probe 
for the current address pair, or when it looks like that probe might 
have failed?)

> And the amount of traffic would be fairly low. For instance, when  
> traffic is flowing the timer could be 15 seconds. So when there is no  
> more traffic, after 15 seconds the sender would have to send a  
> keepalive. The keepalive could contain a new timer value, which could  
> be X times the previous value (30, 60, 120 and so on), or immediately  
> go to infinity. When traffic starts to flow again, the timer would  have 
> to be reinitialized, or maybe this can happen automatically.

Take a single UDP request/response interaction as an example, but the 
shim doesn't know that there will be a single request/response, and 
where was was some initial TCP traffic that was sufficient to create the 
shim state up front.

The worst case could easily be
1. Send UDP request A->B
2. Send UDP response B->A
3. 14 seconds later: send "timeout is 30 seconds now" A->B
    send "timeout is 30 seconds now" from B to A
4. 29 seconds later: send "timeout is 60 seconds now" A->B and B->A
5. 59 seconds later ...
etc

This means that the total number of packets (when there are no 
failures!) doesn't have a natural bound related to the number of ULP 
packets; 2 ULP packets can result in a large number of "update timeout 
packets" - above there are 6 such packets are a result of 2 ULP packets. 
The only way you can provide a limit of 100% overhead would be to 
immediately switch to infinite timeouts, but then the timeout doesn't 
seem useful?

Compare this with the positive advise ULP triggered probing, and where 
the ULP (in user space) doesn't provide any advice:
1. Send UDP request A->B
    Since the earlier TCP connection just ended the sender has recent
    reachability confirmation for A1->B1, so no probe is needed
2. Send UDP response B->A
    Same as in #1, no probe needed.

If the reachability from the preceding TCP connection is old, we'd get 
the worst-case behavior of
1. Send UDP request A->B
    Reachability confirmation for A1->B1 is >10 seconds old; send a probe
    A1->B1
2. Send UDP response B->A
    Reachability confirmation for B1->A1 is >10 seconds old; send a probe
    B1->A1
3. B receives probe from A; responds
4. A receives probe from B; responds

With the introduction of the DELAY state in RFC 2461 one can probably 
drop the above worst case 6 packets to a worst case of 4 packets. Thus 
the worst possible overhead is 100% (2 probe packets for 2 data 
packets), but this only happens when
  - the application sends a request/response every >10 seconds
  - there is no ULP advice

> For TCP, your positive advice would probably be cleaner, but for non- 
> TCP ULPs which typically don't have a similarly advanced  retransmission 
> mechanism, and especially for ULPs that don't provide  advice, this 
> mechanism, or the other that I mentioned earlier would  probably be a 
> good catch-all backup.

But such ULPs can also provide positive advise if they'd like. I 
actually thought we had added such a capability in section 11 in RFC 
3542, but I don't see it in there.

In any case, it sounds a bit complex to do one thing for TCP and a 
different thing for UDP. I don't know if that was what you were suggesting.

> On the receiver: 5 minutes. On the sender: min: 5 mins + some RTTs,  
> max: 5 mins + N times exponential backoff + some RTTs.

What would happen in this case:
1. One UDP request/response
2. Silent for 5 minutes or so (so a large timeout is conveyed to the peer)
3. There is a failure, for instance in the A->B path
4. Application on A sends UDP to B, which is lost
5. 4 seconds later the app on A retransmits the UDP packet, which is 
also lost
6. 8 seconds later the app on A retransmits the UDP packet; lost
etc

at which point in time would the shim on A and B see a need to probe?

We'd want this to have some relationship to the fact that the 
application is transmitting packets in 4,5,6, and I don't see how the 
receive side timer can do that.
The shim on the sender can decide at some point that packets have been 
sent, but none have been received, but that wouldn't require the 
receiver to run a timer (and having the sender tell the receiver which 
timer to use). But the receive side timer was suggested to avoid sending 
probes on idle connections, so it seems to be necessary.

Unless we take the NUD approach of ULP packet triggered probes that can 
be suppressed by positive advice from the ULP...

    Erik

Delivery-date: Wed, 01 Jun 2005 23:07:35 +0000
Date: Wed, 01 Jun 2005 16:07:19 -0700
From: Erik Nordmark <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: shim6 <[email-masked]>
Subject: Re: address pair exploration, flooding and state loss

Iljitsch van Beijnum wrote:

> I'm not familiar with DCCP. Will it be implemented directly on top of  
> IP? That won't be much fun in IPv4, with firewalls and NATs and all.  
> Since general purpose transports need port numbers and a checksum  
> anyway, running over UDP is generally a no-brainer.

See draft-ietf-dccp-spec-11.txt
Runs on IP - not on UDP.

> I would agree about actually being able to work. However, the only  
> thing we're talking about here is recovering from state loss. This is  
> hardly a critical function, especially if we can make it work for 95%  
> of all applications without any trouble.

I'm under the impression that one of the reasons to multihome is to get 
better reliability/availability. With that goal in mind it seems a bit 
odd to introduce some new failure mode in the shim protocol - such as 
not being able to recover from state loss.

> Obviously an alternative approach that can do this without any other  
> downsides would be great. But robbing a bit isn't easy.

I recall some noid draft where I and Tony Li had worked out a way to do 
this a long time ago. That approach works when the flow label is used to 
carry the context tag.

If we carry the context tag in a new extension header/option we don't 
need to rob a bit, since the precense of that protocol element would 
indicate that the shim is in use and has failed over to alternate locators.

> This is about state loss, not failure detection.

Sorry, I got confused. More about this in the other thread.

    Erik

> However, if we use the "let me know if you don't see traffic from me  
> for N seconds" approach for failure detection, it could also detect  
> state loss in many cases. (I.e., when the session isn't in idle state.)

Delivery-date: Wed, 01 Jun 2005 21:51:10 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Wed, 1 Jun 2005 23:50:49 +0200
To: Erik Nordmark <[email-masked]>

On 1-jun-2005, at 23:01, Erik Nordmark wrote:

>> What I propose is a mechanism that purely looks whether traffic  
>> is  flowing in both directions. This doesn't require parsing any  
>> headers  except source and destination addresses which the shim  
>> must look at  anyway.

> I guess I don't understand what this would add beyond what your  
> proposed negative advice already can do.

> Perhaps you can explain how this would work in this example:
> A is using locator pair <A1, B1> and B is using <B1, A1>.
> TCP is sending data from A to B, with B responding with ACKs.

> The path from B->A fails.
> The shim layer on A observes this because it stops receiving packets.
> (But TCP on A also starts retransmitting.)
> The shim layer on B thinks everything is fine because it sees the  
> packets from A, and it sees the ACKs that B is sending back to A.

What would happen here is that A doesn't see any return traffic so it  
starts sending reachability probes. In this case B knows about the  
problem very soon because the first probe presumably uses address  
pair A1, B1 which still works in the A -> B direction.

>> But how about this: each side tells the other side a timeout  
>> value:  after not having seen any traffic from A, B starts  
>> probing. Now one  of three situations can happen:

> But doesn't this lead to a choice between having to send probes  
> when there is no ULP traffic, or having a long timer resulting in a  
> long time until a failure would be detected?

Yes. But as long as we switch from having a short timeout when there  
is traffic to having a long or infinite timout when there is no  
traffic this shouldn't be problematic.

>> - regular traffic: the timer is restarted before it expires by  
>> regular
>>   traffic, so the timer never expires and there are no probes
>> - irregular traffic: in order to make sure the timer doesn't  
>> expire if
>>   there isn't any traffic for some time, the sender injects  
>> keepalives
>>   so there are no probes

> The shim on the sender? Or the ULP? In either case, you end up  
> adding packets when things would have otherwise been idle.

The shim.

That's true, but it's hard to avoid unless you want to rely on ULP  
timeouts or behavior very similar to ULP timeouts.

And the amount of traffic would be fairly low. For instance, when  
traffic is flowing the timer could be 15 seconds. So when there is no  
more traffic, after 15 seconds the sender would have to send a  
keepalive. The keepalive could contain a new timer value, which could  
be X times the previous value (30, 60, 120 and so on), or immediately  
go to infinity. When traffic starts to flow again, the timer would  
have to be reinitialized, or maybe this can happen automatically.

For TCP, your positive advice would probably be cleaner, but for non- 
TCP ULPs which typically don't have a similarly advanced  
retransmission mechanism, and especially for ULPs that don't provide  
advice, this mechanism, or the other that I mentioned earlier would  
probably be a good catch-all backup.

There could be slight complexities in determining which failure  
detection mechanism to use, though. (Ugh: doing failure dectection in  
middlebox shim implementations.)

An interesting difference between this mechanism and the positive  
advice mechanism is that with positive advice, failures are initially  
detected by the sender, while with my timer announcement/timeout  
mechanism, failures are initially detected by the receiver.

>> - no traffic: the sender sets the timer to a very large value or
>>   infinity, so there are no probes

> If the timer has been set to e.g. 5 minutes, how quickly can the  
> shim detect a failure?

On the receiver: 5 minutes. On the sender: min: 5 mins + some RTTs,  
max: 5 mins + N times exponential backoff + some RTTs.

>> I'm assuming that when reachability probes are sent, probes with   
>> different address pairs are sent until a working pair is found in   
>> both directions, or it is determined that there is no  
>> bidirectional  connectivity anymore. So B would be informed unless  
>> there is no  longer any reachability possible.

> Yes, but it takes much longer time for B to be informed, because B  
> will not know there is a problem until A manages to get its first  
> probe through to B.
> If B has data driven probes (suppressed when there is positive  
> advice from the ULP), then it can find out sooner.

You'd have to wait for a TCP retransmit before the shim at the sender  
knows that something is wrong. Depening  on many things this may be  
quick or not so quick. TCP isn't tuned for quick regular timeout  
retransmissions as the "fast retransmit" mechanism covers the cases  
where only a single packet is lost.

> But I think probing that is suppressed when the ULP doesn't send  
> anything is a useful approach, because it avoids adding any probes  
> or keepalives when the ULP is silent.

Agree.

>> Hm, ok. But we have to be careful about mandating continuous   
>> communication between layers. In a properly layered  
>> implementation,  this type of communication can be quite expensive  
>> (context switches  and so on).

> FWIW the implementation of this that I know the best doesn't add  
> any communication events between the layers. When TCP goes to send  
> a packet to the IPv6 layer it can set a single flag "positive  
> reachability advise" as part of the packet that goes down to the  
> IPv6 transmit routine.

Ok. Still need to get some input on this before casting it into  
stone, though: it may not be implementable in this way on some systems.

Delivery-date: Wed, 01 Jun 2005 21:01:36 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: address pair exploration, flooding and state loss
Date: Wed, 1 Jun 2005 23:01:27 +0200
To: Erik Nordmark <[email-masked]>

On 1-jun-2005, at 22:27, Erik Nordmark wrote:

>> AFAIK, applications that use raw sockets for actual user traffic  
>> are  extremely rare. (I can't think of any, not counting IP tunnels.)

> So what happens when DCCP starts being implemented, perhaps as  
> initially in user-level using the raw socket interface?

I'm not familiar with DCCP. Will it be implemented directly on top of  
IP? That won't be much fun in IPv4, with firewalls and NATs and all.  
Since general purpose transports need port numbers and a checksum  
anyway, running over UDP is generally a no-brainer.

> I am not comfortable assuming that the transport protocol or its  
> implementation must change in order for shim6 to be able to work,

I would agree about actually being able to work. However, the only  
thing we're talking about here is recovering from state loss. This is  
hardly a critical function, especially if we can make it work for 95%  
of all applications without any trouble.

> especially when there are alternative approaches which do not  
> require this.

Obviously an alternative approach that can do this without any other  
downsides would be great. But robbing a bit isn't easy.

> An approach where the shim failure detection can work more  
> efficiently when there is some optional advice from the ULP is  
> quite different than requiring that the ULP code change.

This is about state loss, not failure detection.

However, if we use the "let me know if you don't see traffic from me  
for N seconds" approach for failure detection, it could also detect  
state loss in many cases. (I.e., when the session isn't in idle state.)

Delivery-date: Wed, 01 Jun 2005 21:01:03 +0000
Date: Wed, 01 Jun 2005 14:01:03 -0700
From: Erik Nordmark <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: shim6 <[email-masked]>
Subject: Re: soft state (was Re: shim6 and bit errors in data packet headers

Iljitsch van Beijnum wrote:

> (BTW: advise is a verb, advice is a noun.)

Oops. Thanks.

>> Thus the lack of positive advise is not the same as negative advise.
> 
> 
> So what we really need is: good / unknown / bad rather than either  good 
> / unknown or bad / unknown.

I'm not sure we need that level of complexity. With ULPs providing a 
single bit with the semantics 1 = good advice, 0 = I have nothing to 
say, I think we'll have sufficient information in the shim.

> I imagine some optimizations such as recognizing that there is no  reply 
> for TCP ack only packets, but let's ignore those for now.
> 
> What I propose is a mechanism that purely looks whether traffic is  
> flowing in both directions. This doesn't require parsing any headers  
> except source and destination addresses which the shim must look at  
> anyway.

I guess I don't understand what this would add beyond what your proposed 
negative advice already can do.

Perhaps you can explain how this would work in this example:
A is using locator pair <A1, B1> and B is using <B1, A1>.
TCP is sending data from A to B, with B responding with ACKs.

The path from B->A fails.
The shim layer on A observes this because it stops receiving packets.
(But TCP on A also starts retransmitting.)
The shim layer on B thinks everything is fine because it sees the 
packets from A, and it sees the ACKs that B is sending back to A.

> Yes, but it opens the door for continuous reachability probes, which  is 
> a bad thing because it wastes bandwidth and because it is likely  to 
> detect failures when the link is idle, which we shouldn't do IMO.

Quite the contrary. Just as RFC 2461 NUD probing, the probes would be 
data driven. Thus if there are no ULP packets to send there will be no 
probes.

> But how about this: each side tells the other side a timeout value:  
> after not having seen any traffic from A, B starts probing. Now one  of 
> three situations can happen:

But doesn't this lead to a choice between having to send probes when 
there is no ULP traffic, or having a long timer resulting in a long time 
until a failure would be detected?

> - regular traffic: the timer is restarted before it expires by regular
>   traffic, so the timer never expires and there are no probes
> - irregular traffic: in order to make sure the timer doesn't expire if
>   there isn't any traffic for some time, the sender injects keepalives
>   so there are no probes

The shim on the sender? Or the ULP? In either case, you end up adding 
packets when things would have otherwise been idle.

> - no traffic: the sender sets the timer to a very large value or
>   infinity, so there are no probes

If the timer has been set to e.g. 5 minutes, how quickly can the shim 
detect a failure?

> I'm assuming that when reachability probes are sent, probes with  
> different address pairs are sent until a working pair is found in  both 
> directions, or it is determined that there is no bidirectional  
> connectivity anymore. So B would be informed unless there is no  longer 
> any reachability possible.

Yes, but it takes much longer time for B to be informed, because B will 
not know there is a problem until A manages to get its first probe 
through to B.
If B has data driven probes (suppressed when there is positive advice 
from the ULP), then it can find out sooner.

> It's hard to say for sure what's going to work best in practice, so  we 
> need some experimentation at some point. However, that doesn't  help us 
> now, as we need to find a good candidate or a small number of  
> candidates for the initial experiments.

Agreed we need experiments down the road.

>> If the ULP retransmits 10 times with binary exponential backoff  
>> starting with a timeout of 4 seconds, and it has been told to send  
>> negative advise after consuming half the retransmits, then the shim  
>> will see negative advise after 4+8+16+32+64 seconds.
> 
> 
> (Note that this doesn't apply to what I was talking about as the ULP  
> itself isn't involved.)

Understood.
But I think probing that is suppressed when the ULP doesn't send 
anything is a useful approach, because it avoids adding any probes or 
keepalives when the ULP is silent.

> I agree 10 secs is a good starting point but I'm afraid using 10  
> seconds will clash with transports that use 10 seconds themselves...  We 
> probably need to review a bunch of ULPs to make a good decision here.

I think both TCP and SCTP start off with 4 seconds RTO if they have no 
RTT estimate for the peer.
But it makes sense to validate this.

>>  - send 3 probes at time 10 seconds
>>  - send 3 other probes at time 14 seconds
>>  - send 3 more at time 22 seconds
> 
> 
> Why send 3 at the same time and then wait? Even with a small packet  
> train of 3 packets we're unnecessarily bursty. I think sending one at  a 
> time would be better, for instance:

Agreed. My "3 at a time" was just to say that I think we can send a 
fixed small number of probes for each exponential backoff "epoch" and 
still not cause a congestion control problem. But it makes sense to 
space them out as you suggest.

> Hm, ok. But we have to be careful about mandating continuous  
> communication between layers. In a properly layered implementation,  
> this type of communication can be quite expensive (context switches  and 
> so on).

FWIW the implementation of this that I know the best doesn't add any 
communication events between the layers. When TCP goes to send a packet 
to the IPv6 layer it can set a single flag "positive reachability 
advise" as part of the packet that goes down to the IPv6 transmit routine.

    Erik

Delivery-date: Wed, 01 Jun 2005 20:28:02 +0000
Date: Wed, 01 Jun 2005 13:27:12 -0700
From: Erik Nordmark <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
CC: shim6 <[email-masked]>
Subject: Re: address pair exploration, flooding and state loss

Iljitsch van Beijnum wrote:

> AFAIK, applications that use raw sockets for actual user traffic are  
> extremely rare. (I can't think of any, not counting IP tunnels.)  
> Typically, applications use UDP for "strange" protocols. So if we  
> modify TCP, UDP and SCTP we should be out of the woods here. And if  
> there really are applications that don't use these transports but are  
> interested in recovering quickly from shim state loss, it shouldn't  be 
> too hard for them to implement the "icmp bad transport checksum".

So what happens when DCCP starts being implemented, perhaps as initially 
in user-level using the raw socket interface?

I am not comfortable assuming that the transport protocol or its 
implementation must change in order for shim6 to be able to work, 
especially when there are alternative approaches which do not require this.

An approach where the shim failure detection can work more efficiently 
when there is some optional advice from the ULP is quite different than 
requiring that the ULP code change.

    Erik

Delivery-date: Wed, 01 Jun 2005 10:31:57 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: address pair exploration, flooding and state loss
Date: Wed, 1 Jun 2005 12:31:43 +0200
To: Erik Nordmark <[email-masked]>

On 31-mei-2005, at 23:03, Erik Nordmark wrote:

>> Another way to do this would be the "ICMP bad  checksum" message  
>> that I suggested a few days ago.

> But as I responded, such an approach only makes sense if we are  
> going to modify all the protocols which run over IP to generate  
> such errors. And assuming that an implementation can do this is  
> problematic due to "raw socket" APIs which allow there to be any  
> number of IP protocols implemented unbeknown to the provider of the  
> IP and shim6 layers.

AFAIK, applications that use raw sockets for actual user traffic are  
extremely rare. (I can't think of any, not counting IP tunnels.)  
Typically, applications use UDP for "strange" protocols. So if we  
modify TCP, UDP and SCTP we should be out of the woods here. And if  
there really are applications that don't use these transports but are  
interested in recovering quickly from shim state loss, it shouldn't  
be too hard for them to implement the "icmp bad transport checksum".

Delivery-date: Wed, 01 Jun 2005 10:25:19 +0000
Cc: shim6 <[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: soft state (was Re: shim6 and bit errors in data packet headers
Date: Wed, 1 Jun 2005 12:24:42 +0200
To: Erik Nordmark <[email-masked]>

On 31-mei-2005, at 22:56, Erik Nordmark wrote:

>>> This is the debate about positive vs. negative advise from the ULPs.

>> Hm, in a binary world saying no or not saying yes are the same   
>> thing... Not sure if there is a big difference here.

> Ah - but this is not binary.
> A ULP can do three things at any given time:
>  - provide no advise
>  - provide positive advise (things are making progress)
>  - provide negative advise (I see some problems/retransmissions)

(BTW: advise is a verb, advice is a noun.)

> Thus the lack of positive advise is not the same as negative advise.

So what we really need is: good / unknown / bad rather than either  
good / unknown or bad / unknown.

>> I want the shim to monitor ULP progress rather than depend  on the  
>> actual ULP to provide feedback. The problem with that would be   
>> that in many cases (= 99% of the time when UDP is the payload),  
>> the  "real" ULP is implemented in the application.

> Adding code in the shim that parses ULP headers to determine  
> "progress" doesn't make an implementation perform better, and  
> requiring that the shim understands this for all possible ULPs  
> (think raw sockets) doesn't make it easier to deploy the shim.

I imagine some optimizations such as recognizing that there is no  
reply for TCP ack only packets, but let's ignore those for now.

What I propose is a mechanism that purely looks whether traffic is  
flowing in both directions. This doesn't require parsing any headers  
except source and destination addresses which the shim must look at  
anyway.

> A technique based on positive advise from the ULP is robust against  
> the case when the ULP doesn't provide any advise, since this would  
> trigger the shim to do it's own data driven probes.
> So we can handle the UDP case just fine in this approach. The  
> positive advise from the ULP is a performance optimization; when  
> the advise arrives in the shim it removes the need for the shim to  
> probe.

Yes, but it opens the door for continuous reachability probes, which  
is a bad thing because it wastes bandwidth and because it is likely  
to detect failures when the link is idle, which we shouldn't do IMO.

But how about this: each side tells the other side a timeout value:  
after not having seen any traffic from A, B starts probing. Now one  
of three situations can happen:

- regular traffic: the timer is restarted before it expires by regular
   traffic, so the timer never expires and there are no probes
- irregular traffic: in order to make sure the timer doesn't expire if
   there isn't any traffic for some time, the sender injects keepalives
   so there are no probes
- no traffic: the sender sets the timer to a very large value or
   infinity, so there are no probes

Actual reachability probes sent by the receiving end would pretty  
much only happen when there is a failure, and by selecting a good  
value for the timer it should be possible to suppress keepalives  
pretty much all the time except when traffic volume for a session  
changes. At the same time, this allows a large amount of flexibility  
for accommodating application needs.

I think this can be implemented fairly efficiently in the shim code  
path.

>>> But things are problematic on B, because there isn't an  
>>> (efficient)  strategy for the TCP on B to generate negative  
>>> advise - it doesn't  run a retransmit timer.

>> Ah, but if A can detect the failure in this case, then that would  
>> be  good enough if A can tell B about it at some point.

> Yes, but this assumes that packets from A to B in fact get delivered.
> Since the failure could be for the A->B direction, the B->A  
> direction, or both, there are cases when B would not be informed  
> that A is seeing problems,

I'm assuming that when reachability probes are sent, probes with  
different address pairs are sent until a working pair is found in  
both directions, or it is determined that there is no bidirectional  
connectivity anymore. So B would be informed unless there is no  
longer any reachability possible.

> I agree that we can have either or both ends try all the N*M  
> locator pairs, i.e. that the technique works. But the issue is how  
> efficient it would be and whether positive vs. negative advise  
> makes a difference here on what we can do in the shim.

It's hard to say for sure what's going to work best in practice, so  
we need some experimentation at some point. However, that doesn't  
help us now, as we need to find a good candidate or a small number of  
candidates for the initial experiments.

>>> Thus when something fails it will always be up to A to initiate  
>>> the  exploration of alternate locator pairs. Also, the time at  
>>> which the  exploration of alternates start is a function of the  
>>> retransmit  behavior of the ULP, which makes it harder to tightly  
>>> control the  failover time.

>> No, in my plan the shim wouldn't know about retransmissions, it  
>> only  looks for return traffic. So either the timeout is  
>> relatively long to  accommodate ULPs that don't send traffic in  
>> the low-traffic direction  very often (I think streaming A/V  
>> protocols send an ack every 10  seconds or so) or relatively short  
>> but then there would be almost  continuous reachability probes in  
>> at least one direction.

> But then the best you can do is determined by the ULPs (re) 
> transmission behavior, which is why I say that you can't control  
> the failover time in the shim.

In this scenario the failover time depends on the assumptions about  
the maximum delay between traffic in one direction and traffic (acks)  
in the other direction. I think this would be some 10 seconds, adding  
a few extra seconds for network delays and random variations that  
would make for around 15 seconds. Compared to what we have now worst  
case or average case that's pretty good.

> If the ULP retransmits 10 times with binary exponential backoff  
> starting with a timeout of 4 seconds, and it has been told to send  
> negative advise after consuming half the retransmits, then the shim  
> will see negative advise after 4+8+16+32+64 seconds.

(Note that this doesn't apply to what I was talking about as the ULP  
itself isn't involved.)

>> What kind of failover time do you imagine, BTW?

> 10 seconds for sending a probe might not be a bad default.

I agree 10 secs is a good starting point but I'm afraid using 10  
seconds will clash with transports that use 10 seconds themselves...  
We probably need to review a bunch of ULPs to make a good decision here.

> If we think we can send a small number of probes in parallel (3 or  
> so) with binary exponential backoff for the probes, we might be  
> able to recover from the failure in one RTT after those 10 seconds,  
> but if there are lots of address pairs to try and more than one has  
> failed, it can take a lot longer. If the shim itself is  
> conservative and has a 4 second probe timeout with exponential  
> backoff, then it would
>  - send 3 probes at time 10 seconds
>  - send 3 other probes at time 14 seconds
>  - send 3 more at time 22 seconds

Why send 3 at the same time and then wait? Even with a small packet  
train of 3 packets we're unnecessarily bursty. I think sending one at  
a time would be better, for instance:

- at 10 seconds
- at 11 seconds
- at 12 seconds
- at 14 seconds
- at 16 seconds
- at 19 seconds
- at 22 seconds
- at 25 seconds
- at 29 seconds

>>> With such a strategy the shim implementation can do a check  
>>> after  sending a ULP packet: "how long time ago since some  
>>> positive advise?"

>> AFter every packet...?

> Yes, but a node which implements the mandatory NUD in RFC 2461  
> already has that test in the code path, so it might very well be  
> possible to implement the shim6 liveness check without adding an  
> extra test to the code path.

Hm, ok. But we have to be careful about mandating continuous  
communication between layers. In a properly layered implementation,  
this type of communication can be quite expensive (context switches  
and so on).