Subject: [homegate] restart avalanche

From: "Dan Wing" <[email-masked]>
Date: Fri, 7 May 2010 08:02:15 -0700

As we continue to design PCP (port control protocol), a worry is restart
avalanches caused by a metropolitan-wide power outage/restoration event.
However, restart avalanche concerns are not unique to PCP; restart avalanches
are also a problem for DHCP servers, DNS servers (as devices query for things
like pool.ntp.org), NTP servers, SIP proxies, XMPP servers, and so on.

The restart avalanche problem has been discussed before in the context of VoIP
devices in PacketCable.  Text exists in Section 7.4.3.5 of PacketCable's NCS
specification, titled "Fighting the Restart Avalanche",
http://www.cablelabs.com/specifications/PKT-SP-NCS1.5-I03-070412.pdf and in
Section 4.4.6 of RFC3435, titled "Fighting the Restart Avalanche",
http://tools.ietf.org/html/rfc3435#section-4.4.6

I would prefer that HOMEGATE describe the problem and require the customer
premise router's coldstart initialization be randomly delayed.  The
alternative is that each protocol include a similar mechanism (waste of code,
testing, etc.) and the risk is a device will include both its own delay and a
per-protocol delay, adding unnecessary delay to coldstart initialization.

Thoughts?

-d

(Thanks to Flemming Andreasen for the citations to the PacketCable specs.)

---

From: Marc Blanchet <[email-masked]>
Date: Fri, 07 May 2010 11:06:10 -0400

FYI, IPv6 Neighbor discovery also has that anti-restart avalanche 
mechanism. RFC4861.

Marc.

Le 10-05-07 11:02, Dan Wing a écrit :
> _______________________________________________
> homegate mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/homegate

-- 
=========
IPv6 book: Migrating to IPv6, Wiley. http://www.ipv6book.ca
Stun/Turn server for VoIP NAT-FW traversal: http://numb.viagenie.ca
DTN news service: http://reeves.viagenie.ca
NAT64-DNS64 Opensource: http://ecdysis.viagenie.ca

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 7 May 2010 18:17:07 +0200 (CEST)

On Fri, 7 May 2010, Dan Wing wrote:

> premise router's coldstart initialization be randomly delayed.

Is that really the best way to solve this (if you by this means minutes of 
backoff instead of tens of seconds?)

I'd rather see more intelligent backoff if the initial request failed, 
because we don't want to delay customer getting their access restored when 
they cold-reboot (or initial-startup) on purpose.

Quite a substantial part of the population power-off their equipment when 
they don't use it (to save power and to protect against power-surges etc).

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 7 May 2010 09:51:17 -0700

> Is that really the best way to solve this (if you by this 
> means minutes of backoff instead of tens of seconds?)
>
> I'd rather see more intelligent backoff if the initial 
> request failed, because we don't want to delay customer 
> getting their access restored when they cold-reboot (or 
> initial-startup) on purpose.

I am purposefully not discussing the duration of the delay,
or if the delay is fixed or based on some 
random_value*exponential backoff.

Rather, I am first trying to see if HOMEGATE feels restart 
avalanche is something in HOMEGATE's scope.

> Quite a substantial part of the population power-off their 
> equipment when they don't use it (to save power and to 
> protect against power-surges etc).

That can be accomodated by 'the detection of a local 
user activity' which short-cuts the delay.  Anything that
differentiates an unattended device from an attended
device would be sufficient.  For examples: pressing and 
holding the 'power-on' button or some other button, or 
detecting it was the physical power switch being toggled
(versus mains power being restored).

One idea to avoid restart delay is for the network to
signal the CP router (e.g., using DHCP) that the network
is (or is not) currently experiencing a restart avalanche 
and the CP router should delay (or should not delay) its 
initialization.  This would allow the DHCP server to 
protect the rest of the network elements from the 
avalanche restart.  It is just one idea.

But rather than details such as that, I would like to 
first understand if HOMEGATE wants to include restart 
avalanche into its scope.

-d

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 7 May 2010 10:04:20 -0700

>avalanche is something in HOMEGATE's scope.

It would only be in scope if it was the gateway that was the one doing the restart. If it is a server behind a gateway that is initiating the restart, the gateway should not participate in avalanche prevention. That is, I would *not* want a gateway to say "I just started up, and now I see a request from something behind me, and I know that request is on my do-not-avalanche list, so I'm going to delay that by a random amount".

Having said that, I would like to see a generic model for intelligent backoff that could be used in protocols, including those that are used in gateways. Once that is established, listing which protocols that gateways themselves use that should observe that model should be in scope here.

>avalanche restart.  It is just one idea.

...and a good one. There could also be a generic, trusted, ultra-low-overhead service on a network that the gateway or host behind the gateway could ask either before it does something, or when it gets an anomalous result (a "try again later" response, no reply when it really expects one, and so on). This could also protect a DHCP server if done correctly.

--Paul Hoffman, Director
--VPN Consortium

---

From: Fred Baker <[email-masked]>
Date: Fri, 7 May 2010 13:58:08 -0700

randomization is a good thing, and exponential backoffs are a proven thing. Personally, I would recommend having, on the Nth request, the thing wait a random number of milliseconds between 2^(N-1) and 2^N seconds between retries.

On May 7, 2010, at 9:17 AM, Mikael Abrahamsson wrote:

> Is that really the best way to solve this (if you by this means minutes of backoff instead of tens of seconds?)
> 
> I'd rather see more intelligent backoff if the initial request failed, because we don't want to delay customer getting their access restored when they cold-reboot (or initial-startup) on purpose.
> 
> Quite a substantial part of the population power-off their equipment when they don't use it (to save power and to protect against power-surges etc).
> 
> -- 

http://www.ipinc.net/IPv4.GIF

---

From: Ralph Droms <[email-masked]>
Date: Fri, 7 May 2010 18:21:19 -0400

I agree in principle...

...but a couple of devils in the details include:

* is the same initial short and randomized delay appropriate for all  
protocols?
* how would the delay interact with existing randomized delays; e.g.,  
in DHCP and ND

- Ralph

---

From: Ralph Droms <[email-masked]>
Date: Fri, 7 May 2010 18:24:45 -0400

Considering the multitude of protocols involved and the lack of  
clarity about what constitutes "failure" and "retry", I think the only  
sensible system-wide delay would be to wait some short delay before  
starting the normal initialization.  Perhaps delay a number of seconds  
chosen from a random distribution between 0 and 10 seconds?  Then let  
all the specific protocols such as DHCP and ND start up with their own  
initial delays and exponential backoffs.

- Ralph

On May 7, 2010, at 4:58 PM 5/7/10, Fred Baker wrote:

> randomization is a good thing, and exponential backoffs are a proven  
> thing. Personally, I would recommend having, on the Nth request, the  
> thing wait a random number of milliseconds between 2^(N-1) and 2^N  
> seconds between retries.
>
>> Is that really the best way to solve this (if you by this means  
>> minutes of backoff instead of tens of seconds?)
>>
>> I'd rather see more intelligent backoff if the initial request  
>> failed, because we don't want to delay customer getting their  
>> access restored when they cold-reboot (or initial-startup) on  
>> purpose.
>>
>> equipment when they don't use it (to save power and to protect  
>> against power-surges etc).
>>
>> --

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Fri, 7 May 2010 21:54:47 -0500

I agree with Ralph.  Further, one should not only look at RFC 4861 but
also RFC 4862.  I think there is little or no interaction of ND with
DHCPv6 because DHCPv6 travels over the link-local address. The home
gateway has to create a link-local address and then send a NS(DAD) for
the link-local before the router can send any DHCPv6 message.  See
section 5.4.2 of RFC 4862 than recommends the randomized delay in
sending of the first NS. DHCPv6 has its own initial.  I don't see a
restart avalanche problem.  The protocols that are involved with the
home gateway initialization have randomized delay and start built into
them.

Hemant

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Ralph Droms
Sent: Friday, May 07, 2010 6:25 PM
To: Fred Baker (fred)
Cc: [email-masked]
Subject: Re: [homegate] restart avalanche

- Ralph

---

From: "Frank Bulk" <[email-masked]>
Date: Fri, 7 May 2010 23:11:22 -0500

There's four different access technologies and only one of them possibly
faces this restart avalanche issue for 'homegate':
a) xDSL: if a homegate device is attached to a DSL modem, modems will take
varying lengths of time to sync based on brand/model DSL modem (as each
modem boots up), line conditions, back end PPP processes (if any), and DHCP.
So the homegate device attached to the DSL modem will have an inherent
random dely
b) DOCSIS: much the same as (a).  With ~200 CMs, we see that it takes up to
15 minutes for all the cable modems to get connected.  
c) BWA: much the same as (a)
d) FTTH: if the ONT is battery-backed, the 'homegate' device would be on as
quickly as the power returns.  This is the only situation where I can see
restart avalanche control having substantive value.

Unless we're looking to use restart avalanche control to delay start up by
several minutes, I believe the other processes mention in (a) and (b) will
act as the random delay mechanism.  Rather than build delay in 'homegate',
affecting every user's experience, I'd encourage better mechanisms in the
service provider environment to deal with the "avalanches".

Frank

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Dan Wing
Sent: Friday, May 07, 2010 10:02 AM
To: [email-masked]
Subject: [homegate] restart avalanche

As we continue to design PCP (port control protocol), a worry is restart
avalanches caused by a metropolitan-wide power outage/restoration event.
However, restart avalanche concerns are not unique to PCP; restart
avalanches
are also a problem for DHCP servers, DNS servers (as devices query for
things
like pool.ntp.org), NTP servers, SIP proxies, XMPP servers, and so on.

The restart avalanche problem has been discussed before in the context of
VoIP

I would prefer that HOMEGATE describe the problem and require the customer
premise router's coldstart initialization be randomly delayed.  The
alternative is that each protocol include a similar mechanism (waste of
code,
testing, etc.) and the risk is a device will include both its own delay and
a
per-protocol delay, adding unnecessary delay to coldstart initialization.

Thoughts?

-d

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 8 May 2010 06:52:29 +0200 (CEST)

> So the homegate device attached to the DSL modem will have an inherent

Millions of devices will pound RADIUS backend systems if PPPoX and 
authentication is used, some kind of backoff here is definitely needed. 
When you have millions of devices this is definitely an avalance (I've 
seen this myself in a production network and the BRAS didn't have any 
ratelimiting features so it was hard to mitigate).

> Unless we're looking to use restart avalanche control to delay start up 
> by several minutes, I believe the other processes mention in (a) and (b) 
> will act as the random delay mechanism.  Rather than build delay in 
> 'homegate', affecting every user's experience, I'd encourage better 
> mechanisms in the service provider environment to deal with the 
> "avalanches".

I think the recommendation should be that any mechanism that talks to the 
provider network should have some kind of backoff if the initial query 
isn't answered. I believe DHCP already does this, I don't know about PPP 
establishment requests.

I tend to agree that this should be handled in the network as well, but 
all protocols should have backoff in case of non-response, nothing should 
run on a simple interval timer without backoff.

---

From: "Frank Bulk" <[email-masked]>
Date: Sat, 8 May 2010 12:18:54 -0500

The number of events where *millions* of devices point a RADIUS backend
systems nearly simultaneously are few -- I don't think that's worth
designing for.  And if they happened, it would be very unlikely that they
were caused by a power outage, but more likely by some kind of connectivity
failure between the CPE and system at the core, and that would be restart
avalanche control as much as reconnect avalanche control.

And just to be clear, where are we looking for avalanche control?  PPPoA/E?
IPv6 RA? DHCP? DNS?

I'm following the general outlines of the concern, but I think we had better
flesh out the real life scenarios before adding in random startup delays for
something at layer 2/3/4/7 that will frustrate customers 99% of the time for
an event that happens less than 0.01% of power on events.

Frank

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Mikael Abrahamsson
Sent: Friday, May 07, 2010 11:52 PM
To: [email-masked]
Subject: Re: [homegate] restart avalanche

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 8 May 2010 19:34:01 +0200 (CEST)

> avalanche control as much as reconnect avalanche control.

In this case it was a connectivity event, but the same reasoning is true 
for a widespread power event (which are rare but they obviously happen).

> And just to be clear, where are we looking for avalanche control?  PPPoA/E?

All of the above, and more. All protocols where the device talks to the 
network, NTP as well (wasn't there a CPE that kept asking NTP once per 
minute even if it didn't get any response, that meant quite a lot of 
traffic with millions of devices). There should be a general guideline 
that all these mechanism use a sensible backoff algorithm.

> an event that happens less than 0.01% of power on events.

I don't want a startup delay, I want backoff.

---

From: Jason Livingood <[email-masked]>
Date: Mon, 10 May 2010 09:55:23 -0400

Seems like an interesting question and one that's prompted some discussion
on the list so far.  I think though, given the lower order questions I just
posted to the list, I should defer to the advising ADs on the question of
scope/interest.

Jason