Subject: [v6ops] SLAAC renum: Problem Statement & Operational workarounds

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 30 Oct 2019 21:01:47 +0000

Hello:

DHC folks may want to check out the recent discussion on the v6ops mailing list (https://mailarchive.ietf.org/arch/browse/v6ops/) with the "SLAAC renum: Problem Statement & Operational workarounds" subject.

Basically, one issue that appears to happen in networks where CPEs receive delegated prefixes and are renumbered when a CPE cycles power - why the renumbering occurs is not material. The issue is that some of the devices serviced by the CPE may still have configuration (i.e., from Prefix Information Options received via RAs or perhaps even DHCP) based on the original delegated prefix and if the CPE does not have stable storage (into which it stored the old delegated prefix), nothing initiates deprecating this information. This is because everyone is honoring the lifetimes, but the renumbering caused the SP DHCP server to provide a totally new delegated prefix and there's no record in the CPE of the old.

For a CPE that has stable storage, it could initiate steps to deprecate the old delegated prefix.

Note that this issue is discussed in https://tools.ietf.org/html/draft-fkhp-dhc-dhcpv6-pd-relay-requirements-00#section-3.2 and also in section 4.3 (S-1) of that draft.

One idea that I suggested is that the SP DHCP server can provide this information to the CPE. The DHCP server should still have a record of the previous delegated prefix (as it has not yet expired for the binding) and it could therefore send this, with 0 lifetimes, in the Reply to the Request (as it would do in a Renew). This would avoid the need for the CPE to keep this information in stable storage.

There isn't any direct mention of sending IA_PD (or IA_NA) with 0 lifetimes in the processing of Requests, but it isn't a stretch to consider this. And, without something, it isn't obvious that servers should (or would) do this?

Some possibilities are (from least conservative to most):

  1.  Always have the server do this (sending 0 lifetimes) for a Reply to a Request for this condition.
  2.  (1) but only for Relayed messages.
  3.  Have a server configuration knob (at whatever granularities desired - global, prefix, client, ...).
  4.  Define a new option that a client would include in (Solicit/)Request to request this behavior. Note that (3) could always override.

Hopefully existing clients would handle this correctly, though they likely would not take the action to deprecate the old delegated prefix. But we don't know (and at least early in the DHCPv6 days, clients weren't always happy when they received multiple delegated prefixes). Hence, how conservative should we be?

Perhaps we can discuss as part of draft-fkhp-dhc-dhcpv6-pd-relay-requirements (which is on the agenda for IETF-106), or I could request a separate slot for it - if we need a higher bandwidth discussion.

We can also just discuss here on the mailing list as it may be possible to resolve fairly quickly.

It will require a short draft as it is something that would need to be folded into a RFC8415 bis or continue on it own. Probably adding it to draft-fkhp-dhc-dhcpv6-pd-relay-requirements is not best since these are the delegating relay, not server, requirements.

  *   Bernie

---

From: Bud Millwood <[email-masked]>
Date: Wed, 30 Oct 2019 22:52:59 +0100

Hi Bernie:

If suggestion (1) or (2), an older CPE that borks over two delegated
prefixes would stay borked until the old prefix expired. And it seems
like suggestion (3) still runs the risk of catching that older CPE.
Configuring that knob at the client level seems like a lot of
management overhead.

I wonder if introducing suggestions (1) or (2) aren't pushing existing
clients to expose new bugs they may have? Suggestion (4) seems
reasonable to me.

- Bud

On Wed, Oct 30, 2019 at 10:02 PM Bernie Volz (volz) <[email-masked]> wrote:
>
> Hello:
>
>
>
> DHC folks may want to check out the recent discussion on the v6ops mailing list (https://mailarchive.ietf.org/arch/browse/v6ops/) with the “SLAAC renum: Problem Statement & Operational workarounds” subject.
>
>
>
> Basically, one issue that appears to happen in networks where CPEs receive delegated prefixes and are renumbered when a CPE cycles power – why the renumbering occurs is not material. The issue is that some of the devices serviced by the CPE may still have configuration (i.e., from Prefix Information Options received via RAs or perhaps even DHCP) based on the original delegated prefix and if the CPE does not have stable storage (into which it stored the old delegated prefix), nothing initiates deprecating this information. This is because everyone is honoring the lifetimes, but the renumbering caused the SP DHCP server to provide a totally new delegated prefix and there’s no record in the CPE of the old.
>
>
>
> There isn’t any direct mention of sending IA_PD (or IA_NA) with 0 lifetimes in the processing of Requests, but it isn’t a stretch to consider this. And, without something, it isn’t obvious that servers should (or would) do this?
>
>
>
> Always have the server do this (sending 0 lifetimes) for a Reply to a Request for this condition.
> (1) but only for Relayed messages.
> Have a server configuration knob (at whatever granularities desired – global, prefix, client, …).
> Define a new option that a client would include in (Solicit/)Request to request this behavior. Note that (3) could always override.
>
>
>
> Hopefully existing clients would handle this correctly, though they likely would not take the action to deprecate the old delegated prefix. But we don’t know (and at least early in the DHCPv6 days, clients weren’t always happy when they received multiple delegated prefixes). Hence, how conservative should we be?
>
>
>
> Bernie
> _______________________________________________
> dhcwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dhcwg

---

From: Ted Lemon <[email-masked]>
Date: Wed, 30 Oct 2019 17:59:00 -0400

> clients to expose new bugs they may have? Suggestion (4) seems

Clients that have bugs should be updated.  Routers that aren’t updatable should be binned.   Sites that have massive numbers of non-updatable routers that can’t be updated don’t have to turn the feature on.   I think it’s fine to make this a server configuration parameter.

The benefit to doing this correctly is that it’s arguably already correct.  There are likely routers out there that will already do the right thing when they get a deprecated prefix.   The most likely use case for a deprecated prefix is not that it has a valid lifetime of zero, but that it has a preferred lifetime of zero: a graceful renumbering event.   DHCPv6 should already support graceful renumbering.   This was a design goal.

---

From: Ole Troan <[email-masked]>
Date: Wed, 30 Oct 2019 23:08:51 +0100

> The benefit to doing this correctly is that it’s arguably already correct.  There are likely routers out there that will already do the right thing when they get a deprecated prefix.   The most lik…

Right, why wouldn't this just be (accellerated) graceful renumbering as already described in 3633/8415.
I don't understand why this would expose bugs or require a new option?

Cheers,
Ole

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 30 Oct 2019 22:16:17 +0000

See Ted’s message ...

Regarding (3), I was thinking that you might configure it server wide to do this, but if you had specific broken clients, those could be excluded (perhaps based on a class, such as from a specific manufacturer, or by explicit duid). There’s probably never an issuing in providing configuration knobs ... most likely it is just the default behavior that is important.

I’m less inclined to break what is working, so I’d be a bit more cautious than Ted. Also, the “now” broken CPE will cause a customer service call and could be expensive to the SP. SP support teams also would likely need to know that they might see an increase in customer issues, which probably isn’t easy to know (how many people read release notes carefully)?

- Bernie

> On Oct 30, 2019, at 5:53 PM, Bud Millwood <[email-masked]> wrote:
> 
> Hi Bernie:
>

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 30 Oct 2019 22:23:48 +0000

I should point out that having the dhcp server supply this information may not work all the time. It really depends on what configuration changes were made and the server implementation. And of course is useless if you “move” the CPE.

So this isn’t going to be prefect (only storage on CPE will do that), but should improve things and especially for those situations where the SP configures the DHCP server to renumber periodically.

- Bernie

> On Oct 30, 2019, at 6:16 PM, Bernie Volz (volz) <[email-masked]> wrote:
> 
> See Ted’s message ...
> 
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Ted Lemon <[email-masked]>
Date: Wed, 30 Oct 2019 18:43:17 -0400

> I should point out that having the dhcp server supply this information may not work all the time. It really depends on what configuration changes were made and the server implementation. And of cou…

In this case the prefix would presumably no longer be valid; if it was valid and was offered as a deprecated prefix, it would cause no harm.

I am not suggesting that this should just start happening with no testing.   What I am suggesting is that forcing firmware updates on CPE routers isn’t something we should be as chary of as we used to be.

---

From: Bud Millwood <[email-masked]>
Date: Thu, 31 Oct 2019 00:18:28 +0100

It's not so much about the lifetime of the prefix as about putting two
prefixes in a reply to a request, right? And any CPE that can't handle
that gracefully gets hosed. I agree that providers of course need to
test this feature, and a server side configuration makes that
possible. Also, I'm all for firmware upgrades, but requiring it to fix
a hosed CPE is could be a big issue.

> This would avoid the need for the CPE to keep this information in stable storage.
> So this isn’t going to be prefect (only storage on CPE will do that)

I'm leaning towards a server-side configuration option. Creating a new
option to request something that may or may not work doesn't seem
great.

- Bud

---

From: Ted Lemon <[email-masked]>
Date: Wed, 30 Oct 2019 19:31:52 -0400

> a hosed CPE is could be a big issue.

The thing is, if they can’t handle a two-PD response, they are out of spec.  This is already allowed in the RFC.

Granted, there may be plenty of CPEs that won’t handle this correctly.   If they can be bricked by a message with two PDs, then bricking them is the right thing to do, because that’s a zero-day vulnerability wide open on the customer network.

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Thu, 31 Oct 2019 00:53:55 +0000

Mark Smith on v6ops ml wrote:

“I think Ole observed that this is contrary to what the PD prefix's Valid Lifetime said would be the case. The ISP supplied a PD Prefix with a Valid Lifetime of X seconds, and then broke that promise by abruptly changing addressing before X seconds. ISPs should be expected to live up to their Valid Lifetime promises.”

And it would be worth better understanding exactly what happens in these situations (perhaps it was covered earlier but I missed or lost that)  ... if the Prefix configuration really is radically changed, even the SP dhcp server may be unable to assist.

- Bernie

On Oct 30, 2019, at 7:32 PM, Ted Lemon <[email-masked]<mailto:[email-masked]>> wrote:

On Oct 30, 2019, at 7:18 PM, Bud Millwood <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Owen DeLong <[email-masked]>
Date: Wed, 30 Oct 2019 18:38:41 -0700

> On Oct 30, 2019, at 5:53 PM, Bernie Volz (volz) <[email-masked]> wrote:
> 

Sure, but in the real world, there is an entire class of ISPs that have repeatedly demonstrated utter and near complete disregard for such niceties as promises to customers (e.g. most major eyeball ISPs in the US at a minimum), so having CPE behavior that accommodates this fact in favor of the user will likely lead to a better user experience than stomping or feet and insisting that ISPs behave properly.

> And it would be worth better understanding exactly what happens in these situations (perhaps it was covered earlier but I missed or lost that)  ... if the Prefix configuration really is radically c…

With what is being proposed, the SP DHCP server doesn’t need to assist. The CPU should write the set of received prefixes and their expected expiration times (preferred, valid) to persistent storage. The CPU should update this when a packet making a change to these timers is received. It should make every effort to deprecate prefixes it cannot renew from the links where it previously advertised them.

This doesn’t require anything special on the SP side.

Owen

> 
> - Bernie
> 
> On Oct 30, 2019, at 7:32 PM, Ted Lemon <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> On Oct 30, 2019, at 7:18 PM, Bud Millwood <[email-masked] <mailto:[email-masked]>> wrote:

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Thu, 31 Oct 2019 01:48:28 +0000

Owen:

“The CPU should write the set of received prefixes and their expected expiration times (preferred, valid) to persistent storage. The CPU should update this when a packet making a change to these timers is received. It should make every effort to deprecate prefixes it cannot renew from the links where it previously advertised them.”

I fully agree with that. That is the best and most complete solution. No disagreement there.

I was just trying to come up with something that could help in cases where this was not reasonable.

Note that it is technically a bit more than just write to storage as a clock is also needed to know whether the prefix needs to be deprecated or not. Perhaps time can come from the network or the device can just deprecate regardless of time if the new prefix is different when it reboots?

  *   Bernie

From: Owen DeLong <[email-masked]>
Sent: Wednesday, October 30, 2019 9:39 PM
To: Bernie Volz (volz) <[email-masked]>
Cc: Ted Lemon <[email-masked]>; Bud Millwood <[email-masked]>; [email-masked]; IPv6 Operations <[email-masked]>
Subject: Re: [v6ops] [dhcwg] SLAAC renum: Problem Statement & Operational workarounds

On Oct 30, 2019, at 5:53 PM, Bernie Volz (volz) <[email-masked]<mailto:[email-masked]>> wrote:

Owen

- Bernie

_______________________________________________
v6ops mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/v6ops

---

From: Mark Smith <[email-masked]>
Date: Thu, 31 Oct 2019 13:12:30 +1100

On Thu, 31 Oct 2019, 12:40 Owen DeLong, <[email-masked]> wrote:

> “I think Ole observed that this is contrary to what the PD prefix's Valid
> Lifetime said would be the case. The ISP supplied a PD Prefix with a Valid
> Lifetime of X seconds, and then broke that promise by abruptly changing
> addressing before X seconds. ISPs should be expected to live up to their
> Valid Lifetime promises.”
>
>
> Sure, but in the real world, there is an entire class of ISPs that have
> repeatedly demonstrated utter and near complete disregard for such niceties
> as promises to customers (e.g. most major eyeball ISPs in the US at a
> minimum), so having CPE behavior that accommodates this fact in favor of
> the user will likely lead to a better user experience than stomping or feet
> and insisting that ISPs behave properly.
>

So if some enterprise network operators said to the IETF,

"We want to be able to change the VLAN assignments of any of our IPv6 hosts
randomly and unexpectedly, including at 9:10 am when everybody is checking
their email, and have the hosts handle that entirely seamlessly."

would we consider that to be a reasonable request?

> And it would be worth better understanding exactly what happens in these
> situations (perhaps it was covered earlier but I missed or lost that)  ...
> if the Prefix configuration really is radically changed, even the SP dhcp
> server may be unable to assist.
>
>
> With what is being proposed, the SP DHCP server doesn’t need to assist.
> The CPU should write the set of received prefixes and their expected
> expiration times (preferred, valid) to persistent storage. The CPU should
> update this when a packet making a change to these timers is received. It
> should make every effort to deprecate prefixes it cannot renew from the
> links where it previously advertised them.
>
> On Oct 30, 2019, at 7:32 PM, Ted Lemon <[email-masked]> wrote:
>
> The thing is, if they can’t handle a two-PD response, they are out of
> spec.  This is already allowed in the RFC.
>
> Granted, there may be plenty of CPEs that won’t handle this correctly.
> If they can be bricked by a message with two PDs, then bricking them is the
> right thing to do, because that’s a zero-day vulnerability wide open on the
> customer network.
>

---

From: Owen DeLong <[email-masked]>
Date: Wed, 30 Oct 2019 19:08:56 -0700

> On Oct 30, 2019, at 6:48 PM, Bernie Volz (volz) <[email-masked]> wrote:
> 
> Owen:
>  

Argh… That should read CPE, not CPU.
 
> I was just trying to come up with something that could help in cases where this was not reasonable.

Such as? I cannot fathom a case where this isn’t reasonable. All CPE has non-volatile memory for storing settings. We’re talking about adding 24 octets per prefix and unlikely more than 3 prefixes per interface. Assuming no more than 4 (routed) interfaces on your basic home gateway, that’s a maximum of 12 prefixes for a total of 288 octets of storage.
 
> Note that it is technically a bit more than just write to storage as a clock is also needed to know whether the prefix needs to be deprecated or not. Perhaps time can come from the network or the d…

Nope… It isn’t…. The clock is an existing function of the CPE. All you need to write is the expected expiration time. (Current clock + prefix timer) to the non-volatile storage. You don’t need to count down the timer values.

The clock can be internal or synchronized to NTP or some other time service, but virtually every CPE already has an internal clock, so there’s no additional requirement here. In fact, the CPE _HAS_ to have a clock already to know when prefixes time out without a reboot or flash renumbering event.

Owen

>  
> Bernie
>  
> From: Owen DeLong <[email-masked] <mailto:[email-masked]>> 
> To: Bernie Volz (volz) <[email-masked] <mailto:[email-masked]>>
> Cc: Ted Lemon <[email-masked] <mailto:[email-masked]>>; Bud Millwood <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>; IPv6 Operations <[email-masked] <mailto:[email-masked]>>
> On Oct 30, 2019, at 5:53 PM, Bernie Volz (volz) <[email-masked] <mailto:[email-masked]>> wrote:
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/v6ops <https://www.ietf.org/mailman/listinfo/v6ops>

---

From: Owen DeLong <[email-masked]>
Date: Wed, 30 Oct 2019 20:14:36 -0700

> On Oct 30, 2019, at 7:12 PM, Mark Smith <[email-masked]> wrote:
> 
> 
> 
> On Thu, 31 Oct 2019, 12:40 Owen DeLong, <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> "We want to be able to change the VLAN assignments of any of our IPv6 hosts randomly and unexpectedly, including at 9:10 am when everybody is checking their email, and have the hosts handle that entirely seamlessly."
> 

Probably not, but we’re not asking that and we’re not asking for the hosts to handle it.

We’re asking for CPE routers in residential and SMB environments  to handle what is already a common behavior of residential and SMB ISPs through a relatively simple CPE change.

Owen