Subject: Re: [v6ops] A broken promise - "You said PD Prefix Valid Lifetime is going to be X" (Re: SLAAC renum: Problem Statement & Operational workarounds)

From: Fernando Gont <[email-masked]>
Date: Fri, 1 Nov 2019 01:53:05 -0300

On 30/10/19 20:38, Mark Smith wrote:
> On Wed, 30 Oct 2019 at 22:02, Ole Troan <[email-masked]> wrote:
>>
> <snip>
> 
>> In the case when a requesting router loses a prefix, it can of course signal that to the hosts as proposed in Fernando's draft. But that doesn't solve the general problem, and I'm skeptical to using addressing as a reachability signal. E.g. if the CPE's upstream link flaps, is that enough to trigger deprecation of the prefix? Of course it shouldn't.
>>
> 
> Exactly. Transient network faults are supposed to be survived - that's
> why TCP makes quite a number of attempts at re-transmitting when
> packet loss occurs, rather than just immediately terminating the
> connection.
> 
> When an ISP "flash renumbers" a customer what's really happening is
> that the customer's network layer point of attachment to the network
> is being changed - the customer is being abruptly disconnected, moved
> and re-connected. Possibly that is easy to overlook because the
> customers physical location and their physical link to the network
> doesn't change.
> 
> I think Ole observed that this is contrary to what the PD prefix's
> Valid Lifetime said would be the case. The ISP supplied a PD Prefix
> with a Valid Lifetime of X seconds, and then broke that promise by
> abruptly changing addressing before X seconds. ISPs should be expected
> to live up to their Valid Lifetime promises.

"Hope" doesn't make networks run properly.

In any case, as previously noted, there are multiple scenarios that may
lead to this problem.

-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Sander Steffann <[email-masked]>
Date: Fri, 1 Nov 2019 11:22:52 +0300

Hi,

> "Hope" doesn't make networks run properly.

This isn't "Hope", this is breaking promises, and that does break networks. If you can't at least trust that promises are intended to be kept then you have no network at all...

> In any case, as previously noted, there are multiple scenarios that may

Sure, bad things can happen, and there are cases where despite the best intentions you can't keep your promise. But that doesn't mean everybody should go around making promises without even thinking about keeping them…

Cheers,
Sander

---

From: Fernando Gont <[email-masked]>
Date: Fri, 1 Nov 2019 06:29:37 -0300

> This isn't "Hope", this is breaking promises, and that does break networks. If you can't at least trust that promises are intended to be kept then you have no network at all...

You might hope the ISPs do stable prefixes, that CPEs record the leased
prefixes on stable storage, and that CPE routers does not crash and reboot.

However, 37% of surveyed ISPs
(https://indico.uknof.org.uk/event/41/contributions/542/attachments/712/866/bcop-ipv6-prefix-v9.pdf)
indicate they do dynamic prefixes, and thus are likely to be facing this
problem (possibly being masqueraded by falling back to IPv4).

You may want to declare ISPs clueless, start a crusade to switch to
stable prefixes (unlikely success), claim that CPE vendors are cheap, etc.

But at the end of the day, user experience will still suck.

(I have had cases in which I have flash-renumbered my network by
changing and reloading a RA daemon)

> Sure, bad things can happen, and there are cases where despite the best intentions you can't keep your promise. But that doesn't mean everybody should go around making promises without even thinkin…

I certainly can't speak for the 37% above. But a number of folks have
already noted (some on-list, some off-list) that they are running
networks on which this flash-renumbering events do occur.

The home network/cpe case is a notable example, but others exists. And
particularly the host-side mitigations help all of them.
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Philip Homburg <[email-masked]>
Date: Fri, 01 Nov 2019 11:41:52 +0100

> This isn't "Hope", this is breaking promises, and that does break
> networks. If you can't at least trust that promises are intended
> to be kept then you have no network at all...

Maybe somebody can do a text analysis and show which promise is broken.

The typical case is a CPE rebooting, requesting a prefix using DHCPv6 and
getting a different prefix from before.

As far as I know, there is no text that requires a DHCPv6 server to return
the previous lease if it is still valid.

The next question would be, if the CPE would try to renew the old prefix and
the DHCPv6 server declines. Is there any text the requires a DHCPv6 server
to do that.

Then the question is, if a CPE reboots can it continue to use the old lease?
Typically, link attachment requires obtaining a new lease. But I don't know
if that is specified anywhere. 

I do know that some routers do not install a forwarding entry if you don't 
get a new lease. I don't think we have any spec for that.

---

From: Ole Troan <[email-masked]>
Date: Fri, 1 Nov 2019 11:56:55 +0100

Philip,

> On 1 Nov 2019, at 11:42, Philip Homburg <[email-masked]> wrote:
> 
> 
>> 

As an author of rfc3633, let me tell what the intention of prefix delegation is. Then you can choose to bring your lawyer to argue the finer points to justify your use after. :-)

PD was intended as a fax replacement. Which was the typical provisioning mechanism at the time. The intention was that prefixes delegated had a lifetime equal to the length of the user’s contract with the ISP.

Lifetimes in PD give you an expiry date. E.g a value of 864000 means that the delegating entity promises that you are delegated and control this prefix until November 11th. Unless extended. 

There’s really little value in having both valid and preferred lifetimes in PD. 

The finer points of the DHCP machinery is really no excuse for this practice.

Ole

> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 07:05:51 -0400

> There’s really little value in having both valid and preferred lifetimes in PD. 

Ole, I think argument from authority isn’t very useful here.   I can imagine that you might think this, but…

Suppose I want to renumber a customer.   And I have one tool with which to do this: DHCP.   What is the correct behavior?   There are options:

1: just stop renewing the old prefix, but continue to route it
2: provide two prefixes, one with a preferred lifetime of zero, the other with a nonzero preferred lifetime; continue to route both until the valid lifetime on the deprecated prefix expires, at which point stop routing that prefix.
3: stop renewing the old prefix, and don’t route it.

I think that (1) is valid, and is what we would expect, but there is a problem: how does the infrastructure know that that prefix should continue to be routed?   It’s not seeing renewal traffic, and I think that’s how it knows currently.   Maybe it does DHCP leasequery?   If so, then does the DHCP server know the prefix is still valid?   I hope so.   It would be bad if the DHCP server gave out that prefix before it had expired.

(2) seems valid, and I think is correct behavior, but there’s been some theorizing on the DHC wg mailing list (and here, a bit) that some CPE devices might not be expecting this behavior, and might choke on it.

(3) This is a flash renumbering scenario.

Now, I think what you just said was that none of these scenarios are valid, because the ISP should never renumber the customer.   Is that correct?

---

From: Mark Smith <[email-masked]>
Date: Fri, 1 Nov 2019 22:12:48 +1100

> getting a different prefix from before.

It shouldn't be. That's why clients have DHCP Unique Identifiers (DUIDs)
that should be persistent across client restarts (or CPE reboots).

>From RFC 8415 (the new RFC 3315)

binding                   A binding (or client binding) is a group of
                             server data records containing the
                             information the server has about the
                             addresses or delegated prefixes in an
                             Identity Association (IA) or configuration
                             information explicitly assigned to the
                             client.  Configuration information that has
                             been returned to a client through a policy,
                             such as the information returned to all
                             clients on the same link, does not require
                             a binding.  A binding containing
                             information about an IA is indexed by the
                             tuple <DUID, IA-type, IAID> (where IA-type
                             is the type of lease in the IA -- for
                             example, temporary).  A binding containing
                             configuration information for a client is
                             indexed by <DUID>.  See below for
                             definitions of DUID, IA, and IAID.

"The DUID is
   designed to be unique across all DHCP clients and servers, and stable
   for any specific client or server.  That is, the DUID used by a
   client or server SHOULD NOT change over time if at all possible; for
   example, a device's DUID should not change as a result of a change in
   the device's network hardware or changes to virtual interfaces (e.g.,

Mrugalski, et al.            Standards Track                   [Page 32]

------------------------------

  <https://tools.ietf.org/html/rfc8415#page-33>RFC 8415
<https://tools.ietf.org/html/rfc8415>                      DHCP for
IPv6                November 2018

   logical PPP (over Ethernet) interfaces that may come and go in
   Customer Premises Equipment routers).  The client may change its DUID
   as specified in [RFC7844 <https://tools.ietf.org/html/rfc7844>]."

---

From: Ole Troan <[email-masked]>
Date: Fri, 1 Nov 2019 12:15:56 +0100

Ted,

> Now, I think what you just said was that none of these scenarios are valid, because the ISP should never renumber the customer.   Is that correct?

Neither of your options have IETF consensus behind them. DHCP PD supports renumbering, but it also says:

   "Many applications expect stable addresses.  Even though this
   mechanism makes automatic renumbering easier, it is expected that
   prefixes have a long lifespan.  During renumbering it is expected
   that the old and the new prefix co-exist for some time."

RFC4192 is as far as I know the most complete graceful renumbering description.

To continue my example:

(4) Continue to delegate prefix A with a lifetime expiring on November 11. On November 4th start also delegating prefix B with a lifetime expiring December 31st 2022.
    On November 11th, stop advertising prefix A.

This is how IPv6 renumbering has been specified for 20 years... I presume that was not news?

Cheers,
Ole

---

From: [email-masked]
Date: Fri, 01 Nov 2019 12:44:09 +0100 (CET)

> This is how IPv6 renumbering has been specified for 20 years... I presume that was not news?

A comment from an operator perspective:

- We don't "flash renumber" DHCP customers if we can avoid it.
- Sometimes a DHCP customer needs to be moved to a new router or
cluster of routers. This also implies that the customer will be
renumbered.
- We normally perform such moves as a planned activity, during
nighttime, with alerts to customers beforehand.
- We *don't* start delegating new prefixes in advance (and that
would be somewhat difficult in any case since they "belong" to
the router the customer will be moved to). We have no plans to
change this.

So - TL;DR: We don't do "flash renumbering" for fun, on the other
hand we also don't strictly follow RFC 4192.

Steinar Haug, AS2116

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 08:20:53 -0400

> This is how IPv6 renumbering has been specified for 20 years... I presume that was not news?

There have been efforts in the IETF to document how renumbering can be done in as minimally disruptive a way as possible (see e.g. Steinar’s message).   There are ISPs that believe that they are supposed to renumber end user addressing regularly to prevent (or at least complicate) prefix-based user tracking.  And there are ISPs that, if Fernando is to believed, and I think he is, do flash renumbering.

RFC 4192 doesn’t specify timeframes.   I think it’s a good document, but it doesn’t actually prescribe any detailed behavior.

So if none of the options I suggested have IETF consensus, is the answer to say “then we just assume that people will do whatever they want, and if they break things we’re okay with that?”

---

From: Ole Troan <[email-masked]>
Date: Fri, 1 Nov 2019 13:22:09 +0100

Ted,

> So if none of the options I suggested have IETF consensus, is the answer to say “then we just assume that people will do whatever they want, and if they break things we’re okay with that?

Can you please read my message one more time?

Ole

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 08:26:33 -0400

> Can you please read my message one more time?

I’m not seeing whatever it is that you think I missed.   It may help to realize that “long time” is a subjective term, and that “what the IETF has specified” is not always “what people do in practice.”   But beyond that I think I did read and comprehend what you said, so I’m don’t know why you’re asking me to read it again.  As a general rule, rather than making a broad request like this, it may help to expose whatever mental model produced the conclusion that I hadn’t fully read or understood what you wrote.   If I failed to understand it, asking me to read it again isn’t likely to fix that.

---

From: Ole Troan <[email-masked]>
Date: Fri, 1 Nov 2019 13:36:27 +0100

Ted,

> I’m not seeing whatever it is that you think I missed.   It may help to realize that “long time” is a subjective term, and that “what the IETF has specified” is not always “what people do in practi…

You said:
"So if none of the options I suggested have IETF consensus, is the answer to say “then we just assume that people will do whatever they want, and if they break things we’re okay with that?”"

In reply to:
  "(4) Continue to delegate prefix A with a lifetime expiring on November 11. On November 4th start also delegating prefix B with a lifetime expiring December 31st 2022.
   On November 11th, stop advertising prefix A."

(4) is the definition of graceful renumbering. What makes you conclude "then we just assume that people will do whatever they want"?
(especially since your options 1-3 are exactly that, people do whatever they want...)

Ole

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 08:57:31 -0400

On Nov 1, 2019, at 8:36 AM, Ole Troan <[email-masked]> wrote:
> (especially since your options 1-3 are exactly that, people do whatever they want…)

Okay, I missed a step (or five).  We have specified what a renumbering process should ideally look like, without taking into account several likely operational circumstances.  We have not specified in sufficient detail how renumbering works using DHCPv6 PD, e.g. in a home environment.  Consequently, providers do not have a viable way to follow our recommendations for how to renumber, and hence just do whatever they can think of, possibly without realizing that there might be a better way to do it.   Also, DHCP server vendors in particular do not implement what we intended that they implement, because we never said explicitly that they should do so, not specified what a CPE should expect in this situation.

Does that help?

FWIW, I was asked to consult with an operator who shall remain nameless about how to solve this problem, and the solution I came up with relied on the recommendations from RFC 4192. and further relied on DHCPv6 prefix delegation offering two prefixes, one valid and one preferred.  Unfortunately I don’t think this was ever implemented, but I still think it’s the right thing to do; what I’m realizing from this conversation is that not everybody saw this solution as obvious.   Which I think is a problem.

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Fri, 1 Nov 2019 13:09:56 +0000

> Also, DHCP server vendors in particular do not implement what we intended that they implement, because we never said explicitly that they should do so, not specified what a CPE should expect in this situation.

That’s not true. The server I work on allows you to specify an expiration time for a prefix, and we will automatically decrease lifetimes to honor that time and also stop using the prefix for new leases. This allows you to introduce a new prefix to be used for prefix delegation for graceful renumbering. Once the expiration time has passed, you can remove the old prefix as there should no longer be any active leases. We introduced this with the first release with DHCPv6 support a long time ago.

  *   Bernie

From: v6ops <[email-masked]> On Behalf Of Ted Lemon
Sent: Friday, November 1, 2019 8:58 AM
To: Ole Troan <[email-masked]>
Cc: [email-masked]
Subject: Re: [v6ops] A broken promise - "You said PD Prefix Valid Lifetime is going to be X" (Re: SLAAC renum: Problem Statement & Operational workarounds)

On Nov 1, 2019, at 8:36 AM, Ole Troan <[email-masked]<mailto:[email-masked]>> wrote:
(4) is the definition of graceful renumbering. What makes you conclude "then we just assume that people will do whatever they want"?
(especially since your options 1-3 are exactly that, people do whatever they want…)

Does that help?

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 09:17:40 -0400

> That’s not true. The server I work on allows you to specify an expiration time for a prefix, and we will automatically decrease lifetimes to honor that time and also stop using the prefix for new l…

Does it send both prefixes to the CPE, or just the new one?

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Fri, 1 Nov 2019 13:27:49 +0000

Both (until the old has expired).

There might be a condition that on a Solicit we may not send the expiring one after the preferred lifetime has elapsed – but it was a while ago that this was implemented and need to check the details. If I recall correctly, I think was done because in the early days of DHCPv6 there were clients that would not handle multiple IAPREFIXes properly – for example, they would only use the first or fail. And, if done, that might hamper removal of the old prefix in devices behind the CPE as the CPE would not learn of this prefix on a power cycle.

  *   Bernie

From: Ted Lemon <[email-masked]>
Date: Friday, November 1, 2019 at 9:18 AM
To: Bernie Volz <[email-masked]>
Cc: Ole Troan <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [v6ops] A broken promise - "You said PD Prefix Valid Lifetime is going to be X" (Re: SLAAC renum: Problem Statement & Operational workarounds)

On Nov 1, 2019, at 9:09 AM, Bernie Volz (volz) <[email-masked]<mailto:[email-masked]>> wrote:
That’s not true. The server I work on allows you to specify an expiration time for a prefix, and we will automatically decrease lifetimes to honor that time and also stop using the prefix for new leases. This allows you to introduce a new prefix to be used for prefix delegation for graceful renumbering. Once the expiration time has passed, you can remove the old prefix as there should no longer be any active leases. We introduced this with the first release with DHCPv6 support a long time ago.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 09:32:55 -0400

> There might be a condition that on a Solicit we may not send the expiring one after the preferred lifetime has elapsed – but it was a while ago that this was implemented and need to check the detai…

Is it required that the CPE request both prefixes, or are they sent regardless?   Is the deprecated prefix sent with a preferred lifetime of zero?   I think this was the question we were discussing on the DHC mailing list yesterday, and I thought you said that this would be new behavior.  What am I missing?

---

From: Ole Troan <[email-masked]>
Date: Fri, 1 Nov 2019 14:38:58 +0100

Ted,

Ok. Thanks. I don’t think I have heard from anyone who thinks there’s a gap with regards to pd and renumbering before. Or renumbering with addresses for that matter.

The main issue here seems that people want to do renumbering differently. Then we as a community have to figure out how harmful that is to the Internet. Or what other changes would be required for that. 

Cheers 
Ole

> On 1 Nov 2019, at 13:57, Ted Lemon <[email-masked]> wrote:
>

---

From: Fernando Gont <[email-masked]>
Date: Fri, 1 Nov 2019 09:29:10 -0300

On 1/11/19 08:12, Mark Smith wrote:
> 
> 
> On Fri, 1 Nov 2019, 21:42 Philip Homburg, <[email-masked]
> <mailto:[email-masked]>> wrote:
> 

RFC7844 might lead to non-stable DUIDs. You might argue that that in
some scenarios you wouldn't do this (non-portable CPEs), but in others
(portable ones) you might.

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Fri, 1 Nov 2019 13:43:59 +0000

> and I thought you said that this would be new behavior.

I think at present the exact behavior is not specified in RFC8415 and it is left up to the implementation as to how they handle these situations.

The point of documenting this was to assure that the server always returns all of the prefixes (in Reply messages) as long as the delegated prefix has a non-zero valid lifetimes. That should help “fix” the operational issues with renumbering and CPE reboots – but of course only covers the case where the server still has a record of that prefix delegation that it is able to locate. Thus, it is not a complete solution to this issue (that does require the CPE to store this information).

  *   Bernie

Date: Friday, November 1, 2019 at 9:33 AM

On Nov 1, 2019, at 9:27 AM, Bernie Volz (volz) <[email-masked]<mailto:[email-masked]>> wrote:
Both (until the old has expired).

---

From: Owen DeLong <[email-masked]>
Date: Fri, 1 Nov 2019 09:21:59 -0700

> On Nov 1, 2019, at 2:29 AM, Fernando Gont <[email-masked]> wrote:
> 

Dynamic prefixes do not necessarily imply flash renumbering and broken promises as Sander calls it.

He’s right that even in the case of dynamic prefixes, the ISP _SHOULD_ delegate or at least preserve the operability of the prefix for the duration promised in the lease, even across reboot of the delegated router, etc.

However, you’re also right that vendors and ISPs don’t always do what they should and these proposals can help mitigate the damage when that happens.

If there are problematic tradeoffs associated with these measures, they should be considered, but taking an absolutist approach of “this shouldn’t happen, so we shouldn’t make efforts to handle this case” isn’t useful IMHO.

> particularly the host-side mitigations help all of them.

I’d argue that the CPE side mitigations are the low hanging fruit here. I think it will be far faster and easier to update CPE firmware and that the changes proposed to CPE are, in fact, in most cases simpler than the host changes.

Owen

---

From: Owen DeLong <[email-masked]>
Date: Fri, 1 Nov 2019 09:42:22 -0700

> hand we also don't strictly follow RFC 4192.

Put another way, from the customer perspective…

“We don’t do flash renumbering for fun, but we do value our convenience
above the impact this has on our customers and therefore when we find it
convenient, we do flash renumber customers as needed.”

I’m not even saying this is necessarily an invalid approach. We all know
(or at least should know) that the margins in this industry don’t support
the cost involved in doing things differently, but, it does make a good
case for why we should be looking at ways to mitigate this issue because
in the real world, ISPs really do break DHCP lease promises on a regular
basis.

Owen

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 12:47:45 -0400

On Nov 1, 2019, at 12:42 PM, Owen DeLong <[email-masked]> wrote:
> the cost involved in doing things differently,

I don’t think that’s the issue.   I think the issue is that “no flash renumbering” is a nice-to-have, not a must, because the layers above layer two can adapt when layer two breaks.   We’ve done a lot of efforts to try to fix this problem at layer two, and none of them have succeeded.

This is not to say that making layer two as solid as possible shouldn’t be a goal; just that it is merely one of the tools we have for making the network reliable, not the /only/ tool.

---

From: [email-masked]
Date: Fri, 01 Nov 2019 19:19:51 +0100 (CET)

> convenient, we do flash renumber customers as needed.”

Not sure if you understood my example:

1. DHCP customer needs to be moved to a different router.
2. Because customer will be moved to a different router, customer
*will* be down for some time.
3. Moving the customer is done as a planned activity, at night, with
alert to the customer several days in advance, in order to minimize
customer inconvenience.

Also, not directly related to the above: Business customers can also
choose static addresses. If such a customer needs to be moved to a
new router, addresses are also moved.

I fail to see how this can be described as "we value our convenience
above the impact this has on our customers". But I'll stop here.

Steinar Haug, AS2116

---

From: Owen DeLong <[email-masked]>
Date: Fri, 1 Nov 2019 11:23:54 -0700

> On Nov 1, 2019, at 9:47 AM, Ted Lemon <[email-masked]> wrote:
> 
> On Nov 1, 2019, at 12:42 PM, Owen DeLong <[email-masked] <mailto:[email-masked]>> wrote:

Flash renumbering is NOT a layer 2 issue, it’s a layer 3 issue.

> This is not to say that making layer two as solid as possible shouldn’t be a goal; just that it is merely one of the tools we have for making the network reliable, not the /only/ tool.

Yeah, but we’re really talking more about layer 3 than layer 2 here.

Owen

---

From: Philip Homburg <[email-masked]>
Date: Fri, 01 Nov 2019 19:32:58 +0100

>    This is not to say that making layer two as solid as possible
>    shouldn't be a goal; just that it is merely one of the tools we
>    have for making the network reliable, not the /only/ tool.
> 
>    Yeah, but we're really talking more about layer 3 than layer 2
>    here.

Conceptual it should not be hard for MPTCP to deal with this issue.
In practice, we may have a bit of deployment problem. 

But the bigger question would be, if MPTCP would iterate over all possible
sources addresses in a happy-eyeball fashion, is that something we want?
We already have the issue that HE masks brokenness. If we promote that 
as a default mode in transport protocols, we may invite quite a big mess.

---

From: Owen DeLong <[email-masked]>
Date: Fri, 1 Nov 2019 11:58:35 -0700

> On Nov 1, 2019, at 11:19 AM, [email-masked] wrote:
> 

Well, consider this optional (but less convenient) approach:

Customers moving to other routers are generally groups of customers. (e.g. CMTS splits, etc.)

In such a case, creating a new prefix for that class of customers and advertising it to them several days in advance as well as shortening the preferred and valid lifetimes on the old prefix at that same time could be done.

The new prefix would then move with the customers.

This is inconvenient, but achievable. If you then wanted to migrate the customers to the prefix associated with the new router instead of making the transition prefix permanent, you could use short preferred and valid lifetimes on the transition prefix and start expiring it once the move was completed and the new permanent prefix had been advertised to them.

> new router, addresses are also moved.

Sure.

> above the impact this has on our customers". But I'll stop here.

See above.

Owen

---

From: Ted Lemon <[email-masked]>
Date: Fri, 1 Nov 2019 15:39:42 -0400

> Yeah, but we’re really talking more about layer 3 than layer 2 here.

Sorry, I was referring to layer 3.   I think I was still thinking about RA Guard for some reason, and that’s an L2 thing.

---

From: Fernando Gont <[email-masked]>
Date: Fri, 8 Nov 2019 09:30:18 -0300

On 1/11/19 05:22, Sander Steffann wrote:
> Hi,
> 
>>> I think Ole observed that this is contrary to what the PD 
>>> prefix's Valid Lifetime said would be the case. The ISP supplied 
>>> a PD Prefix with a Valid Lifetime of X seconds, and then broke 
>>> that promise by abruptly changing addressing before X seconds. 
>>> ISPs should be expected to live up to their Valid Lifetime 
>>> promises.
>> 
> networks. If you can't at least trust that promises are intended to 
> be kept then you have no network at all...

They are intended to be kept, but at times s* happens e.g., CPE routers
don't crash and reboot on purpose.

>> In any case, as previously noted, there are multiple scenarios
>> that may lead to this problem.
> 
> Sure, bad things can happen, and there are cases where despite the 
> best intentions you can't keep your promise. But that doesn't mean 
> everybody should go around making promises without even thinking 
> about keeping them…

Could you please point to anything that we are proposing that shouldn't
be there anyway?

Thanks!

Cheers,

---

From: Ole Troan <[email-masked]>
Date: Fri, 8 Nov 2019 13:47:18 +0100

> don't crash and reboot on purpose.

This comment worries me.
That a CE reboots is _not_ the problem, nor does it cause the problem.

Cheers,
Ole

---

From: Fernando Gont <[email-masked]>
Date: Fri, 8 Nov 2019 09:55:49 -0300

> That a CE reboots is _not_ the problem, nor does it cause the problem.

Well, if the CPE did keep state, there would be no problem. (is there a
formal requirement for prefixes to be stable across crash & reboots?).

(just as a reminder, this is just one scenario. there have been at least
three operators that have noted that their setups experience renumbering
scenarios for other reasons... and slaac could certainly do much better
in those)

Thanks,

---

From: Ole Troan <[email-masked]>
Date: Fri, 8 Nov 2019 14:09:29 +0100

> formal requirement for prefixes to be stable across crash & reboots?).

CPE is not required to keep state.
In PD the DR is in control, and should continue to delegate the same prefix as previously delegated.
If the RR includes prefixes in the request, those are purely there as hints for prefixes it would prefer.

> scenarios for other reasons... and slaac could certainly do much better

You have to be clear what the root cause of a problem is.
The DHCP PD protocol is designed to carry leases over CPE reboots.

If an SP uses the opportunity caused by a reboot to flash renumber, then say so.

Ole

---

From: Alexandre Petrescu <[email-masked]>
Date: Fri, 8 Nov 2019 14:15:42 +0100

Le 08/11/2019 à 13:55, Fernando Gont a écrit :
>>>>>> prefix's Valid Lifetime said would be the case. The ISP
>>>>>> supplied a PD Prefix with a Valid Lifetime of X seconds,
>>>>>> and then broke that promise by abruptly changing addressing
>>>>>> before X seconds. ISPs should be expected to live up to
>>>>>> their Valid Lifetime promises.
>>>>> 
>>>> This isn't "Hope", this is breaking promises, and that does
>>>> break networks. If you can't at least trust that promises are
>>>> intended to be kept then you have no network at all...
>>> 
>>> They are intended to be kept, but at times s* happens e.g., CPE
>>> routers don't crash and reboot on purpose.
>> 
>> This comment worries me. That a CE reboots is _not_ the problem,
>> nor does it cause the problem.
> 
> Well, if the CPE did keep state, there would be no problem. (is there
> a formal requirement for prefixes to be stable across crash &
> reboots?).

Probably there is no formal requirement for prefixes to be stable across
crashes and reboots[*], but there is a behaviour of the client to send
CONFIRM after reboot or wake-up from sleep, as described in the RFC DHCPv6.

That is how my home network work with DHCPv6 for addresses: the Windows
client reboots and sends CONFIRM to DHCPv6 server on CPE, who agrees
with it in turn.  So the same address is being used by the Client after
reboot.

I suspect it should work the same way when PD is implemented.

Alex

[*]: as for formal requirements for prefix stability:
- there are commitments from certain cellular operator to maintain a
stable IPv4 address for a cellular User Equipment;
- there are cellular operators open to at least try to provide stable
IPv6 addresses to smartphone users by using DHCPv6.
- there are are commitments (some times unwritten, but visible in their
design of use of static IPv6 addresses per intermediary equipment) from
some ADSL ISPs to provide stable IPv4 and IPv6 addresses to home networks;

> 
> (just as a reminder, this is just one scenario. there have been at
> least three operators that have noted that their setups experience
> renumbering scenarios for other reasons... and slaac could certainly
> do much better in those)
> 
> Thanks,
>

---

From: Philip Homburg <[email-masked]>
Date: Fri, 08 Nov 2019 15:20:10 +0100

> Probably there is no formal requirement for prefixes to be stable
> across crashes and reboots[*], but there is a behaviour of the
> client to send CONFIRM after reboot or wake-up from sleep, as
> described in the RFC DHCPv6.

Sending a CONFIRM after a reboot requires that the client writes the
lease to presistant storage.

As far as I know, there is no requirement for clients to have suitable
persistent storage.

Of course, the big issue is: do we want to delay IPv6 by making IPv6
deliberately incompatible with common IPv4 deploy strategies? If the answer
is that we don't care about IPv6 roll-out, then indeed there is no need
to fix this scenario.

There are other scenarios, but I believe they should be fixed in the host.
Or not at all, if we also declare those operational errors.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 8 Nov 2019 16:30:31 +0100

why would we have to delay deployment?

> On Nov 8, 2019, at 15:20, Philip Homburg <[email-masked]> wrote:
> 
> 
>>

---

From: Mark Smith <[email-masked]>
Date: Sat, 9 Nov 2019 08:50:07 +1100

One of the best questions to often ask is "why?"

On Sat, 9 Nov 2019, 01:20 Philip Homburg, <[email-masked]>
wrote:

> deliberately incompatible with common IPv4 deploy strategies?

Why do you think that this "strategy" exists in IPv4 deployments?

Why is it relevant and preferable today for IPv6?

Why is it best for IPv6 when people get something different in IPv6 than in
IPv4 - public address space to use on their LAN.

If the answer

---

From: Fernando Gont <[email-masked]>
Date: Fri, 8 Nov 2019 19:15:41 -0300

On 8/11/19 18:50, Mark Smith wrote:
> On Sat, 9 Nov 2019, 01:20 Philip Homburg, <[email-masked]
> Why is it best for IPv6 when people get something different in IPv6 than
> in IPv4 - public address space to use on their LAN.

Asking "why" is nice. BUt as for many other uses of the question, it
doesn't solve the problem. Over 30% of IPv6 deployments do dynamic
prefixes, and hence are prone to face this. That's the deployed reality.

That aside, I reiterate this is *one of many* possible scenarios where
this issue may be faced.

Thanks,

---

From: Mark Smith <[email-masked]>
Date: Sat, 9 Nov 2019 10:41:04 +1100

On Sat, 9 Nov 2019, 09:15 Fernando Gont, <[email-masked]> wrote:

> doesn't solve the problem.

It gets to the root of the problem, so you can determine where the problem
truly exists, and where to then prevent the problem.

The Five whys method suggests you need to ask "why" around 5 times to get
to the root cause of a problem.

Five whys
https://en.m.wikipedia.org/wiki/Five_whys

Prevention is better than cure or mitigation.

Mitigation is all that is being proposed here. As someone who has
implemented and had had distributed mitigations for this in 'radvd' (the
DecrementLifetimes and DeprecatePrefix options), they're not as effective
as preventing the problem where it actually exists.

Have a look at how horrible the problem gets and how ineffective these
mitigations are when someone has a routed network using the dynamic PD
prefix.

https://github.com/reubenhwk/radvd/issues/116

Over 30% of IPv6 deployments do dynamic
> prefixes, and hence are prone to face this. That's the deployed reality.

Why are they doing it that way when it causes problems for their customers?
Why don't we know the answer to that question?

Why do we always have to try to solve technology problems with more
technology, when sometimes education about a different and better way to
solve the problem can be cheaper, quicker and more effective?

Why is "the ISP is always right" assumed?

Why aren't the IETF allowed to say to ISPs, "this is not the way this
protocol was designed to be deployed, the correct way is 'X'"?

> That aside, I reiterate this is *one of many* possible scenarios where

If it is common, why haven't we heard about it here in the IETF in any
other context other than ISPs doing dynamic PD prefixes?

---

From: Philip Homburg <[email-masked]>
Date: Mon, 11 Nov 2019 09:47:00 +0100

>why would we have to delay deployment?

Assume that an ISP has a setup with IPv4 that results in flash renumbering.
On IPv4 that is masked by NAT in the CPE.

On IPv6 that results in downtime for the customer because the problem is
not masked.

So the ISP delays rolling out IPv6 until either the problem is fixed in
IPv6 or, less likely, the ISP has a chance to re-engineer their setup to
avoid flash renumbering.

---

From: Philip Homburg <[email-masked]>
Date: Mon, 11 Nov 2019 09:52:23 +0100

>Why do you think that this "strategy" exists in IPv4 deployments?

One way to end up in this scenario is to have a pool of IPv4 addresses in
the access router. A redundant access network makes it possible for 
customers to end up on different access routers.

So when the CPE reboots, the line goes down, and the customer gets assigned
to the best access router. If that is a different one from before then
the customer experiences flash renumbering.

>Why is it relevant and preferable today for IPv6?

We want to make it easy for ISPs to turn on IPv6. If we first make them 
redesign their networks then it will be more costly and take more time,
so there will be less IPv6.

>IPv4 - public address space to use on their LAN.

It doesn't have much to with wether addresses are public or not.
Flash renumbering happens with public IPv4 addresses.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 11 Nov 2019 06:01:26 -0500

> IPv6 or, less likely, the ISP has a chance to re-engineer their setup to

The ISP has to install a DHCPv6 server in order to do an IPv6 deployment.  We’ve already heard from one of the main vendors of DHCPv6 servers that their server behaves correctly and does not do flash renumbering.   The two DHCPv6 servers that I have worked on don’t either.  So why is the ISP going to have to delay?  The one time I was asked to help an ISP to set up DHCPv6 in a way that would allow frequent renumbering, it was a major engineering effort (which we wound up not doing).   The ISP would have to go out of their way to not support stable prefixes.

So where is the impediment to deployment?

---

From: Richard Patterson <[email-masked]>
Date: Mon, 11 Nov 2019 16:21:33 +0000

On Mon, 11 Nov 2019 at 11:02, Ted Lemon <[email-masked]> wrote:

>
> The ISP has to install a DHCPv6 server in order to do an IPv6 deployment.
> We’ve already heard from one of the main vendors of DHCPv6 servers that
> their server behaves correctly and does not do flash renumbering.   The two
> DHCPv6 servers that I have worked on don’t either.  So why is the ISP going
> to have to delay?  The one time I was asked to help an ISP to set up DHCPv6
> in a way that would allow frequent renumbering, it was a major engineering
> effort (which we wound up not doing).   The ISP would have to go *out of
> their way* to not support stable prefixes.
>

This is an oversimplification, ignoring the complexities introduced by
various topology, architectural and indeed vendor decisions.
i.e.
- Centralised vs distributed DHCP server.
Distributed helps scale but centralised is easier to manage.
Centralised would require some topology-awareness to know where a
subscriber is to be numbered from, unless you're OK with deaggregation and
the scaling impact it has.

- Via a relay or direct?
Direct could mean large broadcast domains and may not scale, but helps
minimise the need for the server to be topology-aware.
Relays would help scale but will need to keep state, and to completely
avoid flash renumbering during planned or unplanned outages, they would
need to write leases to persistent storage also.    (see
draft-fkhp-dhc-dhcpv6-pd-relay-requirements
)

- DHCPv6 server colocated on the BNG/CMTS, or an off-box solution?
Off-box solution is yet-another-thing to manage and scale.  Presumably
there'll also need to be an IPAM, if we're to do 100% fixed statics, just
for subscriber assignments.
BNG/CMTS Vendor server may have it's on constraints and limitations.

As a network operator, we can make sure we do as much as we can to keep PDs
stable and honour leases up to the lease-time advertised, but there will
always be the risk that sometimes we have to revoke the validity of a
lease, either intentionally or unintentionally.  And sadly in the worse
case scenarios, we may not be able to offer the same PD or NA as previously
leased.
If we can offer some recommendations to CPE & DHCP client/relay/server
developers, as well as Operators, to help minimise this impact, I think
it's worthwhile to do.
If we can slightly tweak protocols to help minimise this impact, I
understand that it may be a contentious idea, but would like to think it's
also worthwhile looking in to?

---

From: Ted Lemon <[email-masked]>
Date: Mon, 11 Nov 2019 12:11:16 -0500

Nobody is expecting you to never renumber.   Well, at least I am not. What I am talking about here is deliberate, opportunistic renumbering. If your topology changes and you have to renumber, so be it. If you can avoid that it’s better, and if you can give notice it’s also better. If you can do it gracefully with temporary routes, even better. But you shouldn’t be renumbering just ‘cuz. 

>> On Nov 11, 2019, at 11:21, Richard Patterson <[email-masked]> wrote:
> 
> This is an oversimplification, ignoring the complexities introduced by various topology, architectural and indeed vendor decisions.
> i.e.
> Centralised would require some topology-awareness to know where a subscriber is to be numbered from, unless you're OK with deaggregation and the scaling impact it has. 
> 
> - Via a relay or direct?
> Direct could mean large broadcast domains and may not scale, but helps minimise the need for the server to be topology-aware.
> Relays would help scale but will need to keep state, and to completely avoid flash renumbering during planned or unplanned outages, they would need to write leases to persistent storage also.    (see draft-fkhp-dhc-dhcpv6-pd-relay-requirements )
> 
> Off-box solution is yet-another-thing to manage and scale.  Presumably there'll also need to be an IPAM, if we're to do 100% fixed statics, just for subscriber assignments.
> As a network operator, we can make sure we do as much as we can to keep PDs stable and honour leases up to the lease-time advertised, but there will always be the risk that sometimes we have to revoke the validity of a lease, either intentionally or unintentionally.  And sadly in the worse case scenarios, we may not be able to offer the same PD or NA as previously leased.
> If we can offer some recommendations to CPE & DHCP client/relay/server developers, as well as Operators, to help minimise this impact, I think it's worthwhile to do.
> If we can slightly tweak protocols to help minimise this impact, I understand that it may be a contentious idea, but would like to think it's also worthwhile looking in to?

---

From: Richard Patterson <[email-masked]>
Date: Mon, 11 Nov 2019 18:55:28 +0000

On Mon, 11 Nov 2019 at 17:11, Ted Lemon <[email-masked]> wrote:

> Nobody is expecting you to never renumber.   Well, at least I am not. What
> I am talking about here is deliberate, opportunistic renumbering. If your
> topology changes and you have to renumber, so be it. If you can avoid that
> it’s better, and if you can give notice it’s also better. If you can do it
> gracefully with temporary routes, even better. But you shouldn’t be
> renumbering just ‘cuz.
>

In that case I think we agree, an operator shouldn't intentionally renumber
"just 'cuz".  However the scenarios I mentioned are far from the Operator
going *out of their way* to not support stable prefixes. The question is
just around how stable they are, and once you start having to ask that
question, then shouldn't we be looking at ways to mitigate the impact to an
end-user?

---

From: Ted Lemon <[email-masked]>
Date: Mon, 11 Nov 2019 14:31:43 -0500

Definitely. 

> On Nov 11, 2019, at 13:55, Richard Patterson <[email-masked]> wrote:
> 
> 
> In that case I think we agree, an operator shouldn't intentionally renumber "just 'cuz".  However the scenarios I mentioned are far from the Operator going out of their way to not support stable prefixes. The question is just around how stable they are, and once you start having to ask that question, then shouldn't we be looking at ways to mitigate the impact to an end-user?
>

---

From: Mark Smith <[email-masked]>
Date: Tue, 12 Nov 2019 06:47:22 +1100

On Tue, 12 Nov 2019, 04:11 Ted Lemon, <[email-masked]> wrote:

> Nobody is expecting you to never renumber.   Well, at least I am not.
>

I'm not either.

IPv6's Preferred and Valid lifetime mechanisms are for planned renumbering.

What I am talking about here is deliberate, opportunistic renumbering. If
> your topology changes and you have to renumber, so be it.
>

Unplanned "flash" renumbering due to a topology change implies an unplanned
"flash" topology change. That's going to be a rare event, and ideally
shouldn't ever occur, because I think it is a sign that some other planning
hasn't been occurring.

I don't really understand what is so hard about all this.

You either provide prefix stability using a client's DUID, or you can use
RADIUS.

Stable PD prefixes are similar to IPv4 RADIUS Framed-Routes.

IETF RADIUS attributes exist to provide IPv6 PD prefixes to BNGs that use
them with their local DHCPV6-PD server - see RFC 6911.

This is the method used on the IPv6 residential broadband project I worked
on in 2010, and I'm pretty confident it is still being used today, as I've
been a customer of it for the past 8 years.

If you can avoid that it’s better, and if you can give notice it’s also
> better. If you can do it gracefully with temporary routes, even better. But
> you shouldn’t be renumbering just ‘cuz.
>

>> The ISP has to install a DHCPv6 server in order to do an IPv6
>> deployment.  We’ve already heard from one of the main vendors of DHCPv6
>> servers that their server behaves correctly and does not do flash
>> renumbering.   The two DHCPv6 servers that I have worked on don’t either.
>> So why is the ISP going to have to delay?  The one time I was asked to help
>> an ISP to set up DHCPv6 in a way that would allow frequent renumbering, it
>> was a major engineering effort (which we wound up not doing).   The ISP
>> would have to go *out of their way* to not support stable prefixes.
>>
>
> need to write leases to persistent storage also.    (see draft-fkhp-dhc-dhcpv6-pd-relay-requirements
> )
>
> As a network operator, we can make sure we do as much as we can to keep
> PDs stable and honour leases up to the lease-time advertised, but there
> will always be the risk that sometimes we have to revoke the validity of a

---

From: Ted Lemon <[email-masked]>
Date: Mon, 11 Nov 2019 15:05:16 -0500

On Nov 11, 2019, at 2:47 PM, Mark Smith <[email-masked]> wrote:
> Unplanned "flash" renumbering due to a topology change implies an unplanned "flash" topology change. That's going to be a rare event, and ideally shouldn't ever occur, because I think it is a sign that some other planning hasn't been occurring.

It’s not quite as simple as that.   A topological bifurcation (or any similar division) should be pretty easy to manage without flash renumbering, because you can just renumber before you change the topology.   Assuming you have time.  But anything more complicated is going to require the propagation of network routes for each customer to multiple places in the network.   I think this is eminently doable, but it’s not trivial, and I wouldn’t be surprised if it’s not something an operator can easily do with their existing tooling.

---

From: Mark Smith <[email-masked]>
Date: Tue, 12 Nov 2019 08:05:56 +1100

On Tue, 12 Nov 2019, 05:56 Richard Patterson, <[email-masked]> wrote:

>> your topology changes and you have to renumber, so be it. If you can avoid
>> that it’s better, and if you can give notice it’s also better. If you can
>> do it gracefully with temporary routes, even better. But you shouldn’t be
>> renumbering just ‘cuz.
>>
>
> In that case I think we agree, an operator shouldn't intentionally
> renumber "just 'cuz".  However the scenarios I mentioned are far from the
> Operator going *out of their way* to not support stable prefixes.
>

"What you allow, you encourage." - Michael Josephson.

If the IETF believe that stable prefixes are the correct solution, because
that is the Internet architecture (transport layer connections are to
tolerate transient connectivity faults through retransmission, and that
relies on persistent host IP addresses across transient connectivity
faults), then I don't think the IETF should try to purposefully accommodate
unstable prefixes.

Purposefully accommodating unstable prefixes will only encourage ISPs to
use them.

If an ISP chooses to provide a bad experience to their customers through
unstable prefixes, then the pain and consequences of that decision should
be entirely borne by the ISP, rather than them then expecting the IETF, CPE
and host implementation vendors to then come up with and wear the
development and deployment costs of imperfect work arounds.

These ISPs are choosing to create a problem that doesn't need to exist.

The question is just around how stable they are,

The answer is easy.

What the ISP said it would be in the Valid Lifetime field.

It's a broken promise if it isn't, and ISPs should only break promises when
they have no other choice i.e. emergency network reconfiguration in a dire
fault situation.

If an ISP wants to provide short life addresses, they can specify a
ValidLifetime as low as 2 hours, per RFC 4862, 5.5.3, (e).

That would seem to me to be contradictory to the "always on" nature of the
Internet service they're selling, however that's up to them as long as they
wear the consequences of that decision.

and once you start having to ask that question, then shouldn't we be
> looking at ways to mitigate the impact to an end-user?
>

---

From: Ole Troan <[email-masked]>
Date: Mon, 11 Nov 2019 22:25:28 +0100

> In that case I think we agree, an operator shouldn't intentionally renumber "just 'cuz".  However the scenarios I mentioned are far from the Operator going out of their way to not support stable pr…

I agree with you, we should look at ways to mitigate the impact for the end-user.
Can we find a way that do that without relegating IPv6 end-users to be clients and consumers only, like they are on the NATed IPv4 Internet?

Would you join in effort to try to define how that would look like and what the gaps are?
The big question:
"What would it take to make a network with ephemeral addressing work (work as in not limit the user to only run "client only" applications)?"

And sure, there might be lower hanging fruits, but let's not pick those that can serve as encouragement that this addressing model works correctly today.

Best regards,
Ole

---

From: Mark Andrews <[email-masked]>
Date: Tue, 12 Nov 2019 08:56:51 +1100

> On 12 Nov 2019, at 08:25, Ole Troan <[email-masked]> wrote:
> 

One low piece of hanging fruit is getting nodes to register themselves in the DNS
when their addresses change.  This is easy to do with DNS UPDATE (RFC 2136) and SIG(0)
(RFC 2931).  It just requires that a KEY record be added to the DNS for the UPDATE
request to be authenticated against when the node is commissioned.  That can be done
with DNS UPDATE and TSIG (RFC 2845) or SIG(0).

Similarly registering PTR records for themselves when they get new addresses.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Fernando Gont <[email-masked]>
Date: Mon, 11 Nov 2019 20:35:56 -0300

On 11/11/19 16:47, Mark Smith wrote:
> 
> 
> On Tue, 12 Nov 2019, 04:11 Ted Lemon, <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     What I am talking about here is deliberate, opportunistic
>     renumbering. If your topology changes and you have to renumber, so
>     be it. 
> 
> 
> Unplanned "flash" renumbering due to a topology change implies an
> unplanned "flash" topology change. That's going to be a rare event, and
> ideally shouldn't ever occur, because I think it is a sign that some
> other planning hasn't been occurring.
> 
> 
> You either provide prefix stability using a client's DUID, or you can
> use RADIUS.

As noted elsewhere, we have an RFC for DUID radnomization ("dhcp
anynimity profile", iirc). Besides, it's also understandable the
possible expectation of a user to get a new prefix upon CPE reboot (due
to privacy considerations).

---

From: Fernando Gont <[email-masked]>
Date: Mon, 11 Nov 2019 21:13:06 -0300

> "What would it take to make a network with ephemeral addressing work (work as in not limit the user to only run "client only" applications)?"

+ IPv6 UPnP support for punching holes in the fw (a big fail for many CPEs)
+ Resilience to renumbering (slaac-renum)
+ Dynamic DNS updates

In the IPv4 world, assuming no CGNAT, this works just fine:

+ there's IPv4 UPnP suport
+ renumbering is masqueraded by NATs
+ You do dynamic DNS updates with small TTLs

P.S.: Since nobody guarantees there will be no fw and not renumbering,
you have to accommodate for them, or plan to fail.

Thanks,

---

From: Mark Smith <[email-masked]>
Date: Tue, 12 Nov 2019 11:33:48 +1100

On Tue, 12 Nov 2019 at 10:38, Fernando Gont <[email-masked]> wrote:

<snip>

> to privacy considerations).

The trouble with this discussion is that it still hasn't defined the
problem properly. Different people have different scenarios in mind,
and then scenarios are sometimes changed to suit an argument.

What are the specific design requirements?

Is the fundamental goal a stable prefix or not?

To suit commonly deployed transport layer protocols' requirements,
address stability is needed so that connections can survive transient
connectivity faults - which is what a CPE reboot is, or what a BRAS
reboot is.

On the other hand, the privacy argument says that an unstable prefix
is optimal, with as unstable as possible the most optimal. Addresses
used by transport layer protocols should only last as long as the
transport layer protocol connection using the address and no longer -
basically "Transient Addressing for Related Processes: Improved
Firewalling by Using IPV6 and Multiple Addresses per Host" -
https://www.cs.columbia.edu/~smb/papers/tarp/tarp.html

Are we trying to address a fault situation, or an intentional,
non-fault action to achieve some other goal (CPE reboot to acquire new
prefix for privacy)?

Is the "flash" renumber event a very rare event that an ISP does as a
last resort during troubleshooting, despite them having infrastructure
to support providing stable prefixes (i.e. RADIUS/DUID above)? Say
occurring no more than once every 2 years, ideally no more than once
every 5, and ideally never.

Is the "flash" renumber event a reasonably rare event that coincides
with an occasional BRAS/BNG reboot that causes customers to land on a
different BRAS/BNG with a different PD pool, and the ISP hasn't gone
to the effort of putting in stable prefix infrastructure? Say
occurring no more than once every six months (that was my minimum up
time expectation for a production BRAS/BNG).

Is the "flash" renumber event a reasonably rare event triggered by a
customer who wants a stable prefix for as long as they're comfortable
with the privacy they have, and will actively trigger a prefix change
when they consider they need to "reinitialise" their privacy with a
different prefix? Say once every month or two.

Is the "flash" renumber event a fairly common event because ISPs don't
want to provide stable prefix infrastructure, because they've divided
the their customers into two classes - "clients only" and a higher
paying, typically SOHO/SME premium "full service" (client, and
server/service if desired - i.e. peer) class. Say, once every week or
two for "client only" residential customers.

Until requirements are locked down, we can never know if you've
achieved them or not, and the discussion goes in never ending circles
- as it is.

All of the above are impossible to achieve at once because there are
unresolveable conflicts.

For example, for an ISP that provides a stable prefix, how can they
know that a CPE reboot is intentional to get a different prefix for
privacy? It could also be because of a mains power failure, or a CPE
reboot due to a bug.

---

From: Owen DeLong <[email-masked]>
Date: Mon, 11 Nov 2019 20:16:38 -0800

> On Nov 11, 2019, at 00:52 , Philip Homburg <[email-masked]> wrote:
> 

Does anyone actually do IPv4 this way? I am not aware of anyone. To my knowledge,
the two most common IPv4 multihoming scenarios are:

1.	Public addressing of hosts using an RIR-issued prefix or a prefix from one
	provider routed to two providers via local BGP announcement.

2.	An external address (pool) from each provider which is used for source NAPT
	of outbound packets from internal hosts using RFC-1918 or similar addressing.

Neither of these scenarios has the issue described. In scenario 1, there is no
such thing as renumbering without notice.

In scenario 2, renumbering affects only the outbound NAPT router(s). Sessions may
be dropped in a failure, but retries will receive different external source addresses
via NAPT and be routed via said alternate provider.

> Flash renumbering happens with public IPv4 addresses.

This hasn’t been my experience. However, if the prior poster can’t understand
the benefits of end-to-end addressing in IPv6 vs. the lack thereof in the majority
of IPv4 deployments, then it’s difficult to know where to start in explaining it.

Owen

---

From: Owen DeLong <[email-masked]>
Date: Mon, 11 Nov 2019 20:18:12 -0800

> On Nov 11, 2019, at 03:01 , Ted Lemon <[email-masked]> wrote:
> 

We’ve also heard from at least one cable provider how flash renumbering is inflicted on their customers due to splits/combines/moves of blocks of customers from one CMTS to another.

Owen

---

From: Tassos Chatzithomaoglou <[email-masked]>
Date: Tue, 12 Nov 2019 09:11:40 +0200

> privacy? It could also be because of a mains power failure, or a CPE

The subscriber should have the choice to decide and the technology should give the means to support his/her choices.
We're building a page inside our subscriber portal, where the subscriber chooses how frequently the LAN IPv6 assigned prefix should change.
Default is to change at every PPP reconnect, but the subscriber can choose specific periods (x weeks/months) or action (once at next reconnect).
SOHO/SME accounts do not have these options and have stable prefixes forever. Non-SOHO/SME accounts are not guaranteed to follow subscriber's choice due to network conditions.

--
Tassos

---

From: Fernando Gont <[email-masked]>
Date: Tue, 12 Nov 2019 11:28:31 -0300

> Is the fundamental goal a stable prefix or not?

It can't be a fundamental goal. It is not stated anywhere, and that ship
has sailed: over 30% of surveyed ISPs do dynamic prefixes. SO that ship
has sailed. Besides, as noted by numerous folks on-list, even when it's
not the goal, it can happen. And when it does happen, you should ba able
to handle it gracefully.

> connectivity faults - which is what a CPE reboot is, or what a BRAS

Not really. IN a world where in the vast majority of cases a CPE does
NAT (in IPv4) or a stateful firewall, when the firewall reboots, you
loose state, and with it your ongoing connections.

> non-fault action to achieve some other goal (CPE reboot to acquire new

Mostly a fault situation that, at times, may arise due to an intentional
action. (e.g. an ISP moving customers around, a user "fixing" their
network by reseting the CPE, etc.)

> every 5, and ideally never.

You can't know. BUt if you handle it gracefully, it doesn't matter.

> different prefix? Say once every month or two.

May happen. And may also happen when a user just decides to "fix" his
network by rebooting the CPE.

[...]
> achieved them or not, and the discussion goes in never ending circles

I don't think we need to enumerate every possible reason for which this
might happen. In fact, it would probably be impossible to do so.

For example, when you implement resiliency to packet loss or duplication
in TCP, you don't go around analyzing every possible reason for which
that might happen. As long as you have concrete indications that the
scenario is possible, you design mechanisms to mitigate the problems.

> privacy? It could also be because of a mains power failure, or a CPE

I think you're trying to boil the ocean. This work is on resiliency to
flsh-renumbering. The scenario of an ISP trying to figure out whether
they should deliver the same or different prefix (by inferring what the
user wants) is a totally different problem.

I reiterate a previous question, that remains unanswered: is there in
cpe-slaac-renum that you think shouldn't be there anyway?

Thanks!

Cheers,

---

From: Fernando Gont <[email-masked]>
Date: Tue, 12 Nov 2019 11:33:15 -0300

>> On Tue, 12 Nov 2019 at 10:38, Fernando Gont <[email-masked]> wrote:
[....]
> Default is to change at every PPP reconnect, but the subscriber can choose specific periods (x weeks/months) or action (once at next reconnect).

So, upon a CPE crash and reboot, the subscriber would get a different
prefix. Right?

Thanks!

Cheers,

---

From: Tassos Chatzithomaoglou <[email-masked]>
Date: Tue, 12 Nov 2019 23:20:16 +0200

> So, upon a CPE crash and reboot, the subscriber would get a different
Yes, the current default is for the subscriber to get a different prefix after a CPE crash and reboot (like in IPv4). But a CPE crash is not a common case. The most common case for prefix change is a DSL disconnect (due to line issues) or a PPP disconnect (due to network issues).
Also, many of our CPEs store internally the prefix, so after a PPP reconnect they inform the LAN hosts about its release.

--
Tassos

---

From: Mark Smith <[email-masked]>
Date: Wed, 13 Nov 2019 09:28:49 +1100

On Wed, 13 Nov 2019 at 08:20, Tassos Chatzithomaoglou
<[email-masked]> wrote:
> Yes, the current default is for the subscriber to get a different prefix after a CPE crash and reboot (like in IPv4).

"like in IPv4"

Why are you giving customers delegated IPv6 prefixes - that's not like IPv4?

Other than "like in IPv4", why are you making the PD prefix change
upon reconnect when you don't have to?

What if your competitors provide stable prefixes, and their customers
get a better service experience than your customers do, without having
to wait possibly years for CPE and/or host upgrades per this ID's
proposal?

What is stopping them doing that? If they already have stable prefix
supporting infrastructure, why wouldn't they switch it on when they
can, when it could mean stealing customers from you and reducing their
numbers of helpdesk calls?

>But a CPE crash is not a common case. The most common case for prefix change is a DSL disconnect (due to line issues) or a PPP disconnect (due to network issues).

---

From: Fred Baker <[email-masked]>
Date: Wed, 13 Nov 2019 10:38:11 +0530

> On Nov 12, 2019, at 8:03 PM, Fernando Gont <[email-masked]> wrote:
> 

Change "would" to "could", and you're probably close. In general, I would expect ISPs to not want to randomly change their routing, and so it would not be deterministic - if the customer is no longer using a prefix (e.g., the CPE crashed and routing has changed anyway), when the CPE comes back up they might give it a different prefix. However, I'd be surprised if they did it "just because".

---

From: Tassos Chatzithomaoglou <[email-masked]>
Date: Wed, 13 Nov 2019 09:32:49 +0200

> Why are you giving customers delegated IPv6 prefixes - that's not like IPv4?
Between ULA+NAT66 and PD we chose PD due to CPEs supporting it by default (and following the end-to-end connectivity principle).
> upon reconnect when you don't have to?
We do the same in IPv4 because many subscribers had asked for it. They wanted to easily (disconnect the line or reboot the CPE) change addresses.
> numbers of helpdesk calls?
That's mostly a commercial decision. Commercially we want to differentiate static vs dynamic.
The biggest technical issue i see is routing de-aggregation due to many different BRAS serving the same customers.
Aggregating at another level is not an option, because we don't want to have hundreds of thousands of /56s at this level.

Note: Largest percentage of our IPv6 related helpdesk calls is about DS-Lite subscribers having problems using a service requiring port forwarding (and their CPE doesn't support PCP), which we then change to the default Dual-Stack.

---

From: Richard Patterson <[email-masked]>
Date: Wed, 13 Nov 2019 10:50:28 +0000

On Wed, 13 Nov 2019 at 05:08, Fred Baker <[email-masked]> wrote:

> Change "would" to "could", and you're probably close. In general, I would
> expect ISPs to not want to randomly change their routing, and so it would
> not be deterministic - if the customer is no longer using a prefix (e.g.,
> the CPE crashed and routing has changed anyway), when the CPE comes back up
> they might give it a different prefix. However, I'd be surprised if they
> did it "just because".
>

Depends on the vendor.  For example, one of our BNGs (that's operating as a
DHCPv6 server) will keep a lease in a "held" state for a configurable
duration after the Valid Lifetime expires.  We set that value to 7 days,
allowing a CPE to come back and get the same PD within a week.

A different BNG vendor of ours however, doesn't have this configurable hold
time concept; once the Valid LIfetime expires, the MAC+DUID binding to the
PD is cleared.
They do have setting to retain the MAC+DUID->PD lease binding indefinitely,
however they didn't satisfactorily answer my question about cleanup if
these bindings are exhausted (CPEs that use dynamic DUID values perhaps?
Even though even the DUID-LLT method is *supposed* to be stable).

---

From: Ted Lemon <[email-masked]>
Date: Wed, 13 Nov 2019 10:18:53 -0500

> We do the same in IPv4 because many subscribers had asked for it. They wanted to easily (disconnect the line or reboot the CPE) change addresses.

Supposing IPv6 customers ask for this.   Why would you not give it to them with graceful renumbering rather than flash renumbering?

> That's mostly a commercial decision. Commercially we want to differentiate static vs dynamic.

So you are charging customers to not do something that costs you money and them reliability to do.   Customers who pay you get nothing, and customers who don’t pay you get what amounts to a punishment.   I guess that’s your prerogative, but again, if you are going to do this, why not do it right?  Don’t flash renumber—renumber gracefully.   You still are able to punish customers who don’t pay you, but you aren’t creating a support problem for yourself in the process.

> The biggest technical issue i see is routing de-aggregation due to many different BRAS serving the same customers.

So am I to understand that you have many different routes to the customer, and no way to be consistent about which route the customer gets when they disconnect and reconnect?   Isn’t that a problem with the BRAS setup?

> Aggregating at another level is not an option, because we don't want to have hundreds of thousands of /56s at this level.

Absolutely.   But is that your only option?   E.g., if you were to just have a route at this level for every customer that had been renumbered and still needed their old prefix to be routed, would that tend to be hundreds of thousands?   If so, isn’t that evidence that there’s something wrong with your setup?

---

From: Ted Lemon <[email-masked]>
Date: Wed, 13 Nov 2019 10:20:25 -0500

On Nov 13, 2019, at 5:50 AM, Richard Patterson <[email-masked]> wrote:
> A different BNG vendor of ours however, doesn't have this configurable hold time concept; once the Valid LIfetime expires, the MAC+DUID binding to the PD is cleared.
> They do have setting to retain the MAC+DUID->PD lease binding indefinitely, however they didn't satisfactorily answer my question about cleanup if these bindings are exhausted (CPEs that use dynamic DUID values perhaps?  Even though even the DUID-LLT method is *supposed* to be stable).

FWIW, I don’t think there’s anything wrong with this.   The network is not obliged to remember that you had a prefix after the valid lifetime of the prefix (the promise you made) has expired.   I know that you’ve heard a few people here say that you should keep this prefix for the customer pretty much forever, but I don’t think that’s the consensus.   If you keep it until the valid lifetime has expired, I think you have kept your promise.

---

From: Tassos Chatzithomaoglou <[email-masked]>
Date: Thu, 14 Nov 2019 01:41:16 +0200

Ted Lemon wrote on 13/11/19 17:18:
> On Nov 13, 2019, at 2:32 AM, Tassos Chatzithomaoglou <[email-masked] <mailto:[email-masked]>> wrote:
I can see a disadvantage (it's a customer choice after all), but not a punishment. As i said in my previous emails, we haven't got any such complaints. So, either the customers don't care/understand, or the customers don't experience such issues (invalidating the previous prefix at next reconnect helps).
> Absolutely.   But is that your only option?   E.g., if you were to just have a route at this level for every customer that had been renumbered and still needed their old prefix to be routed, would…

We do have this for static customers who are a very small percentage. I guess for dynamic customers it wouldn't be hundreds of thousands in time t0, but i would see it easily increasing to tens of thousands in time t1 (after few weeks). Not to mention all the routing updates which would move deeper into the core of the network. That's because our N+1 BRAS clustering scheme has been designed with a simple concept in mind: provide the highest possible availability using the lowest possible budget. By increasing the budget, we can have more M+1 (M<N) BRAS clusters, so we can decrease the endpoints for a specific subscriber, until we reach the 1+1 model (active/hot-standby) where it would be silly not to have stable prefixes per subscriber.

--
Tassos

---

From: Ted Lemon <[email-masked]>
Date: Wed, 13 Nov 2019 19:02:17 -0500

Your lifetimes should not be so long. Graceful renumbering means advertising two prefixes, one deprecated. You keep the route for the deprecated prefix up until the valid lifetime expires. Then you eliminate it. The number of routes should not grow over time. 

> On Nov 13, 2019, at 18:41, Tassos Chatzithomaoglou <[email-masked]> wrote:
> 
>

---

From: Ole Troan <[email-masked]>
Date: Fri, 15 Nov 2019 00:42:59 +0800

> Your lifetimes should not be so long. Graceful renumbering means advertising two prefixes, one deprecated. You keep the route for the deprecated prefix up until the valid lifetime expires. Then you…

Actually that's not quite what graceful renumbering means, since it wouldn't be so graceful if you did it that way.
Basically it's two valid prefixes overlapping. Where you ensure no new sessions use it by letting preferred lifetime go to zero, and ensure that the deprecated timers doesn't expire before after the longest expected session lifetime has passed.

Cheers,
Ole