Subject: Re: Disabling temporary addresses by default?

From: Gyan Mishra <[email-masked]>
Date: Sat, 1 Feb 2020 16:24:09 -0500

On Wed, Jan 29, 2020 at 6:13 AM Lorenzo Colitti <[email-masked]> wrote:

> On Wed, Jan 29, 2020 at 2:46 AM Gyan Mishra <[email-masked]> wrote:
>
>>    The main reason this topic comes has up is due to possible impact of
>> usage of the temporary address when it gets deprecated with long lived
>> session.  That’s the crux of why this topic is critical and has severe
>> operational impact. When the address changes for long lived connections
>> from the deprecated temporary address to the new preferred address, the
>> session would terminate and have to re-establish, which is impacts the user.
>>
>
> If there is a permanent address as well, then an application that is not
> able to deal with session resets (like SSH) can use
> the IPV6_PREFER_SRC_PUBLIC socket options to use the permanent address
> instead.
>

    See section 5.  A lot more needs to be done as far as API development -
https://tools.ietf.org/html/draft-gont-6man-address-usage-recommendations-04#section-5

>
>> This would allow us to maintain privacy extension temporary address
>> enabled by default change to benefit privacy advocates and also eliminate
>> impact for enterprise users where availability and stability is utmost
>> importance.
>>
>
> But don't enterprises care about not leaking the habits of their
> employees? If I was on an enterprise security team, I would be pretty
> concerned about the levels of leakage and cross-site tracking that are
> enabled by never changing the IPv6 addresses of a host. This doesn't happen
> in IPv4 because we use NAT, but it will definitely happen in IPv6.
>

   From an enterprise IT perspective network access is all monitored at the
application layer with content filtering blocking URI access.  So the
anonymity or privacy concerns for an individual at home connected to the
internet at home is an IETF objective to maintain per RFC 7258 pervasive
monitoring.  That same objective does not exist for enterprise IT hosts.
In most if not all cases, the endpoint host is owned by the company and has
monitoring software loaded to maintain IT security, and ensue security
patches levels are up to date on endpoints, as well as web content
filtering and restricted access to install or remove software.  For BAU
IPv4 connections on a corporate intranet the IPv4 address is stable and not
changing.  With IPv4 Nat is only used for “hide Nat” to hide internal space
and use of web proxy.  For internal communications within a corporate
intranet Nat is used when address overlap exists. Even with IPv6 6to6 Nat
and is employed within the DMZ for “hide Nat” to eliminate attack vector
from any external entity extranet or internet.  As well as 6to6 web proxy
for internet web access.  From an IT operations perspective, as far as
IPv6, limiting the number of addresses on the endpoint for MTTR ( mean time
to recovery) for availability is the primary objective for an enterprise.
Stable random IPv6 address works best to meet the objective of an
enterprise.

> --

Gyan  Mishra

Network Engineering & Technology

Verizon

Silver Spring, MD 20904

Phone: 301 502-1347

Email: [email-masked]

---

From: Jared Mauch <[email-masked]>
Date: Sun, 2 Feb 2020 03:04:50 -0500

They are also useful and needed when debugging hashing or 802.3ad related issues. To debug the flow hash you often need stable addresses which are not easily changed or provided by the average user who just expects the technology to work. 

Sent from my iCar

> On Feb 1, 2020, at 4:24 PM, Gyan Mishra <[email-masked]> wrote:
> 
> Stable random IPv6 address works best to meet the objective of an enterprise.

---

From: Mark Smith <[email-masked]>
Date: Sun, 2 Feb 2020 19:53:18 +1100

On Sun, 2 Feb 2020, 19:05 Jared Mauch, <[email-masked]> wrote:

> They are also useful and needed when debugging hashing or 802.3ad related
> issues. To debug the flow hash you often need stable addresses which are
> not easily changed

Can you explain how you do this troubleshooting? Do you spoof end users
addresses?

or provided by the average user who just expects the technology to work.

Temporary addresses have been the default on Apple OS X since Lion in 2011
and Windows Vista in 2007. That doesn't seem to have prevented IPv6 working
for the average user.

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 2 Feb 2020 07:30:01 -0500

> Can you explain how you do this troubleshooting? Do you spoof end users

  802.1ad is a QinQ  technology used by providers for backbone bridges
Ethernet relay services instances.  Inner tag is provider label and outer
tag is customer label.  The flow hash entropy is based on
source/destination hash, however if the address is not stable and changing,
that changes the hash entropy for load balancing flows.  That throws off
the lb entropy and now you don’t get close to 50/50 lb entropy with 2 links
in an Ethernet bundle. There are many instances when bundled ethernet
aggregation or MLAG is employed where a Src/dest hash is used to create
entropy for load balancing of flows ; as well as in MPLS L2 vpn  and NG L2
VPN ethernet service instance e-line, e-lan, e-tree where a fat pseudo
wire” is deployed requiring stable IP for lb entropy to work properly.

Another instance where stable IP is necessary is related along those same
lines of thought to BGP IGP ISIS or OSPF ECMP flow based load balancing
hash hash where flows lb entropy is created to get the even load balancing
over multiple parallel L3 links in a enterprise or provider operators
network.  When the address keeps changing that changes the load balancing
entropy for the hash.  So now instead of the flow going over L3 link A it
is now going over link B.

Gyan  Mishra

Verizon

Silver Spring, MD 20904

Phone: 301 502-1347

---

From: Jared Mauch <[email-masked]>
Date: Sun, 2 Feb 2020 08:16:31 -0500

> On Feb 2, 2020, at 3:53 AM, Mark Smith <[email-masked]> wrote:
> 
> 
> 
> 
>> They are also useful and needed when debugging hashing or 802.3ad related issues. To debug the flow hash you often need stable addresses which are not easily changed
> 
> 
> Can you explain how you do this troubleshooting? Do you spoof end users addresses?
> 

No you try to find a combination of source ip, dest ip, protocol and source plus dest ports that aren't working and debug from there. 

The routers/hardware compute a hash from these tuples plus some initial entropy to assign traffic to a specific link member. If you have 4 or 6x 100g links (for example) you may see one link member misbehaving from a software bug or programming issue. You have to debug these layers to find the bad link. 

Some of the vendors make seeing their hash calculation easier than others. If we want to fix the bug we have to leave it broken or reproduce it for the developers. 

You may see this as 1:16 flows behaves poorly or is dropped. Sometimes you are more unlucky than others. 
> Temporary addresses have been the default on Apple OS X since Lion in 2011 and Windows Vista in 2007. That doesn't seem to have prevented IPv6 working for the average user.

HE may mask the issue so the end user isn't seeing issues but doesn't stop things from working. 

Day in the life of an operator perspective on making things too hard to debug automatically. We try but sometimes it takes that user report. 

This is aside from my concerns about the hyper privacy people blowing out ND. 

If you are spending all your time in software/centralized lookup platforms you are missing out on the joys of debugging networks based on ingress interface and these hash calculators!

---

From: Jared Mauch <[email-masked]>
Date: Sun, 2 Feb 2020 08:17:57 -0500

I said 802.3ad not 802.1Q* of which these variables can also be used in hash calculations. 

Sent from my iCar

> On Feb 2, 2020, at 7:30 AM, Gyan Mishra <[email-masked]> wrote:
> 
> 802.1ad is a QinQ  technology used by providers for backbone bridges Ethernet relay services instances.

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 2 Feb 2020 11:11:18 -0500

On Sun, Feb 2, 2020 at 8:16 AM Jared Mauch <[email-masked]> wrote:

> No you try to find a combination of source ip, dest ip, protocol and
> source plus dest ports that aren't working and debug from there.
>
> The routers/hardware compute a hash from these tuples plus some initial
> entropy to assign traffic to a specific link member. If you have 4 or 6x
> 100g links (for example) you may see one link member misbehaving from a
> software bug or programming issue. You have to debug these layers to find
> the bad link.
>
> Some of the vendors make seeing their hash calculation easier than others.
> If we want to fix the bug we have to leave it broken or reproduce it for
> the developers.
>

   Gyan> The L2 or L3 hash algorithm in generating the hash is vendor
proprietary usually an XOR of the tuple.  For example, Cisco lb hash for
all of their IOS flavors is generated using source/destination by default,
however you can add port to the lb hash alg making it a tuple.

   The lb hash alg  feature is pretty basic and is standard operation for
most all vendors which have a  CLI show command to determine what member
ink of L2 bundle the src/dest flow is taking.  I am not aware of any Cisco
bugs.

    Which vendor are you referring to regarding software bug or programming
issue?

> Day in the life of an operator perspective on making things too hard to
> debug automatically. We try but sometimes it takes that user report.
>
> If you are spending all your time in software/centralized lookup platforms
> you are missing out on the joys of debugging networks based on ingress
> interface and these hash calculators
>

> Gyan> Most all newer platforms across all vendors are hardware based and
> of course the larger platforms have distributed LC FIB versus centralized
> processor.
>
>
>
>
>> Sent from my iCar
>>

-- 

Gyan  Mishra

Verizon

Silver Spring, MD 20904

Phone: 301 502-1347

---

From: Jared Mauch <[email-masked]>
Date: Sun, 2 Feb 2020 11:33:09 -0500

I've experienced it on most, cisco, juniper and the BCM based silicon as well. 

Finding which port or LC is having issues without just playing a search method of taking things out of service is always painful. 

Hopefully there is a syslog or hardware counter that tells us what's happening but that's not always the case. 

At $dayjob we have constant measurements to detect some of this but it doesn't make it much easier for the humans. 

Sent from my iCar

> On Feb 2, 2020, at 11:12 AM, Gyan Mishra <[email-masked]> wrote:
> 
> Which vendor are you referring to regarding software bug or programming issue?

---

From: Michael Richardson <[email-masked]>
Date: Sun, 02 Feb 2020 17:40:15 -0500

> Finding which port or LC is having issues without just playing a search
    > method of taking things out of service is always painful.

    > Hopefully there is a syslog or hardware counter that tells us what's
    > happening but that's not always the case.

I have been a user behind such ECMP (in v4-only era) systems where the flow
cache expiry has been "tuned" for port-80 HTTP/1.0 traffic, and things like an
SSH login that lasts more than ten minutes just stops getting forwarded.  
You don't want to know what happens to an rsync or Windows-Update download.

It's pretty much impossible to report such bugs to the ISP, because they
simply haven't enough clues to understand the problem.  I could, at least,
explain it to local IT, and one of the results was that we changed ISPs,
because it affected our ability to support customers.
(Back then 10Mb/s links were $$$/month).  

Temporary addresses aren't going to change in the order of minutes, nor are
OS stacks going to pick them in some round-robin situation.

So, I agree that the debug problem is hard, but I don't think that temporary
addresses will make it worse.
The hash is going to depend upon the port number too, right? And that changes
for every connection anyway.

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Lorenzo Colitti <[email-masked]>
Date: Mon, 3 Feb 2020 09:11:08 +0900

On Fri, Jan 31, 2020 at 1:52 AM Gyan Mishra <[email-masked]> wrote:

> Lorenzo Colitti wrote on 29/01/2020 11:13:
>> employees?
>> This is only a problem in SLAAC environments where privacy addresses are
>> turned off.  In DHCP environments, the network operator can change
>> endpoint addresses by policy if this is what they want to do.
>>
>
>    Gyan> Enterprises fall into a totally different camp altogether as far
> as “employee” privacy as that is non existent as all traffic is monitored
> to be business related traffic to the extent of web content blocking which
> most all companies implement.   The use of temporary addresses does not
> make sense for enterprises to employ as operations stability and 99.99999%
> available and MTTR( mean time to recovery) is of utmost importance.  By
> disabling the temporary address - simplified the enterprise network
> troubleshooting tremendously as only a single “stable” random address
> exists at all times.  As far as temporary address and “long lived”
> connections for enterprise business critical applications that can all be
> solved by disabling the temporary address..
>

I'm not talking about surveillance by the company. I'm talking about
surveillance externally. Think: it's not that hard to an employee's IP
address (e.g., send them an image in an email, call them using a p2p
application like skype, etc. etc.) If that IP address lives forever then
anyone who has that IP address could see if that employee went to their
website, would know what their browsing habits are, could serve them
tailored content, could attempt to mount attacks on that IP address, etc.
etc. If I were a corp IT admin I wouldn't want to be in a situation where
someone can publish all the IP addresses permanently used by my employees,
including the CEO, the CIO, all the VPs, etc.

---

From: Erik Kline <[email-masked]>
Date: Mon, 3 Feb 2020 14:42:17 -0800

> and Windows Vista in 2007. That doesn't seem to have prevented IPv6 working
Android's been doing this for a while too, and has no Happy Eyeballs to
mask any problems it might have caused.  I'm inclined to think there is no
real issue with tempaddrs being enabled for general purpose nodes.