Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing

From: Fernando Gont <[email-masked]>
Date: Thu, 3 Dec 2020 05:02:33 -0300

Hi, Tom,

On 29/11/20 13:25, Tom Herbert wrote:
>> Modulo the comments I've already made, I'm afraid we might be in the
>> path of making the FL unusable...
>>
> 
> Fernanda,,
> 
> I really don't understand how this makes the flow label "unusable", to
> the contrary this is a useful case with good effect and justifies the
> work in the host to set the field. In any case, it would be great if
> you could provide the draft with the normative requirements that would
> make the flow label "usable".

Have the FL identify a flow. packets RTO'ing does not imply that the 
retransmissions belong to a different flow.

FL changing for established sessions will break load-balancers. OTOH, 
one would still need to anylize what would happen if a SYN RTO-es, gets 
retransmitted, but both the original and the retransmitted SYN is delivered.

> Please include requirements for both hosts and network nodes. If there
> is going to the be a requirement that hosts MUST make a 1:1 mapping to
> a transport connection and the flow label MUST be consistent for the
> life of connection, then I would expect there to also be a requirement
> that the flow label MUST be immutable by all devices in the path-- in
> particular the requirement of RFC6437 would need to be updated: "A
> forwarding node MUST either leave a non-zero flow label value
> unchanged or change it only for compelling operational security
> reasons..."-- if hosts are not allowed to change a field they are
> primarily responsible for setting, then the network should not be
> allowed to modify it either.

I don't recall _(of the top of my head) why we ended up making the field 
mutable ("SHOULD NOT  be modified" rather than "MUST NOT.."). IIRC, it 
had to do with allowing middle-boxes to mitigate FL-based covert 
channels.  If that's the case (Brian CC'ed) then, in retrospective, that 
seems to have been a bad idea.  -- I was part of the discussion, so I 
take my share of blame. :-)

> Additionally, if the underlying design principle is that flow
> identification in a packet is immutable and persistent, then I'd like
> to see the discussion on how that rectifies with NAT where
> intermediate nodes routinely change IP addresses and transport port
> numbers without the host endpoints' knowledge-- unlike the flow label,
> port numbers are formally part of the transport connection
> identification, so changing them in flight is far more invasive and
> potentially harmful than changing the flow label (we've seen far more
> problems caused by NAT than flow label due to NAT state evictions,
> packets of a flow being re-routed to not hit the same NAT device,
> etc.).

You seem to be referring to IPv4 NAT, since IPv6 NPT does not modify the 
ports.   That said, even if IPv6 NAT did, then as long as the FL is 
consistent from the outside, that's fine. (i.e., the FL remains constant 
from both the "external realm" pov, and also from the "internal realm" pov).

Thanks!

Regards,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Erik Kline <[email-masked]>
Date: Thu, 3 Dec 2020 07:43:48 -0800

> take my share of blame. :-)

My understanding/recollection is that since there were never any
restrictions on modification of the flow label from the very beginning
it wasn't super meaningful to try to close the barn door after all
these years.

---

From: Fernando Gont <[email-masked]>
Date: Thu, 3 Dec 2020 13:08:03 -0300

> it wasn't super meaningful to try to close the barn door after all

Well, yeah, but there was also no use of the FL, either ;-) . And for 
load-balancing among a group of servers, you need the FL to be stable 
for the life of the flow, or otherwise packet may end up ebig delivered 
to different servers.

Otherwise, the FL is likely to be inored, and we'll continue figuring 
out what to use those 20 bits for...

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Thu, 3 Dec 2020 10:30:45 -0700

> it wasn't super meaningful to try to close the barn door after all

Erik,

There are two apropos statements in RFC6437:

"There is no way to verify whether a flow label has been modified en
route or whether it belongs to a uniform distribution.  Therefore, no
Internet-wide mechanism can depend mathematically on unmodified and
uniformly distributed flow labels; they have a "best effort" quality."

"From an upper-layer viewpoint, a flow could consist of all packets in
one direction of a specific transport connection or media stream.
However, a flow is not necessarily 1:1 mapped to a transport
connection."

So building a load balancer, or any other intermediate network device,
that *depends* on the flow label to be consistent for the life of a
transport connection and 1:1 mapped to a transport connection is
contrary to this guidance. The ship has already sailed on this in
deployment, and even if the protocol requirements were to be
sufficiently tightened that would take years to reach full effect and
that would preclude other use cases where modifying the flow label is
useful. If a stable 1:1 mapped identifier for transport ports is
needed in the network layer, it might make sense to create a new HBH
option for that. Personally, if I were building a load balancer I'd
use neither the flow label nor transport ports for entropy, I'd look
at using the destination address since 128 bits allows plenty of bits
of entropy, it's consistent per transport flow and always set in an
IPv6 packet.

Tom

Tom

---

From: Fernando Gont <[email-masked]>
Date: Thu, 3 Dec 2020 16:40:21 -0300

On 3/12/20 14:30, Tom Herbert wrote:
[....]
> contrary to this guidance. 

We don't need 1:1. We just need 1:N where N is how flows would be 
aggregated/grouped for the purpose of load-sharing.

> The ship has already sailed on this in
> option for that.

The above said, I would probably agree that if havinga non-stable FL 
breaks something, whatever it breaks is probably already broken, since 
there's not even a checksum to try to detect whether the IPv6 header has 
been corrupted.

> Personally, if I were building a load balancer I'd
Problem is that if you have two systems that need a lot of banswitdth 
(e.g., they are transferring file via several TCP connections)< you 
wouldn't be able to load-share them -- since the result of the hash 
would be the sme for all TCP connections....

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Thu, 3 Dec 2020 13:06:43 -0700

> breaks something, whatever it breaks is probably already broken, since

Fernando,

Not to belabor the point, but stable flow labels were never
guaranteed, so if some node has a problem with non-stable flow labels
then they are themselves broken because they are making an incorrect
assumption.

> there's not even a checksum to try to detect whether the IPv6 header has
Neither is there checksum that covers the UDP port numbers when UDP
checksum is zero. A "stable" identifier that the network can use to
conclusively identify transport flows does not generally exist.

> would be the sme for all TCP connections....
Use a different IP address per connection. This idea has already been
brought up in the context of security since it could make it harder
for malicious network observers to correlate different flows as having
the same source, and the same principle could be applied to load
balancers where each of the backends is assigned a block of IP
addresses (like a /64). After that, it's a matter of getting clients
to use some random address from the load balancer blocks (instead of
always using one). IIRC, Brian Carpenter once suggested that we really
didn't need port numbers any more since they could be part of the
address-- for load balancers there might be some merit to that.

 Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 4 Dec 2020 09:17:52 +1300

> uniformly distributed flow labels; they have a "best effort" quality."

Yes, a MUST NOT is futile because it cannot be enforced.

Also, the firewall folk  seemed very insistent that they would change
the flow label (or zero it) because paranoia. So specifying that if they
changed it, they should do it consistently, seemed better than a MUST NOT
which they would definitely ignore.

> of entropy, it's consistent per transport flow and always set in an

But that doesn't help with server load balancing. For ECMP/LAG in a
transit network, the DA alone doesn't help either, if a particular 
destination is prone to very heavy flows for some reason. And of course
the ports don't help if they're encrypted.

There's no simple answer here.

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Thu, 3 Dec 2020 13:38:39 -0700

On Thu, Dec 3, 2020 at 1:17 PM Brian E Carpenter
<[email-masked]> wrote:
> destination is prone to very heavy flows for some reason.

Brian,

Specifically, that is a problem for a host that uses a _single_ IPv6
address for all its connections. What I am suggesting is to give hosts
many addresses, which we already do anyway like when mobile devices
and data center servers are assigned /64s. If a host then randomly
chooses an address from that block for each connection it creates
thenl the address 2-tuple contains all the entropy needed for purposes
of load balancing, ECMP, etc. is contained in the source address.
Addresses are guaranteed to be present and plaintext in each packet so
no DPI required, they're as stable as any flow identifying information
in a packet can be and they offer much more entropy than sixteen bits
of port number of twenty bit flow label.

Since the block address assignments or already happening, this seems
more to be a host implementation matter where we perform address
selection for a new connection by random selection in the block. It's
on my wish list of things to implement :-)

Tom

> And of course

---

From: Fernando Gont <[email-masked]>
Date: Thu, 3 Dec 2020 19:41:33 -0300

Tom,

[....]
> address for all its connections. 

That that's the vast majority of current hosts - if not all.

> What I am suggesting is to give hosts

Agreed. Although that move is not-trivial: e.g. if you move from shared 
link to /64 per hosts, then stuff like mDNS breaks.

There's also an API to be specifiied.

P.S.: And, in the process, we have kind of burned 64-bits of the 128-bit 
IPv6 addresses, too...

Thanks,