Subject: [tcpm] Flow Label Load Balancing

From: Alexander Azimov <[email-masked]>
Date: Thu, 19 Nov 2020 13:48:06 +0300

Dear colleagues,

I have added in the cc both v6op and tcpm for a reason and let me explain
why.

It's clear that we are moving forward with load balancing that uses flow
label (FL). And the pressure will increase with SRv6 adoption. But at the
moment wide adoption of FL-based load-balancing may create significant
issues for TCP Anycast services.

RFC6437 suggests putting hash from 5-tuple into FL value. And as far as I
know, there is no document that updates this behavior. This description is
perfectly fine, but what is implemented in the Linux kernel is different:
FL is carrying hash from 5-tuple with an additional seed, and this seed is
randomly changed after each RTO/SYN_RTO event. Here are related patches:

   -
   https://lore.kernel.org/netdev/[email-masked]/
   -
   https://lore.kernel.org/netdev/[email-masked]/
   - https://lore.kernel.org/netdev/[email-masked]/

This is a great thing by the way because in the data center environment with
multiple equal paths it gives a way to have pseudo-multipath TCP which
jumps between paths in case of an outage. There might be interest to write
down an informational document for this.

But the problem happens when TCP flows start jumping between anycast
points. If anycast service provides connection tracking (L7/TCP proxy,
stateful L4 balancer) the 'jump' may redirect traffic to another point, and
with the adoption of FL load balancing by the transit providers - to
another location, where no TCP state is available. In other words, an
established TCP session breaks after RTO event. And it's not just a theory
- we already faced this kind of issue in our network, and it is really hard
to debug.

I wonder what you think is a proper solution:

   - Making FL related RTO change as knob instead of default behavior;
   - Adding negotiation behavior in TCP;
   - Something else?

I'm looking forward to your advice. If there is a document that describes
the above problem - please give me a reference.

-- 
Best regards,
Alexander Azimov

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 Nov 2020 08:36:29 -0700

On Thu, Nov 19, 2020 at 3:49 AM Alexander Azimov <[email-masked]> wrote:
>
> Dear colleagues,
>
> I have added in the cc both v6op and tcpm for a reason and let me explain why.
>
> It's clear that we are moving forward with load balancing that uses flow label (FL). And the pressure will increase with SRv6 adoption. But at the moment wide adoption of FL-based load-balancing may create significant issues for TCP Anycast services.
>
> RFC6437 suggests putting hash from 5-tuple into FL value. And as far as I know, there is no document that updates this behavior. This description is perfectly fine, but what is implemented in the Linux kernel is different: FL is carrying hash from 5-tuple with an additional seed, and this seed is randomly changed after each RTO/SYN_RTO event. Here are related patches:
>
> https://lore.kernel.org/netdev/[email-masked]/
>
Alexander,

The initial intent of the Linux flow label code was that the hash was
persistent for the lifetime of the connection; in fact we fixed a
nasty bug where the hash would change when a connection moved to TW
state causing some firewalls not to see the final ACK so they can
remove state. The feature where the flow hash is recalculated for a
failing connection should be optional and not the default behavior. If
this isn't what the implementation is doing I can take it up on Linux
netdev to fix it.

Have you been able to show this behavior is actually happening on a
running system?

Tom

>
> This is a great thing by the way because in the data center environment with multiple equal paths it gives a way to have pseudo-multipath TCP which jumps between paths in case of an outage. There might be interest to write down an informational document for this.
>
> But the problem happens when TCP flows start jumping between anycast points. If anycast service provides connection tracking (L7/TCP proxy, stateful L4 balancer) the 'jump' may redirect traffic to another point, and with the adoption of FL load balancing by the transit providers - to another location, where no TCP state is available. In other words, an established TCP session breaks after RTO event. And it's not just a theory - we already faced this kind of issue in our network, and it is really hard to debug.
>
> Making FL related RTO change as knob instead of default behavior;
> Adding negotiation behavior in TCP;
> Something else?
>
> I'm looking forward to your advice. If there is a document that describes the above problem - please give me a reference.
>
> --
> Best regards,
> Alexander Azimov
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Alexander Azimov <[email-masked]>
Date: Fri, 20 Nov 2020 12:05:46 +0300

Tom,

As far as I know, the auto_flowlabels default is 1, which stands for:

> 1: automatic flow labels are enabled by default, they can be disabled on a
> per socket basis using the IPV6_AUTOFLOWLABEL socket option

And you get it all-in-one: FL changes after both SYN_RTO and RTO events.
The change after SYN_RTO looks safe and can bring benefit for the end-user
experience, while the FL change after the RTO outside of a controlled
environment is causing a problem.

And yes, we were able to detect it on a working system. One of the
developers contacted NOC team that he has constant connection failures
while pushing a significant amount of data, and it began when he started
working from home. It proved out that there was RTO happening during the
lifetime of the TCP session, the session jumped to another stateful L4
balancer and, as a result, the session got a timeout. At the moment we had
to disable FL load balancing at our border routers and DCI - this solved
the issue at the moment.

If it is possible to remove from the default behavior change of flow label
after RTO event, while keeping it for SYN_RTO would be great.

чт, 19 нояб. 2020 г. в 18:36, Tom Herbert <[email-masked]>:

> > RFC6437 suggests putting hash from 5-tuple into FL value. And as far as
> I know, there is no document that updates this behavior. This description
> is perfectly fine, but what is implemented in the Linux kernel is
> different: FL is carrying hash from 5-tuple with an additional seed, and
> this seed is randomly changed after each RTO/SYN_RTO event. Here are
> related patches:
> >
> >
> > This is a great thing by the way because in the data center environment
> with multiple equal paths it gives a way to have pseudo-multipath TCP which
> > I'm looking forward to your advice. If there is a document that
> describes the above problem - please give me a reference.
> >
> > --
> > Best regards,
> > Alexander Azimov

-- 
Best regards,
Alexander Azimov

---

From: Fernando Gont <[email-masked]>
Date: Tue, 24 Nov 2020 16:05:50 -0300

On 19/11/20 07:48, Alexander Azimov wrote:
> label (FL). And the pressure will increase with SRv6 adoption. But at 
> the moment wide adoption of FL-based load-balancing may create 
> significant issues for TCP Anycast services.
> 
> I know, there is no document that updates this behavior. This 
> description is perfectly fine, but what is implemented in the Linux 
> kernel is different: FL is carrying hash from 5-tuple with an additional 
> seed, and this seed is randomly changed after each RTO/SYN_RTO event. 

Changing the FL upon RTO is a bug.

I guess/assume that when you say SYN-RTO, you really mean "user 
timeout", rather than RTO. If you don't, then that's also a bug.
If you do, I fail to understand what's the reason for wanting the FL to 
change in that case, because as a result of port randomization, it 0s 
unlikely that the same four-tuple is employed for the next connection retry.

> Here are related patches:
> 
>   * https://lore.kernel.org/netdev/[email-masked]/
>   * https://lore.kernel.org/netdev/[email-masked]/
>   * https://lore.kernel.org/netdev/[email-masked]/
> 
> 
> This is a great thing by the way because in the data center 
> environment with multiple equal paths it gives a way to have 
> pseudo-multipath TCP which jumps between paths in case of an outage. 
> There might be interest to writedown an informational document for this.

That's a bad idea, since specs-wise the Flow-Label is not guaranteed to 
remain unchanged from source to destination. If you want to ahve 
multiple paths, then you should implement that in routing.

>   * Making FL related RTO change as knob instead of default behavior;
>   * Adding negotiation behavior in TCP;
>   * Something else?

Just make the FL a function of the connection "identifier". And keeo it 
constant for the lifetime of that conenction.

Thanks,
-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Alexander Azimov <[email-masked]>
Date: Tue, 24 Nov 2020 22:28:21 +0300

Hi Fernando,

Stating that FL change during TCP session lifetime is a bug - is a bit
harsh.

It is a fantastic idea to change the FL value if RTO or SYN_RTO happens
in a controlled environment.
These are very specific TCP timeouts, that provide enough guarantee that
there will be no out-of-order packets, though your packets will reach
the destination even in case of an outage inside your network. Zero
influence on your services in case of the network outage - doesn't sound
like a bug for me.

The problem is with the current Linux (though I haven't checked other OSes)
defaults. Specifically, with the default behavior after RTO event.

вт, 24 нояб. 2020 г. в 22:06, Fernando Gont <[email-masked]>:

> >   *
> >   *
> >   *

-- 
Best regards,
Alexander Azimov

---

From: Tom Herbert <[email-masked]>
Date: Tue, 24 Nov 2020 12:43:52 -0700

> constant for the lifetime of that conenction.

Fernando,

Modulating the flow label is a means to affect the routing of packets
through the network that uses flow labels as input to the ECMP hash.
The basic idea is that the flow label associated with a connection is
randomly changed when the stack observes that the connection is
failing (e.g. and an RTO). There is nothing in the specs that prevents
this since the source is at liberty to set the flow label as it sees
fit. The feature is useful in large datacenter networks, like
pparently Facebook where the patches originate, since information
discerned by TCP can opportunistically be applied to route selection.
The practical issue is that there are stateful devices like firewalls
that require consistent routing in the network in which case changing
the flow label can confuse them. As I mentioned, the original intent
was that the flow label randomization feature should be opt-in instead
of on by default.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 25 Nov 2020 08:45:34 +1300

On 25-Nov-20 08:28, Alexander Azimov wrote:
> Hi Fernando,
> 
> Stating that FL change during TCP session lifetime is a bug - is a bit harsh.

It's a bug. It definitely breaks server load balancing, which in practice is more important than en route load balancing, because it will probably switch your session to a different server. (OK, I know that there are mechanisms sometimes for rescuing a session that has unexpectedly moved from one server to another, but in general it's a fail.)

I assume you have read https://tools.ietf.org/html/rfc7098

Regards
    Brian

> 
> It is a fantastic idea to change the FL value if RTO or SYN_RTO happens in a controlled environment. 
> These are very specific TCP timeouts, that provide enough guarantee that there will be no out-of-order packets, though your packets will reach the destination even in case of an outage inside your network. Zero influence on your services in case of the network outage - doesn't sound like a bug for me.
> 
> The problem is with the current Linux (though I haven't checked other OSes) defaults. Specifically, with the default behavior after RTO event.
> 
> вт, 24 нояб. 2020 г. в 22:06, Fernando Gont <[email-masked] <mailto:[email-masked]>>:
> 
>     e-mail: [email-masked] <mailto:[email-masked]> || [email-masked] <mailto:[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 24 Nov 2020 12:59:46 -0700

> It's a bug. It definitely breaks server load balancing, which in practice is more important than en route load balancing, because it will probably switch your session to a different server. (OK, I…
Brian,

IMO it's not that this breaks server load balancing, it's really that
server load balancing and other stateful mechanisms break multipath,
multihoming, and adaptive routing! In any case, I would agree that in
the current Internet it is pragmatic in the current Internet to aim
for consistent routing as the default.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Wed, 25 Nov 2020 04:04:10 -0300

> Stating that FL change during TCP session lifetime is a bug - is a bit 

Apologies if it came across like that -- the intent was to be blunt, 
rather than harsh.

> in a controlled environment.

As noted in my previous email, I'm not sure what you mean by RTO -- but 
if you really mean "retransmission timeout", doing that fails the 
definition of FL by RFC6437 and others. -- a retransmitted packet 
belongs to the ame flow, not to a different one.

That aside, please define "controlled environment". :-)

> there will be no out-of-order packets,

Strictly speaking, that's incorrect:: the RTO is an estimation, and may 
be fired incorrectly (although the whole point of measuring the RTT anc 
computing the RTO is for that case to be rather rare).

>  though your packets will reach 

It seems to me that this sort of behavior is the reason for which folks 
actually disable the use of the FL. (i.e., think what happens for load 
balancers).

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Wed, 25 Nov 2020 04:13:20 -0300

Hi, Tom,

On 24/11/20 16:43, Tom Herbert wrote:
[....]
> through the network that uses flow labels as input to the ECMP hash.

What's the point?

1) You cannot tell *if* the FL is being used.

2) Changing the FL does not necessarily mean that packets will employ a 
different link.

3) If the network is failing, shouldn't you handle this via routing?

> fit.

The FL is expected to remain constant for the life of a flow. A 
retransmitted packet is part of the same flow as the 
originally-transmitted packet. So this seems to be contradicting the 
very specification of the FL.

For instance, If a RTO for a flow causes the FL to change, then one may 
possibly argue that the FL is not naming/labeling what is said/expected 
to be anming/labeling.

> The feature is useful in large datacenter networks, like

So... where is the "source" of the packet that would be "modulating" the FL?

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Wed, 25 Nov 2020 12:35:07 -0700

Hi Fernando, comments in line...

> 1) You cannot tell *if* the FL is being used.
Generally true, but in a limited domain this information could be
discerned. I'd note that it's also generally true that we don't know
if there is a load balancer or stateful firewall in the path that
requires consistent routing, but in a limited domain we could know
that also.

> 2) Changing the FL does not necessarily mean that packets will employ a

It's an opportunistic mechanism. If a connection is failing and we get
a better path that fixes it by simply changing the flow label then
what's the harm?

> 3) If the network is failing, shouldn't you handle this via routing?
Sure, but then that requires an out of band feedback loop from a TCP
implementation to the network infrastructure to indicate there is a
problem and then the network needs to respond. That's significant
infrastructure and higher reaction time than doing something in TCP
and IP. Think of modulating the flow label is an inexpensive form of
source routing within a limited domain that doesn't need any
infrastructure or heavyweight protocols or something like segment
routing.

> possibly argue that the FL is not naming/labeling what is said/expected

Specifically, RFC6437 states:

"It is therefore RECOMMENDED that source hosts support the flow label
by setting the flow label field for all packets of a given flow to the
same value chosen from an approximation to a discrete uniform
distribution."

So that is clearly a just recommendation, and not a requirement (and
definitely not a MUST). Furthermore, RFC6437 states:

"A forwarding node MUST either leave a non-zero flow label value
unchanged or change it only for compelling operational security
reasons as described in Section 6.1."

So there's no guarantee in the protocol specs that flow labels are
consistent for the life of the connection, which means that the
network cannot assume that and thus it would be incorrect if the
network tried to enforce flow label consistency as a protocol
requirement. As I said, it is prudent to try to be consistent with
flow labels and the default behavior in Linux should be changed,
however I do not believe there's a valid claim of non-conformance that
motivates removal of the feature that is already deployed.

Tom

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 25 Nov 2020 14:39:55 -0500

This kind of thing is why, as I understand it, MPTCP has discovery 
mechanisms ot know if both sides use it, and can select alternative 
addresses for communication.

Trying to guess flow labels that might avoid a problem because it might 
be an ECMP problem, is just flailing about.  Not a good design for 
operational protocols.

And in general, designing protocols around "I know exactly what is going 
on"  (the requirement for what you describe that goes well beyond just 
"limited domains") is also a recipe for failure.

The Flow Label RFCs are actually very explicit that a flow label is 
supposed to be stable for the life of the flow.  Otherwise, it isn't a 
flow label.

Yours,
Joel

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 26 Nov 2020 09:04:42 +1300

Hi Joel,

> supposed to be stable for the life of the flow.  Otherwise, it isn't a 

That's correct. However, the first paragraph of RFC6437 tells you that
it's the prerogative of the source of a flow to decide what is a flow.
It even says "a flow is not necessarily 1:1 mapped to a transport
connection." Therefore, Tom is correct that a source is conformant even
if it changes its mind halfway through a TCP connection. However, it's
only safe to do that if the source also knows that no harm will result,
which it cannot know in the general case. In a limited domain, the source
could know that there is no server load balancing in place, in which case
changing the label is not automatically damaging.

Thus, this is not something that should ever become a default behaviour,
and I think this thread began with a complaint that in some o/s release
it was the default.

Regards
   Brian

---

From: Tom Herbert <[email-masked]>
Date: Wed, 25 Nov 2020 13:05:00 -0700

Joel,

Is there an RFC that clearly and unambiguously states that a host MUST
use the same flow label for the lifetime _and_ clearly defines exactly
what a flow is with respect to such a requirement (for instance, how
would you define a flow and enforce such a requirement in UDP? IPsec?
other encapsulations?). If there is such a requirement then we'll
change the code to be conformant.

Tom

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 25 Nov 2020 16:27:48 -0500

No, as Brian says, there are escape clauses in the flow definitions.

But changing the flow label due to traffic problems does not correspond 
to packets being in actually different flows.
If one were using UDP, and mixing loss sensitive packets with loss 
insensitive packets for a special application, sure, one could use two 
flow labels.  But that is not what you are describing.

Yours,
Joel

---

From: Tom Herbert <[email-masked]>
Date: Wed, 25 Nov 2020 14:13:11 -0800

Joel, is there a normative definition of a flow?

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 26 Nov 2020 11:33:42 +1300

I'm not Joel, but I did once spend some time grepping RFCs to find out whether
"flow" or "microflow" was the preferred term. In RFC2474, which is normative,
we have:

   Microflow: a single instance of an application-to-application flow of
   packets which is identified by source address, destination address,
   protocol id, and source port, destination port (where applicable).

But in the flow label work we explicitly avoided being that precise, and
did not use the term "microflow". There might be some load balancing
scenarios where you want a broader definition, even including bidirectional
flows. There are expired drafts on that topic:
draft-tarreau-extend-flow-label-balancing
draft-wang-6man-flow-label-reflection

Regards
   Brian

On 26-Nov-20 11:13, Tom Herbert wrote:
> On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <[email-masked] <mailto:[email-masked]>> wrote:
>     >>
>     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <[email-masked] <mailto:[email-masked]>> wrote:
>     >>>>
>     >>>> Hi, Tom,
>     >>>>
>     >>> [email-masked] <mailto:[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 25 Nov 2020 17:15:49 -0700

Brian,

Random Packet Spraying
(https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.297.529&rep=rep1&type=pdf)
is an interesting idea where packets for a single connection are
purposely distributed across multiple paths for load distribution. Per
packet randomized flow labels with flow label aware ECMP makes this
quite easy to do without requiring any special support in switches
like you'd need with IPv4. I'm not necessarily advocating this, but it
does highlight one potential use case of having a flow label that
doesn;t have rigidly defined requirements on the host.

Tom

---

From: Alexander Azimov <[email-masked]>
Date: Thu, 26 Nov 2020 12:34:04 +0300

Dear colleagues,

We started discussing an incorrect default behavior, and Tom has already
confirmed that it will be fixed.

Later the thread turned into an argument if the flow label can be changed
during connection/flow lifetime according to current RFC documents, though
these documents can be updated. This looks a bit weird for me because I
always thought that it is the IETF community's responsibility to document
proper solutions. If something undocumented but worthy happens in the
industry - IETF should catch up. So, I would like to get back to the
discussion of reasons and their safety.

The general idea of changing the routing path upon network
outage/degradation looks obvious. Getting kind of source-based routing can
significantly reduce the reaction time and improve end-user experience. The
flow label has a perfect match here: transparent for the application, set
by the source, not part of 5-tuple, while it can be used in the load
balancing. IMO poorly documented. I would like to learn your feedback for
the next wording:

Let say that the flow label is a hash of values from the IP packet's
5-tuple and random number. Then

1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be
recalculated.
2. In the case of RTO timeout expiration in the established TCP session the
flow label MAY be recalculated. This setting MUST be switched off by
default.
3. Otherwise flow label SHOULD be preserved unchanged.

The first point stands for redirecting connection from the degraded path
before the connection is established, and it looks safe. The second one can
also improve performance if it is set on the server-side. Please comment if
you see security flaws in such a design.

чт, 26 нояб. 2020 г. в 03:16, Tom Herbert <[email-masked]>:

> (
> https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.297.529&rep=rep1&type=pdf
> )
> > >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <
> [email-masked] <mailto:[email-masked]>> wrote:
> > >     >>
> > >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <
> [email-masked] <mailto:[email-masked]>> wrote:
> > >     >>>>
> > >     >>>> Hi, Tom,
> > >     >>>>
> > >     >>>> e-mail: [email-masked] <mailto:[email-masked]>
> || [email-masked] <mailto:[email-masked]>

-- 
Best regards,
Alexander Azimov

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 26 Nov 2020 08:38:57 -0800

On Thu, Nov 26, 2020 at 1:34 AM Alexander Azimov <[email-masked]>
wrote:

> 2. In the case of RTO timeout expiration in the established TCP session
> the flow label MAY be recalculated. This setting MUST be switched off by
> default.

For completion, we might want to discuss reverse-path failure that
RTO-triggered rehash in (1)(2) alone can not repair the connection in
established or handshake phases.

https://patchwork.ozlabs.org/project/netdev/patch/[email-masked]/

I also agree with Tom that the general feature is very useful in keeping
TCP connected with partial route failures, but the default should be an
opt-in.

> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 27 Nov 2020 08:52:57 +1300

Alexander,

> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be recalculated.
...
> The first point stands for redirecting connection from the degraded path before the connection is established, and it looks safe.

I am convinced that it is often unsafe, because that depends on exactly how a server farm load balancer handles such cases. If it maintains the TCP session on the same actual server if there is SYN(-ACK) retransmission, the flow label must not be changed.
The benefit for possible broken routing is quite hypothetical, whereas the risk to
session persistence at the server is real. (Remember that the source cannot know by magic that there is server load balancing at the far end, but it's the common case today.)

> 2. In the case of RTO timeout expiration in the established TCP session the flow label MAY be recalculated. This setting MUST be switched off by default.

As noted before, this is only safe in a limited domain where it is known with certainty that server load balancing is not in use. I think it's therefore a corner case and specifically, the appropriate normative statement would be SHOULD NOT, which is defined very carefully in RFC2119 exactly for this sort of case.

But in fact I think we should not attempt to legislate on this point. The value is marginal at best. As others have said, this is not the way to tackle operational routing problems.

Regards
   Brian Carpenter

On 26-Nov-20 22:34, Alexander Azimov wrote:
> Dear colleagues,
> 
> We started discussing an incorrect default behavior, and Tom has already confirmed that it will be fixed.
> 
> Later the thread turned into an argument if the flow label can be changed during connection/flow lifetime according to current RFC documents, though these documents can be updated. This looks a bit weird for me because I always thought that it is the IETF community's responsibility to document proper solutions. If something undocumented but worthy happens in the industry - IETF should catch up. So, I would like to get back to the discussion of reasons and their safety.
> 
> The general idea of changing the routing path upon network outage/degradation looks obvious. Getting kind of source-based routing can significantly reduce the reaction time and improve end-user experience. The flow label has a perfect match here: transparent for the application, set by the source, not part of 5-tuple, while it can be used in the load balancing. IMO poorly documented. I would like to learn your feedback for the next wording:
> 
> Let say that the flow label is a hash of values from the IP packet's 5-tuple and random number. Then
> 
> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be recalculated.
> 2. In the case of RTO timeout expiration in the established TCP session the flow label MAY be recalculated. This setting MUST be switched off by default.
> The first point stands for redirecting connection from the degraded path before the connection is established, and it looks safe. The second one can also improve performance if it is set on the server-side. Please comment if you see security flaws in such a design.
> 
> чт, 26 нояб. 2020 г. в 03:16, Tom Herbert <[email-masked] <mailto:[email-masked]>>:
> 
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     >
>     > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     > >
>     > >     > Joel,
>     > >     >
>     > >     > Is there an RFC that clearly and unambiguously states that a host MUST
>     > >     > use the same flow label for the lifetime _and_ clearly defines exactly
>     > >     > what a flow is with respect to such a requirement (for instance, how
>     > >     > would you define a flow and enforce such a requirement in UDP? IPsec?
>     > >     > other encapsulations?). If there is such a requirement then we'll
>     > >     > change the code to be conformant.
>     > >     >
>     > >     > Tom
>     > >     >
>     > >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     > >     >>
>     > >     >> This kind of thing is why, as I understand it, MPTCP has discovery
>     > >     >> mechanisms ot know if both sides use it, and can select alternative
>     > >     >> addresses for communication.
>     > >     >>
>     > >     >> Trying to guess flow labels that might avoid a problem because it might
>     > >     >> be an ECMP problem, is just flailing about.  Not a good design for
>     > >     >> operational protocols.
>     > >     >>
>     > >     >> And in general, designing protocols around "I know exactly what is going
>     > >     >> on"  (the requirement for what you describe that goes well beyond just
>     > >     >> "limited domains") is also a recipe for failure.
>     > >     >>
>     > >     >> The Flow Label RFCs are actually very explicit that a flow label is
>     > >     >> supposed to be stable for the life of the flow.  Otherwise, it isn't a
>     > >     >> flow label.
>     > >     >>
>     > >     >> Yours,
>     > >     >> Joel
>     > >     >>
>     > >     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
>     > >     >>> Hi Fernando, comments in line...
>     > >     >>>
>     > >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     > >     >>>>
>     > >     >>>> Hi, Tom,
>     > >     >>>>
>     > >     >>>> On 24/11/20 16:43, Tom Herbert wrote:
>     > >     >>>> [....]
>     > >     >>>>> Modulating the flow label is a means to affect the routing of packets
>     > >     >>>>> through the network that uses flow labels as input to the ECMP hash.
>     > >     >>>>
>     > >     >>>> What's the point?
>     > >     >>>>
>     > >     >>>> 1) You cannot tell *if* the FL is being used.
>     > >     >>>>
>     > >     >>> Generally true, but in a limited domain this information could be
>     > >     >>> discerned. I'd note that it's also generally true that we don't know
>     > >     >>> if there is a load balancer or stateful firewall in the path that
>     > >     >>> requires consistent routing, but in a limited domain we could know
>     > >     >>> that also.
>     > >     >>>
>     > >     >>>> 2) Changing the FL does not necessarily mean that packets will employ a
>     > >     >>>> different link.
>     > >     >>>
>     > >     >>> It's an opportunistic mechanism. If a connection is failing and we get
>     > >     >>> a better path that fixes it by simply changing the flow label then
>     > >     >>> what's the harm?
>     > >     >>>
>     > >     >>>>
>     > >     >>>> 3) If the network is failing, shouldn't you handle this via routing?
>     > >     >>>>
>     > >     >>> Sure, but then that requires an out of band feedback loop from a TCP
>     > >     >>> implementation to the network infrastructure to indicate there is a
>     > >     >>> problem and then the network needs to respond. That's significant
>     > >     >>> infrastructure and higher reaction time than doing something in TCP
>     > >     >>> and IP. Think of modulating the flow label is an inexpensive form of
>     > >     >>> source routing within a limited domain that doesn't need any
>     > >     >>> infrastructure or heavyweight protocols or something like segment
>     > >     >>> routing.
>     > >     >>>
>     > >     >>>>
>     > >     >>>>
>     > >     >>>>> The basic idea is that the flow label associated with a connection is
>     > >     >>>>> randomly changed when the stack observes that the connection is
>     > >     >>>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
>     > >     >>>>> this since the source is at liberty to set the flow label as it sees
>     > >     >>>>> fit.
>     > >     >>>>
>     > >     >>>> The FL is expected to remain constant for the life of a flow. A
>     > >     >>>> retransmitted packet is part of the same flow as the
>     > >     >>>> originally-transmitted packet. So this seems to be contradicting the
>     > >     >>>> very specification of the FL.
>     > >     >>>>
>     > >     >>>> For instance, If a RTO for a flow causes the FL to change, then one may
>     > >     >>>> possibly argue that the FL is not naming/labeling what is said/expected
>     > >     >>>> to be anming/labeling.
>     > >     >>>
>     > >     >>> Specifically, RFC6437 states:
>     > >     >>>
>     > >     >>> "It is therefore RECOMMENDED that source hosts support the flow label
>     > >     >>> by setting the flow label field for all packets of a given flow to the
>     > >     >>> same value chosen from an approximation to a discrete uniform
>     > >     >>> distribution."
>     > >     >>>
>     > >     >>> So that is clearly a just recommendation, and not a requirement (and
>     > >     >>> definitely not a MUST). Furthermore, RFC6437 states:
>     > >     >>>
>     > >     >>> "A forwarding node MUST either leave a non-zero flow label value
>     > >     >>> unchanged or change it only for compelling operational security
>     > >     >>> reasons as described in Section 6.1."
>     > >     >>>
>     > >     >>> So there's no guarantee in the protocol specs that flow labels are
>     > >     >>> consistent for the life of the connection, which means that the
>     > >     >>> network cannot assume that and thus it would be incorrect if the
>     > >     >>> network tried to enforce flow label consistency as a protocol
>     > >     >>> requirement. As I said, it is prudent to try to be consistent with
>     > >     >>> flow labels and the default behavior in Linux should be changed,
>     > >     >>> however I do not believe there's a valid claim of non-conformance that
>     > >     >>> motivates removal of the feature that is already deployed.
>     > >     >>>
>     > >     >>> Tom
>     > >     >>>
>     > >     >>>
>     > >     >>>
>     > >     >>>
>     > >     >>>>
>     > >     >>>>
>     > >     >>>>
>     > >     >>>>> The feature is useful in large datacenter networks, like
>     > >     >>>>> pparently Facebook where the patches originate, since information
>     > >     >>>>> discerned by TCP can opportunistically be applied to route selection.
>     > >     >>>>> The practical issue is that there are stateful devices like firewalls
>     > >     >>>>> that require consistent routing in the network in which case changing
>     > >     >>>>> the flow label can confuse them. As I mentioned, the original intent
>     > >     >>>>> was that the flow label randomization feature should be opt-in instead
>     > >     >>>>> of on by default.
>     > >     >>>>
>     > >     >>>> So... where is the "source" of the packet that would be "modulating" the FL?
>     > >     >>>>
>     > >     >>>> Thanks,
>     > >     >>>> --
>     > >     >>>> Fernando Gont
>     > >     >>>> e-mail: [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>> || [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>     > >     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>     > >     >>>>
>     > >     >>>>
>     > >     >>>>
>     > >     >>>
>     > >     >>> _______________________________________________
>     > >     >>> v6ops mailing list
>     > >     >>> [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>     > >     >>> https://www.ietf.org/mailman/listinfo/v6ops
>     > >     >>>
>     > >
>     > >

---

From: Alexander Azimov <[email-masked]>
Date: Thu, 26 Nov 2020 23:36:33 +0300

Brian,

May I ask you to give more details on this issue:

I am convinced that it is often unsafe, because that depends on exactly how
> a server farm load balancer handles such cases. If it maintains the TCP
> session on the same actual server if there is SYN(-ACK) retransmission, the
> flow label must not be changed.

If a retransmitted SYN goes to another server/anycast point this won't
bring much harm. At least we haven't experienced such problems in our
datacenters.

чт, 26 нояб. 2020 г. в 22:53, Brian E Carpenter <[email-masked]
>:

> before the connection is established, and it looks safe.
>
> I am convinced that it is often unsafe, because that depends on exactly
> how a server farm load balancer handles such cases. If it maintains the TCP
> session on the same actual server if there is SYN(-ACK) retransmission, the
> flow label must not be changed.
> session persistence at the server is real. (Remember that the source
> cannot know by magic that there is server load balancing at the far end,
> but it's the common case today.)
>
> As noted before, this is only safe in a limited domain where it is known
> with certainty that server load balancing is not in use. I think it's
> therefore a corner case and specifically, the appropriate normative
> statement would be SHOULD NOT, which is defined very carefully in RFC2119
> exactly for this sort of case.
>
> But in fact I think we should not attempt to legislate on this point. The
> value is marginal at best. As others have said, this is not the way to
> tackle operational routing problems.
>
> Regards
>    Brian Carpenter
>
> > Later the thread turned into an argument if the flow label can be
> changed during connection/flow lifetime according to current RFC documents,
> though these documents can be updated. This looks a bit weird for me
> because I always thought that it is the IETF community's responsibility to
> document proper solutions. If something undocumented but worthy happens in
> the industry - IETF should catch up. So, I would like to get back to the
> >     > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <
> [email-masked] <mailto:[email-masked]> <mailto:
> [email-masked] <mailto:[email-masked]>>> wrote:
> >     > >
> [email-masked] <mailto:[email-masked]> <mailto:
> [email-masked] <mailto:[email-masked]>>> wrote:
> >     > >     >>
> [email-masked] <mailto:[email-masked]> <mailto:
> [email-masked] <mailto:[email-masked]>>> wrote:
> >     > >     >>>>
> >     > >     >>>> Hi, Tom,
> >     > >     >>>>
> >     > >     >>>> e-mail: [email-masked] <mailto:
> [email-masked]> <mailto:[email-masked] <mailto:
> [email-masked]>> || [email-masked] <mailto:
> [email-masked]> <mailto:[email-masked] <mailto:
> [email-masked]>>
> >     > >     >>> [email-masked] <mailto:[email-masked]> <mailto:
> [email-masked] <mailto:[email-masked]>>

-- 
Best regards,
Alexander Azimov

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 27 Nov 2020 14:50:36 +1300

On 27-Nov-20 09:36, Alexander Azimov wrote:
> If a retransmitted SYN goes to another server/anycast point this won't bring much harm. At least we haven't experienced such problems in our datacenters.

I think depends on details that are definitively out of the IETF's scope. When exactly does the load balancing system assign a session to a given server, how much state does that create, what resources does it reserve, and when are the state and the reserved resources released?

Also, in some cases, I could imagine that the data centre will not see a problem, but the client might see one that they would not otherwise have seen. Again, it depends on details that are out of our scope. So I think it's a topic that we should intentionally avoid. (Possibly it's a topic that QUIC should consider, when they consider server load balancing in general.)

Regards
   Brian

> 
> чт, 26 нояб. 2020 г. в 22:53, Brian E Carpenter <[email-masked] <mailto:[email-masked]>>:
> 
>     Alexander,
> 
>     > чт, 26 нояб. 2020 г. в 03:16, Tom Herbert <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>:
>     >
>     >     <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     >     >
>     >     > I'm not Joel, but I did once spend some time grepping RFCs to find out whether
>     >     > "flow" or "microflow" was the preferred term. In RFC2474, which is normative,
>     >     > we have:
>     >     >
>     >     >    Microflow: a single instance of an application-to-application flow of
>     >     >    packets which is identified by source address, destination address,
>     >     >    protocol id, and source port, destination port (where applicable).
>     >     >
>     >     > But in the flow label work we explicitly avoided being that precise, and
>     >     > did not use the term "microflow". There might be some load balancing
>     >     > scenarios where you want a broader definition, even including bidirectional
>     >     > flows. There are expired drafts on that topic:
>     >     > draft-tarreau-extend-flow-label-balancing
>     >     > draft-wang-6man-flow-label-reflection
>     >     >
>     >     Brian,
>     >
>     >     Random Packet Spraying
>     >     > Regards
>     >     >    Brian
>     >     > On 26-Nov-20 11:13, Tom Herbert wrote:
>     >     > > Joel, is there a normative definition of a flow?
>     >     > >
>     >     > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>     >     > >
>     >     > >     No, as Brian says, there are escape clauses in the flow definitions.
>     >     > >
>     >     > >     But changing the flow label due to traffic problems does not correspond
>     >     > >     to packets being in actually different flows.
>     >     > >     If one were using UDP, and mixing loss sensitive packets with loss
>     >     > >     insensitive packets for a special application, sure, one could use two
>     >     > >     flow labels.  But that is not what you are describing.
>     >     > >
>     >     > >     Yours,
>     >     > >     Joel
>     >     > >
>     >     > >     On 11/25/2020 3:05 PM, Tom Herbert wrote:
>     >     > >     > Joel,
>     >     > >     >
>     >     > >     > Is there an RFC that clearly and unambiguously states that a host MUST
>     >     > >     > use the same flow label for the lifetime _and_ clearly defines exactly
>     >     > >     > what a flow is with respect to such a requirement (for instance, how
>     >     > >     > would you define a flow and enforce such a requirement in UDP? IPsec?
>     >     > >     > other encapsulations?). If there is such a requirement then we'll
>     >     > >     > change the code to be conformant.
>     >     > >     >
>     >     > >     > Tom
>     >     > >     >
>     >     > >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>     >     > >     >>
>     >     > >     >> This kind of thing is why, as I understand it, MPTCP has discovery
>     >     > >     >> mechanisms ot know if both sides use it, and can select alternative
>     >     > >     >> addresses for communication.
>     >     > >     >>
>     >     > >     >> Trying to guess flow labels that might avoid a problem because it might
>     >     > >     >> be an ECMP problem, is just flailing about.  Not a good design for
>     >     > >     >> operational protocols.
>     >     > >     >>
>     >     > >     >> And in general, designing protocols around "I know exactly what is going
>     >     > >     >> on"  (the requirement for what you describe that goes well beyond just
>     >     > >     >> "limited domains") is also a recipe for failure.
>     >     > >     >>
>     >     > >     >> The Flow Label RFCs are actually very explicit that a flow label is
>     >     > >     >> supposed to be stable for the life of the flow.  Otherwise, it isn't a
>     >     > >     >> flow label.
>     >     > >     >>
>     >     > >     >> Yours,
>     >     > >     >> Joel
>     >     > >     >>
>     >     > >     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
>     >     > >     >>> Hi Fernando, comments in line...
>     >     > >     >>>
>     >     > >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>     >     > >     >>>>
>     >     > >     >>>> Hi, Tom,
>     >     > >     >>>>
>     >     > >     >>>> On 24/11/20 16:43, Tom Herbert wrote:
>     >     > >     >>>> [....]
>     >     > >     >>>>> Modulating the flow label is a means to affect the routing of packets
>     >     > >     >>>>> through the network that uses flow labels as input to the ECMP hash.
>     >     > >     >>>>
>     >     > >     >>>> What's the point?
>     >     > >     >>>>
>     >     > >     >>>> 1) You cannot tell *if* the FL is being used.
>     >     > >     >>>>
>     >     > >     >>> Generally true, but in a limited domain this information could be
>     >     > >     >>> discerned. I'd note that it's also generally true that we don't know
>     >     > >     >>> if there is a load balancer or stateful firewall in the path that
>     >     > >     >>> requires consistent routing, but in a limited domain we could know
>     >     > >     >>> that also.
>     >     > >     >>>
>     >     > >     >>>> 2) Changing the FL does not necessarily mean that packets will employ a
>     >     > >     >>>> different link.
>     >     > >     >>>
>     >     > >     >>> It's an opportunistic mechanism. If a connection is failing and we get
>     >     > >     >>> a better path that fixes it by simply changing the flow label then
>     >     > >     >>> what's the harm?
>     >     > >     >>>
>     >     > >     >>>>
>     >     > >     >>>> 3) If the network is failing, shouldn't you handle this via routing?
>     >     > >     >>>>
>     >     > >     >>> Sure, but then that requires an out of band feedback loop from a TCP
>     >     > >     >>> implementation to the network infrastructure to indicate there is a
>     >     > >     >>> problem and then the network needs to respond. That's significant
>     >     > >     >>> infrastructure and higher reaction time than doing something in TCP
>     >     > >     >>> and IP. Think of modulating the flow label is an inexpensive form of
>     >     > >     >>> source routing within a limited domain that doesn't need any
>     >     > >     >>> infrastructure or heavyweight protocols or something like segment
>     >     > >     >>> routing.
>     >     > >     >>>
>     >     > >     >>>>
>     >     > >     >>>>
>     >     > >     >>>>> The basic idea is that the flow label associated with a connection is
>     >     > >     >>>>> randomly changed when the stack observes that the connection is
>     >     > >     >>>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
>     >     > >     >>>>> this since the source is at liberty to set the flow label as it sees
>     >     > >     >>>>> fit.
>     >     > >     >>>>
>     >     > >     >>>> The FL is expected to remain constant for the life of a flow. A
>     >     > >     >>>> retransmitted packet is part of the same flow as the
>     >     > >     >>>> originally-transmitted packet. So this seems to be contradicting the
>     >     > >     >>>> very specification of the FL.
>     >     > >     >>>>
>     >     > >     >>>> For instance, If a RTO for a flow causes the FL to change, then one may
>     >     > >     >>>> possibly argue that the FL is not naming/labeling what is said/expected
>     >     > >     >>>> to be anming/labeling.
>     >     > >     >>>
>     >     > >     >>> Specifically, RFC6437 states:
>     >     > >     >>>
>     >     > >     >>> "It is therefore RECOMMENDED that source hosts support the flow label
>     >     > >     >>> by setting the flow label field for all packets of a given flow to the
>     >     > >     >>> same value chosen from an approximation to a discrete uniform
>     >     > >     >>> distribution."
>     >     > >     >>>
>     >     > >     >>> So that is clearly a just recommendation, and not a requirement (and
>     >     > >     >>> definitely not a MUST). Furthermore, RFC6437 states:
>     >     > >     >>>
>     >     > >     >>> "A forwarding node MUST either leave a non-zero flow label value
>     >     > >     >>> unchanged or change it only for compelling operational security
>     >     > >     >>> reasons as described in Section 6.1."
>     >     > >     >>>
>     >     > >     >>> So there's no guarantee in the protocol specs that flow labels are
>     >     > >     >>> consistent for the life of the connection, which means that the
>     >     > >     >>> network cannot assume that and thus it would be incorrect if the
>     >     > >     >>> network tried to enforce flow label consistency as a protocol
>     >     > >     >>> requirement. As I said, it is prudent to try to be consistent with
>     >     > >     >>> flow labels and the default behavior in Linux should be changed,
>     >     > >     >>> however I do not believe there's a valid claim of non-conformance that
>     >     > >     >>> motivates removal of the feature that is already deployed.
>     >     > >     >>>
>     >     > >     >>> Tom
>     >     > >     >>>
>     >     > >     >>>
>     >     > >     >>>
>     >     > >     >>>
>     >     > >     >>>>
>     >     > >     >>>>
>     >     > >     >>>>
>     >     > >     >>>>> The feature is useful in large datacenter networks, like
>     >     > >     >>>>> pparently Facebook where the patches originate, since information
>     >     > >     >>>>> discerned by TCP can opportunistically be applied to route selection.
>     >     > >     >>>>> The practical issue is that there are stateful devices like firewalls
>     >     > >     >>>>> that require consistent routing in the network in which case changing
>     >     > >     >>>>> the flow label can confuse them. As I mentioned, the original intent
>     >     > >     >>>>> was that the flow label randomization feature should be opt-in instead
>     >     > >     >>>>> of on by default.
>     >     > >     >>>>
>     >     > >     >>>> So... where is the "source" of the packet that would be "modulating" the FL?
>     >     > >     >>>>
>     >     > >     >>>> Thanks,
>     >     > >     >>>> --
>     >     > >     >>>> Fernando Gont
>     >     > >     >>>> e-mail: [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> || [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>     >     > >     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>     >     > >     >>>>
>     >     > >     >>>>
>     >     > >     >>>>
>     >     > >     >>>
>     >     > >     >>> _______________________________________________
>     >     > >     >>> v6ops mailing list
>     >     > >     >>> [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>     >     > >     >>> https://www.ietf.org/mailman/listinfo/v6ops
>     >     > >     >>>
>     >     > >
>     >     > >
>     >     > > _______________________________________________
>     >     > > v6ops mailing list
>     >     > > [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>     >     > > https://www.ietf.org/mailman/listinfo/v6ops
>     >     > >
>     >     >
>     >
>     >     [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>

---

From: Fernando Gont <[email-masked]>
Date: Fri, 27 Nov 2020 17:41:40 -0300

Hello, Tom,

> requires consistent routing, but in a limited domain we could know

Exactly. So my take is that the drawbacks for the general case outweigh 
the benefits of the specific case.

> a better path that fixes it by simply changing the flow label then

Complexity, and the possible negative implications for the general case.

That said, why don't you fix the problem where you should (i.e., routing)?

> problem and then the network needs to respond.

Not really. TCP does not deal with paths (not should it). If the network 
detects a link is failing, it should route around the problem.

> That's significant

Downsides:

* It breaks load balancing
* It depends on the source doing something -- but you don't necessarily 
control the source -- and you cannot tell if the source implements it
* It doesn't solve the problem for nodes that do not behave as expected

My take is that if you really care about this, you're already (or 
should!) be solving the problem in a different (and more general) way.

> reasons as described in Section 6.1."

My concern is that we may leave the FL unusable.

> requirement.

You're probably right in that respect. And, in retrospective, I think it 
would have been better if we had specified the FL to be unmutable.

> As I said, it is prudent to try to be consistent with

Sorry: is the current Linux behavior to modulate the FL?

> motivates removal of the feature that is already deployed.

I personally consider the changing the FL for a flow to be a bug. That 
said, if the default setting is the correct/compliant behavior, at the 
end of the day we're all adults, and whoever overrides the default 
behavior is supposed to know what he/she does -- and if they don't, they 
get what they deserve  ;-)

So, in that light, I wouldn't push to remove the optional behavior, 
*provided* the default behavior is correct.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Fri, 27 Nov 2020 17:47:21 -0300

Tom,

> doesn;t have rigidly defined requirements on the host.

You don't need to randomize the FL of packets corresponding to a flow to 
get this --- simply do round-robin at the router that wants load-sharing 
over different links.

OTOH, if you do randomize the FL of each packet, then routers that want 
to avoid packet reordering simply can't.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Fri, 27 Nov 2020 17:52:14 -0300

On 26/11/20 06:34, Alexander Azimov wrote:
> changed during connection/flow lifetime according to current RFC 
> documents, though these documents can be updated. This looks a bit weird 
> for me because I always thought that it is the IETF community's 
> responsibility to document proper solutions. If something undocumented 
> but worthy happens in the industry - IETF should catch up. So, I would 
> like to get back to the discussion of reasons and their safety.
> 
> outage/degradation looks obvious. Getting kind of source-based routing 
> can significantly reduce the reaction time and improve end-user 
> experience. The flow label has a perfect match here: transparent for the 
> application, set by the source, not part of 5-tuple, while it can be 
> used in the load balancing. 

If you have two links, then, upon change of the FL, you have 50% change 
that the path will be different -- not good.  What would you do if the 
failure continues? -- keep changing the FL?

> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be 

This one MUST be switched of by default, too. (along with #2).

> before the connection is established, and it looks safe. The second one 
> can also improve performance if it is set on the server-side. Please 
> comment if you see security flaws in such a design.

At the very least, one should analyze the case where in #1, a packet 
gets delayed, you change the FL and retransmit, and finally you get a 
response to the original packet (and/or to both packets).

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Fri, 27 Nov 2020 16:31:38 -0700

Fernando,

All we know in this case is that *TCP* has detected a connection is
failing and there may be a potential problem with a path, we don't
know that the network has detected any problems like some link is
failing or will ever detect any problem. So in such a scenario, what
recourse does the host have to try to salvage its connections? If the
answer is that the host should always patiently wait for the network
to figure out what is happening, then that's going to be a hard sell
to application and host stack developers (remember the network is a
black box from their perspective that they really don't trust). If the
answer is that the host is supposed to inform the network of issues
with it's connections and then host and network work together to solve
then that's great; but then what is the general protocol that has been
established to facilitate that and solved the problem of how hosts and
network work together in concert to solve user's problems?

Tom

---

From: Mark Smith <[email-masked]>
Date: Sat, 28 Nov 2020 11:32:14 +1100

On Sat, 28 Nov 2020, 10:32 Tom Herbert, <[email-masked]> wrote:

> recourse does the host have to try to salvage its connections?

Retransmit their packets, let the network take care of trying to deliver
them again.

If the
> to figure out what is happening,

Yes.

Steering packets is the network's job. People like me who run the network
expect to be in charge of that. We have fancy mechanisms that mean
alternate paths are switched to very quickly.

The trouble with having two parties trying to solve the same problem at the
same time without coordination is they're just as likely to work against
each other as they are inadvertently to work with each other.

If hosts want the network to respect the flow label, then hosts need to
ensure that the a flow label value for a flow of packets with the same
source and destination address consistently represents a flow.

If hosts start trying to take control of steering traffic by varying the
flow label value within a flow, making the network's job harder, and the
network being blamed for the possibility bad network performance that may
result, the response will be both quick and easy - network operators will
switch off looking the Flow Label.

The Flow Label is the IPv6 header analogue for transport layer protocol
ports, so that load balancing mechanisms don't have to dig further into the
packet. Can or are transport layer ports varied in TCP SYNs, SYN/ACKs and
ACKs within the same TCP connection?

then that's going to be a hard sell

---

From: Joseph Touch <[email-masked]>
Date: Fri, 27 Nov 2020 16:38:18 -0800

> On Nov 27, 2020, at 4:32 PM, Mark Smith <[email-masked]> wrote:
> 
> Steering packets is the network's job.

+1

> The Flow Label is the IPv6 header analogue for transport layer protocol ports, so that load balancing mechanisms don't have to dig further into the packet.

+1

> Can or are transport layer ports varied in TCP SYNs, SYN/ACKs and ACKs within the same TCP connection?

In general, no.

I proposed a method to decouple the SYN destination port (which currently is both part of the connection ID *and* identifies the service) from the port used for the rest of the connection, but largely to provide a separate service ID (originally draft-touch-tcp-portnames, later recast as a service name option in draft-touch-tcpm-sno). But that didn’t as much use different ports as decouple the service from connection ID number.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Sat, 28 Nov 2020 00:49:52 -0300

On 27/11/20 20:31, Tom Herbert wrote:
[....]
> recourse does the host have to try to salvage its connections?

None.

TCP times out and restransmits. And *the network* routes around failing 
paths.

> black box from their perspective that they really don't trust).

Either that, of you switch to a different destination address if the 
transport protocol allows.

> then that's great; 

The only thing the host can do is to switch to a different transport 
protocol if it allows.

> but then what is the general protocol that has been

They don't really work together (modulo congestion control, one might 
argue), but rther complement each other (TCP times out and retransmits, 
and the network routes around failing paths).

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Sat, 28 Nov 2020 00:52:35 -0300

On 27/11/20 21:32, Mark Smith wrote:
> 
> On Sat, 28 Nov 2020, 10:32 Tom Herbert, <[email-masked] 
> <mailto:[email-masked]>> wrote:
[....]
> network being blamed for the possibility bad network performance that 
> may result, the response will be both quick and easy - network operators 
> will switch off looking the Flow Label.

THat is *exactly* why I noted that I'm concerned that we seem to be on 
the path of making the FLow Label unusable.

Thanks,

---

From: Mark Smith <[email-masked]>
Date: Sat, 28 Nov 2020 17:42:20 +1100

> the path of making the FLow Label unusable.

I see the Flow Label as a hint, but only a hint, of what the hosts wants
from the network. It's useful as an expression of what the host would like
to have.

However, if that hint causes me any trouble, I'll ignore it and then
devolve to trying to deliver the packet based just on its destination
address.

---

From: Fernando Gont <[email-masked]>
Date: Sat, 28 Nov 2020 05:01:50 -0300

On 28/11/20 03:42, Mark Smith wrote:
> 
> 
> On Sat, 28 Nov 2020, 14:52 Fernando Gont, <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     <mailto:[email-masked]>
>      > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     [....]
>      >
> from the network. It's useful as an expression of what the host would 
> like to have.

As you correctly noted, it's supposed to be an identifier of the flow, 
because it's hard to get for the port numbers.

I don't think it can bee seen as ad expression of what the host wants to 
ahve, because the Flow ID has no semantics, and the host doesn't even 
know what's in the network.

> devolve to trying to deliver the packet based just on its destination 

If you *needed* (or could make use of) the FL, then, if it causes 
trouble, you either resort to finding the five-tuple, or quite likely 
end of dropping packets.

Thanks,

---

From: Alexander Azimov <[email-masked]>
Date: Sat, 28 Nov 2020 13:14:51 +0300

Gentlemen,

> response to the original packet (and/or to both packets).

As the suggestion strictly speaks about changing FL upon RTO timeouts I
don't see how it differs from getting two responses in case of RTO without
FL change. Nothing special here.

> failure continues? -- keep changing the FL?

Yes, and if only one path is affected - you have a chance of hitting a good
path with 1 - (1/2)^n. And it's much better than nothing.

> I think depends on details that are definitively out of the IETF's scope.
> When exactly does the load balancing system assign a session to a given
> server, how much state does that create, what resources does it reserve,
> and when are the state and the reserved resources released?
> Also, in some cases, I could imagine that the data centre will not see a
> problem, but the client might see one that they would not otherwise have
> seen.

I can't say that these details look convincing to me. I also can't agree
that operational considerations are out of IETF's scope. The protocols are
designed with real operations in mind, aren't they?

> alternate paths are switched to very quickly.

While not questioning innovative technologies that take care of traffic, a
'good' repair time counts in seconds. 'Bad' repair time, when the control
plane becomes broken or a famous silent drop happens counts in minutes and
requires a network engineer to be summoned. If TCP can fix itself in
milliseconds - it's great. And it is needed by real operations. You also
missing part when something happens outside of your domain, where you can't
instantly summon an engineer. If you tell me that you've been never
affected by the network outage of your peering partner I will be greatly
suprised.

-- 
Best regards,
Alexander Azimov

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 28 Nov 2020 19:09:08 +0100

> On 28. Nov 2020, at 00:31, Tom Herbert <[email-masked]> wrote:
> 
Hi Tom,

it is perfectly fine that if a TCP connection fails and the application
triggers a setup of a new TCP connection, the new TCP connections
uses a flowlabel different from the one used for the old TCP connection.
However, a single RTO does not indicate a connection failure.
Normally you would need several consecutive RTOs to happen before TCP
declares the connection to be failed.

Best regards
Michael

---

From: Tom Herbert <[email-masked]>
Date: Sat, 28 Nov 2020 11:27:33 -0700

On Sat, Nov 28, 2020 at 3:15 AM Alexander Azimov <[email-masked]> wrote:
> As the suggestion strictly speaks about changing FL upon RTO timeouts I don't see how it differs from getting two responses in case of RTO without FL change. Nothing special here.
>
> Yes, and if only one path is affected - you have a chance of hitting a good path with 1 - (1/2)^n. And it's much better than nothing.
>
>> Also, in some cases, I could imagine that the data centre will not see a problem, but the client might see one that they would not otherwise have seen.
>
> I can't say that these details look convincing to me. I also can't agree that operational considerations are out of IETF's scope. The protocols are designed with real operations in mind, aren't they?
>
>> Steering packets is the network's job. People like me who run the network expect to be in charge of that. We have fancy mechanisms that mean alternate paths are switched to very quickly.
>
> While not questioning innovative technologies that take care of traffic, a 'good' repair time counts in seconds. 'Bad' repair time, when the control plane becomes broken or a famous silent drop happens counts in minutes and requires a network engineer to be summoned. If TCP can fix itself in milliseconds - it's great. And it is needed by real operations. You also missing part when something happens outside of your domain, where you can't instantly summon an engineer. If you tell me that you've been never affected by the network outage of your peering partner I will be greatly suprised.
>
+1

I would also point out this feature has been in deployment for at
least five years in at least two hyperscalers-- there has been no
major meltdown. We'll make the feature opt-in because we may be
exposed to non-conformant devices in the network that make incorrect
assumptions, but I have yet to see any convincing argument that we
should remove the feature becauses it's fundamentally harmful or not
providing benefits.

As for the mentality of "the network will fix the problem", that's
obviously an easy statement to make from a router vendor point of
view, but that luxury simply doesn't match the realities of how host
stacks are implemented deployed at global scale. This starts with the
fact that there is no such thing as "the network"; Linux stack for
instance needs to run on every network on Earth and I have no idea
which of these networks provides adequate security, fault detection
and repair, are well managed, or even have devices that conform to the
standards (non-conformant network devices are common). The only
recourse we, i.e. host side developers, have is to stick with the
least common denominator in the general case, and allow optional, but
still protocol conformant, features in limited use cases such as
limited domains (hence why the flow label modulation feature should be
optional as we've already established in this thread).

Tom

---

From: Fernando Gont <[email-masked]>
Date: Sun, 29 Nov 2020 02:53:01 -0300

Tom,

On 28/11/20 15:27, Tom Herbert wrote:
[....]
> should remove the feature becauses it's fundamentally harmful or not

It breaks FL-based load-sharing.

> stacks are implemented deployed at global scale.

Not sure what you mean. We rely on that for the current Internet (hosts 
do not do source-routing).

> This starts with the

Modulo the comments I've already made, I'm afraid we might be in the 
path of making the FL unusable...

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Sun, 29 Nov 2020 09:25:59 -0700

> path of making the FL unusable...

Fernanda,

I really don't understand how this makes the flow label "unusable", to
the contrary this is a useful case with good effect and justifies the
work in the host to set the field. In any case, it would be great if
you could provide the draft with the normative requirements that would
make the flow label "usable".

Please include requirements for both hosts and network nodes. If there
is going to the be a requirement that hosts MUST make a 1:1 mapping to
a transport connection and the flow label MUST be consistent for the
life of connection, then I would expect there to also be a requirement
that the flow label MUST be immutable by all devices in the path-- in
particular the requirement of RFC6437 would need to be updated: "A
forwarding node MUST either leave a non-zero flow label value
unchanged or change it only for compelling operational security
reasons..."-- if hosts are not allowed to change a field they are
primarily responsible for setting, then the network should not be
allowed to modify it either.

Additionally, if the underlying design principle is that flow
identification in a packet is immutable and persistent, then I'd like
to see the discussion on how that rectifies with NAT where
intermediate nodes routinely change IP addresses and transport port
numbers without the host endpoints' knowledge-- unlike the flow label,
port numbers are formally part of the transport connection
identification, so changing them in flight is far more invasive and
potentially harmful than changing the flow label (we've seen far more
problems caused by NAT than flow label due to NAT state evictions,
packets of a flow being re-routed to not hit the same NAT device,
etc.).

Tom