Subject: [Fwd: I-D Action:draft-carpenter-6man-flow-update-03.txt]

From: Brian E Carpenter <[email-masked]>
Date: Fri, 07 May 2010 13:10:40 +1200

Hi,

This is revised again according to discussion on the list, and some
off-list discussion with Shane Amante in particular.

Firstly, since there seemed to be a feeling that a full update
of RFC 3697 is better than publishing a set of changes, this is
now just informational, although written using normative language.
The major next step would be a 3697bis draft.

Secondly, it offers the WG a binary choice as the main decision:

  "There appear to be two viable approaches:
   1.  Definitively forbid locally defined use of the flow label.
       Strengthen RFC 3697 to say that hosts SHOULD set a pseudo-random
       label value, which would clarify and limit its possible uses.  In
       particular, its use for load balancing and possibly as a nonce
       would be encouraged.
   2.  Encourage locally defined use of the flow label.  This approach
       would make the flow label mutable and would exclude any use case
       depending on end-to-end immutability.  It would encourage
       applications of a pseudo-random flow label, such as load
       balancing, on a local basis, but it would exclude end-to-end
       applications such as [I-D.blake-ipv6-flow-label-nonce]."

Please, can we focus on this choice rather than the fine details,
initially? Also, please read the draft as a whole; looking at diffs
would be quite confusing.

   Brian + Sheng

-------- Original Message --------
Subject: I-D Action:draft-carpenter-6man-flow-update-03.txt
Date: Thu,  6 May 2010 18:00:01 -0700 (PDT)
From: [email-masked]
Reply-To: [email-masked]
To: [email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title           : Update to the IPv6 flow label specification
	Author(s)       : B. Carpenter, S. Jiang
	Filename        : draft-carpenter-6man-flow-update-03.txt
	Pages           : 11
	Date            : 2010-05-06

Various published proposals for use of the IPv6 flow label are
incompatible with its existing specification in RFC 3697.  This
document proposes changes to the specification that permit additional
use cases.  The concept of flow label domains is introduced, with the
label possibly being rewritten at domain boundaries.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-carpenter-6man-flow-update-03.txt

---

From: Tina TSOU <[email-masked]>
Date: Fri, 07 May 2010 11:52:04 +0800

I support approach 2.

B. R.
Tina
http://tinatsou.weebly.com/contact.html

----- Original Message ----- 
From: "Brian E Carpenter" <[email-masked]>
To: "6man" <[email-masked]>
Sent: Friday, May 07, 2010 9:10 AM
Subject: [Fwd: I-D Action:draft-carpenter-6man-flow-update-03.txt]

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Thomas Narten <[email-masked]>
Date: Fri, 07 May 2010 08:55:42 -0400

I haven't decided yet whether I can support choice 2. 

However, should choice 2 be used, what is the recommendation when
IPsec is in use? The recommendation assumes access to port numbers and
the like, which are hidden when IPsec is in use.

This is particularly important when a packet exits one FL Domain and
enters a new one, and the recommendation is "throw out the FL and
figure out a new value to use".

Thomas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 07 May 2010 09:28:57 -0400

The more I think about "encouraging" locally defined use of the flow 
label, the less I like it.

The basic problem is that in the context we are discussing, is for use 
by routers.
If you have locally defined flow label usage, then
1) Any vendor selling to an operator has to somehow manage to support 
their locally defined behavior.
2) Operators MUST configure all their routers to properly understand 
their flow label meaning.
2') Given that it can not be counted on, even the global interpretation 
such as the use of non-zero flow-labels for ECMP/LAG balancing would be 
to be off by default, since there would be no default expect ability to 
match that use.
3) This is quite fragile.  Given that there is no indication in the 
packet of what meaning of flow-label was intended, there is nothing that 
will alert an operator if (as seems periodically inevitable some of the 
routers are attempting to apply a different interpretation than the 
operator intends.

Basically, from a vendor's perspective, a multiplicity of local 
interpretations of flow label is a serious impediment to implementing 
any usage of flow label at all.

One could argue that as long as the routers are all from one vendor, 
they will support the same local interpretation.  I hope we are not 
going there!

Hence, to be clear and specific, I support approach 1, and oppose 
approach 2.

Yours,
Joel

---

From: Christian Huitema <[email-masked]>
Date: Fri, 7 May 2010 13:49:08 +0000

I agree with Joel, in great part because of the IPSEC issue that Thomas mentions. Final payload and ports can be easily obfuscated behind chains of headers, or encrypted in IPSEC. Specifying that the flow label SHOULD be set to hash of specified values would lead to greater inter-operability. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Friday, May 07, 2010 6:29 AM
To: Brian E Carpenter
Cc: 6man
Subject: Re: [Fwd: I-D Action:draft-carpenter-6man-flow-update-03.txt]

Yours,
Joel

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 7 May 2010 17:07:57 +0200

On May 7, 2010, at 15:28, Joel M. Halpern wrote:

> The more I think about "encouraging" locally defined use of the flow label, the less I like it.
> 
> The basic problem is that in the context we are discussing, is for use by routers.
> 1) Any vendor selling to an operator has to somehow manage to support their locally defined behavior.
> 2) Operators MUST configure all their routers to properly understand their flow label meaning.
> 2') Given that it can not be counted on, even the global interpretation such as the use of non-zero flow-labels for ECMP/LAG balancing would be to be off by default, since there would be no default expect ability to match that use.
> 3) This is quite fragile.  Given that there is no indication in the packet of what meaning of flow-label was intended, there is nothing that will alert an operator if (as seems periodically inevitable some of the routers are attempting to apply a different interpretation than the operator intends.
> 
> Basically, from a vendor's perspective, a multiplicity of local interpretations of flow label is a serious impediment to implementing any usage of flow label at all.
> 
> One could argue that as long as the routers are all from one vendor, they will support the same local interpretation.  I hope we are not going there!
> 
> Hence, to be clear and specific, I support approach 1, and oppose approach 2.

+1.

It appears that the flow label has long been an "attractive protocol", i.e. a solution (more precisely: 20 spare bits) looking for a problem.
So far, the only set of problems that it appears to be useful for is *end-to-middle hints*.

-- it is not very useful for end-to-end, as it is not protected by destination functions such as the pseudo-header or AH.
-- it is not very useful for middle-to-middle, as we don't know how to properly nest this usage.
-- it also is not very useful for anything but hints, because it is not protected. 

The end-to-middle hints that we have found so far appear to be:

a) grouping of packets into flows where it is useful to ask for maintaining sequence in ECMP/LAG scenarios
b) selection of forwarding behavior (RPL instance ID) in RPL-based domains.

I'm not sure I like (b) yet, because it is hard to use by nodes outside these domains, who may not know what forwarding classes are available.  It is bad when the packet has to traverse multiple RPL domains.  Otherwise, stealing bits from the flow label for traffic-class-style information (as long as it is about hints) sounds almost reasonable.

Gruesse, CArsten

---

From: Shane Amante <[email-masked]>
Date: Fri, 7 May 2010 10:42:54 -0600

Thomas, Christian,

I'm responding to both of you in a single message, since you both expressed concern with choice 2, (locally-defined use of flow-labels), particularly in the face of IPSec.

Let's first look at the situation as it stands today.  Today, if hosts or IPSec GW's sent IPSec ESP packets, then routers will not be able to find any Transport layer information anyway to be used as input-keys for LAG or ECMP hash operations.  Thus, they would revert to IP source and destination address /only/ as input-keys to LAG or ECMP hash operations.  While that's slightly better than better than nothing, there's definitely a higher risk of congesting a particular component-link inside a LAG or ECMP path, (as opposed to clear-text packets where Transport layer information is available and is used as input-keys for LAG + ECMP calculations).  In summary, if Transport layer information is unavailable to intermediate routers to extract and create a flow-label from (i.e.: locally defined use case), the load-balancing behavior is _no worse_ than it would be on today's network.[1]

Second, IPSec constitutes a very, very small fraction of the overall Internet traffic today and I personally don't see that changing in the near future.  So, IMHO, IPSec is more accurately a "corner case".

I would also note that even if IPSec hosts wanted to encode a non-zero flow-label, (from the inner packet's 5-tuple), administrators of those hosts MAY NOT want to, because it would be less information in the outer packet header that, to some extent, would prevent traffic analysis at intermediate nodes.

Let me try to put things back into a proper perspective wrt the need for a locally-defined flow-label:
1)  We _need_ a meaningful flow-label so that Core routers, with several dozen 100G interfaces, /DO NOT/ have to traverse IPv6 Extension Headers (some of which could be unknown!) to hope to find Transport-layer information (protocol, src port, destination port) in order to extract all those input-keys to use them for LAG and ECMP hash calculations.  I would note that all of my routers today are using a 5-tuple in IPv4 for input-keys for LAG & ECMP, which are pretty straightforward for Core routers to extract given they have known offsets in the IP header.
2)  On the edge of the network, i.e.: PE's, they need to be capable of looking at Transport-layer information in order to perform classification of incoming packets for a variety of reasons:
    a)  Classify traffic based on IP src, dst address and/or _protocol, src port or dst port_ in order to drop packets for a DoS attack, for example;
    b)  Classify traffic based on similar 5-tuple information in order to rate-limit it -or- apply a specific IP Traffic Class value in the header.
3)  The flow-label isn't "protected" in the IPv6 header by a checksum or anything else (to my knowledge).  IOW, a man-in-the-middle attack or misconfiguration in a 3rd-party host or network could scribble a bad value in flow-labels that leads Core routers (if they were using flow-labels as input-keys) to potentially re-order packets or congest component-links in a LAG or ECMP path.  So, as an operator, I'm not sure I want to "blindly" trust flow-labels even if hosts _were_ setting them to a pseudo-random value.  Furthermore, as an operator, I have no control over how good a hash over, say, the 5-tuple every host will use to create a "good" flow-label that will maximize the probability of distributing a flow over very wide component-links contained in LAG or ECMP paths in my network.  

In summary, I'm not sure the incentives of the host vendor's in creating "good" flow-labels are properly aligned with those of the SP's ... thus, as an operator, it seems prudent to take more control over my destiny by being able to set the flow-labels at ingress to my domain boundaries.

-shane

> On May 7, 2010, at 07:49 MDT, Christian Huitema wrote:
>

---

From: Shane Amante <[email-masked]>
Date: Fri, 7 May 2010 10:55:06 -0600

Joel,

> 2) Operators MUST configure all their routers to properly understand their flow label meaning.

Why wouldn't I tell my vendors that I want the _default_ behavior on PE's to be that they automatically set the flow-label to a pseudo-random hash of the incoming 5-tuple hash on all inbound IPv6 interfaces?  Sure, vendors could create a global and/or interface-specific knob for those operators who feel some reason to disable said default behavior (or, do something else "creative" to create a flow-label value).  However, I suspect operators would not want to do that since it would clearly break LAG and ECMP in their networks.

> 2') Given that it can not be counted on, even the global interpretation such as the use of non-zero flow-labels for ECMP/LAG balancing would be to be off by default, since there would be no default…

I disagree.  I would expect it to default to "on".  :-)

> 3) This is quite fragile.  Given that there is no indication in the packet of what meaning of flow-label was intended, there is nothing that will alert an operator if (as seems periodically inevita…

I agree the current -03 draft, as written, is still a bit fragile.  As one example, this rule:
---snip---
   6.  When a locally defined scheme other than the pseudo-random scheme
       is used, packets leaving the FL domain might contain a label that
       would be misinterpreted elsewhere.  Therefore, the boundary
       egress FL router SHOULD set the label according to the pseudo-
       random mechanism defined in rule 1.  If not, it MUST set the
       label to the default value of zero.
---snip---

I think it's operationally infeasible to expect such a FL domain NOT using a pseudo-random hash in the flow-label to consistently "reset" the flow-label to a pseudo-random hash or to the default of zero on egress.  Misconfigurations, bugs, etc. will happen in the real-world.  Thus, unless I /explicitly/ trust an adjacent flow-label domain is doing the right thing (i.e.: using a proper pseudo-random hash of the 5-tuple), then I'd always default to setting the flow-label value upon ingress to my network.

Regardless, I think these problems in the draft _can_ be addressed should the WG agree to pursue Option 2, (locally defined use case).

> Hence, to be clear and specific, I support approach 1, and oppose approach 2.

I hope I can persuade you to change your opinion.  :-)

-shane

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Fri, 7 May 2010 14:20:49 -0500

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Brian E Carpenter

>        applications such as [I-D.blake-ipv6-flow-label-nonce]."

#1 appeals because it could be used instead of the 5-tuple available in IPv4. On the other hand, a lack of header checksum in IPv6 spoils that party at least to some extent. If there's something that can be done to fix the lack of header checksum, I would favor #1.

Bert

---

From: "George, Wes E IV [NTK]" <[email-masked]>
Date: Fri, 7 May 2010 16:03:25 -0500

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Thursday, May 06, 2010 9:11 PM
To: 6man
Subject: [Fwd: I-D Action:draft-carpenter-6man-flow-update-03.txt]

[[WEG]] I prefer option 2. I echo Shane's concerns that I don't trust end hosts to be random enough (or random in the right way) to ensure that I can actually load-balance using flow label as even a partial input on a load-balancing hash decision. Further, that SHOULD in option 1 means that I can't assume that anything other than a zero value will be there when considering whether to use flow label as a part of my hash, making it just one more way that flow label won't be used.

I tend to think that the lack of a header checksum makes the flow label's immutability a bad assumption for end hosts (or end networks) to make anyway. I also haven't seen a compelling use case for an immutable flow label that makes #2 a bad idea by comparison. As I said at the mic in Anaheim, I'd rather see us use Flow label to solve a problem that we actually have, like needing to make loadbalancing work better, rather than reserving it for some compelling future use that no one can seem to come up with yet. [as an aside, blake-nonce is now expired and doesn't appear to have much in the way of support that I could find]

Moreover, I think that there are multiple other ways to manage a need to have persistent data within a header between endpoints across multiple domains, including extension headers, (especially if standardized under draft-krishnan-ipv6-exthdr) as well as transport-layer headers, or even payload data, most of which have mechanisms to ensure that the data hasn't been mucked with somewhere in the middle. I know there are concerns about scale when extension headers are involved, but I can say that in our core, we're basically treating them like IPv4 options - we ignore the options/EH, but pass the packet through the router if it's not "for us." So from that perspective, core routers don't care how many extension headers the packet has, nor what people are trying to do with them. This may still be an issue on PEs and other middleboxes like firewalls, but if we move towards standardizing the headers, perhaps not so much.

Thanks,
Wes George

This e-mail may contain Sprint Nextel Company proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.

---

From: Sheng Jiang <[email-masked]>
Date: Mon, 10 May 2010 10:44:34 +0800

> Behalf Of Joel M. Halpern
> Sent: Friday, May 07, 2010 9:29 PM
> To: Brian E Carpenter
> Cc: 6man

This is not a problem to concern at all. Any vendor would support any useful
features. That's the philosophy how the industry improves itself. The
discussion should be whether the support for locally defined behavior is
useful for operators.

> understand their flow label meaning.

I am not agree on "MUST". There are two levels of it. On operator level, the
locally defined behavior is optional. On the routers level, only routers
that want to distinguish the packet according to locally defined flow label
need to understand the flow label meaning; other routers can ignore flow
label and do simple forwarding like before.

> 2') Given that it can not be counted on, even the global 
> interpretation such as the use of non-zero flow-labels for 
> ECMP/LAG balancing would be to be off by default, since there 
> would be no default expect ability to match that use.

Agree, the collision may happen. One of the possible approach is that FL
domain could be hierarchical. Global FL domain can enforce certain behaviors
that would be default for all FL domain (I guess this would be IETF
standards). A sub FL domain needs to inherit behaviors from its parents FL
domain as default. In flat level, B FL domain does not have to understand A
FL domain and its egress routers may overwrite flow label from A.

> 3) This is quite fragile.  Given that there is no indication 
> in the packet of what meaning of flow-label was intended, 
> there is nothing that will alert an operator if (as seems 
> periodically inevitable some of the routers are attempting to 
> apply a different interpretation than the operator intends.

Except for global FL domain, all FL domains are under control of operators.
So are all routers. The egress routers of FL domains can overwrite flow
labels. So operators can overwrite all flow label with different
interpretations.
 
> Basically, from a vendor's perspective, a multiplicity of 
> local interpretations of flow label is a serious impediment 
> to implementing any usage of flow label at all.

A simple rule can be applied: handle what routers can understand and ignore
others.
 
> One could argue that as long as the routers are all from one 
> vendor, they will support the same local interpretation.  I 
> hope we are not going there!

There is nothing to do with single vendor. Vendors only provide the ability
to enable operator to use/define flow label. Interoperation between
different vendor is the reason we make standards here. Otherwise, private
implementation from vendor can support almost everything.

As one of the author, I personally prefer approach 2 since it encourage
operators to use flow label. It enable operator to define the meaning or
usage of flow label in their network scope as almost whatever they want.

Best regards,

Sheng
 
> > Firstly, since there seemed to be a feeling that a full 
> update of RFC 
> > 3697 is better than publishing a set of changes, this is now just 
> > informational, although written using normative language.

---

From: "George, Wes E IV [NTK]" <[email-masked]>
Date: Mon, 10 May 2010 09:39:14 -0500

One additional thought I had... if we move towards locally-defined flow label, we probably still need a use case and some direction for traffic that originates in tunnels prior to entering a network that is locally defining flow labels. Specifically, whatever method is employed to generate the flow label as packets come in is still going to be based on some set of data from the packet that may or may not be easily found by a PE or ASBR in the multiple layers of headers a tunneled protocol may generate, and the outer packet may not have enough info to generate an appropriate value for the flow label - multiple flows may look like the same flow because they have the same src/dst and protocol because they're being tunneled between the same endpoints, for example.

So, it may make sense to have a method where the device responsible for doing the encapsulation [should | must] insert a random value into the outer header's flow label that is based on some useful data from the inner headers. To the concern about whether it will be random enough, and whether it will conflict with the locally-significant flow labels, if it is based on the same criteria used for the locally significant flow labels, it's at least mostly safe to assume it'll be useful, and likely it'll be better than not having it. If we go with something like the previous draft, where the first few bits should be set using a defined flag value so that devices that see that value know not to re-mark those flow labels as the packets come into the network, we might have a way to make flow label more useful for traffic that is otherwise not visible enough to the transporting network to take advantage of this work, without having to worry about preserving the value across multiple networks.

Thanks
Wes George

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of George, Wes E IV [NTK]
Sent: Friday, May 07, 2010 5:03 PM
To: Brian E Carpenter; 6man
Subject: RE: [Fwd: I-D Action:draft-carpenter-6man-flow-update-03.txt]

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Thursday, May 06, 2010 9:11 PM
To: 6man
Subject: [Fwd: I-D Action:draft-carpenter-6man-flow-update-03.txt]

Thanks,
Wes George

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 11 May 2010 07:49:05 +1200

Wes,

On 2010-05-11 02:39, George, Wes E IV [NTK] wrote:
> So, it may make sense to have a method where the device responsible for doing the encapsulation [should | must] insert a random value into the outer header's flow label that is based on some useful data from the inner headers. 

That is what draft-carpenter-flow-ecmp says

    Brian

To the concern about whether it will be random enough, and whether it will conflict with the locally-significant flow labels, if
it is based on the same criteria used for the locally significant flow labels, it's at least mostly safe to assume it'll be
useful, and likely it'll be better than not having it. If we go with something like the previous draft, where the first few bits
should be set using a defined flag value so that devices that see that value know not to re-mark those flow labels as the
packets come into the network, we might have a way to make flow label more useful for traffic that is otherwise not visible
enough to the transporting network to take advantage of this work, without having to worry about preserving the value across
multiple networks.