Subject: Comments on draft-fz-6man-ipv6-alt-mark-09

From: Tom Herbert <[email-masked]>
Date: Mon, 18 May 2020 14:47:09 -0700

>From the draft: "In case of Destination Option Header carrying
Alternate Marking bits, it is not processed, inserted, or deleted by
any node along the path until the packet reaches the destination
node."

The term "destination node" is a bit ambiguous wrt Destination Options
before the routing header (I realize this wording is probably from
RFC8200). In that case, any visited destination in the route list can
process the option. Might make sense to clarify since the proper use
of the Destination Option that are inspected by intermediate nodes is
mentioned in the context of routing headers.

>From the draft; "The only motivation for the hop by hop usage of
Destination Options can be for compatibility reasons but in general it
is not recommended."

That also would be a nonconformant with RFC8200. IMO this doesn't need
to be suggested.

>From the draft: "Another application that can be mentioned is SRv6..."
This is true of any routing header type, not just SRH. I don't see the
need to explicitly focus on SRv6 or SRH here.

It's unclear to me how the FlowMonID is set, how it's used, and why
the Flow Label isn't sufficient for this purpose. For instance, from
the draft: "The FlowMon identifier field is to uniquely identify a
monitored flow within the measurement domain." That would seem to mean
that monitored flow needs a unique ID assignment. How does this work?
It seems like this will require a pretty complex and stateful control
plane protocol. Also, 20 bit identifiers would seem to be on the light
side for scaling in even moderately large networks.

Why not just identify a measured flow by the tuple of (SrcAddr,
DstAddr, Flow Label). This guarantees a unique tuple per measured
flow. The 3-tuple can easily be hashed 32 or 64 bits to generate a
single, relatively unique identifier. The draft raises the concern
that flow labels might change, however I believe that argument has
been overblown as a concern. Note per RFC 6437:

"A forwarding node MUST either leave a non-zero flow label value
unchanged or change it only for compelling operational security
reasons as described in Section 6.1."

So flow labels really aren't supposed to change in flight. We also
learned the lesson at the source host not to change flow labels during
the lifetime of a flow as that can cause problems especially when the
flow label is used for ECMP and there is a stateful firewall in the
path (there was a bug in Linux we fixed where the flow label was not
retained when a TCP connection entered CLOSE_WAIT and so firewalls
were missing the sent FIN since it was being routed different). AFAIK,
Linux servers will not send consistent flow labels for the like of the
connection (at least by default).

I would note, that if the FlowMonID could be eliminated that the
option data would just be a single byte.

Tom

---

From: Tianran Zhou <[email-masked]>
Date: Tue, 19 May 2020 02:32:28 +0000

Hi Tom,

Thank you very much for your support and this detailed review comments.
Please see in line.

Cheers,
Tianran

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Tuesday, May 19, 2020 5:47 AM
To: 6MAN <[email-masked]>
Subject: Comments on draft-fz-6man-ipv6-alt-mark-09

>From the draft: "In case of Destination Option Header carrying
Alternate Marking bits, it is not processed, inserted, or deleted by any node along the path until the packet reaches the destination node."

The term "destination node" is a bit ambiguous wrt Destination Options before the routing header (I realize this wording is probably from RFC8200). In that case, any visited destination in the route list can process the option. Might make sense to clarify since the proper use of the Destination Option that are inspected by intermediate nodes is mentioned in the context of routing headers.

ZTR> So do you mean the case DOH+RH for the intermediate node? In my opinion, it's also the destination node, since the destination address reached. And that's why the DOH applied.  If destination node is not clear, how about we say the end-point? Or what's your suggestion?

>From the draft; "The only motivation for the hop by hop usage of
Destination Options can be for compatibility reasons but in general it is not recommended."

That also would be a nonconformant with RFC8200. IMO this doesn't need to be suggested.

ZTR> It's OK to remove this suggestion. I am just not sure if it's not conformant with RFC8200 or the RFC8200 is not clear on the word "processed". I do not know if "process" really includes the "inspect". Because this behavior cannot be controlled from the protocol end, IMO. I see this kind of usage, e.g. in firewall.

>From the draft: "Another application that can be mentioned is SRv6..."
This is true of any routing header type, not just SRH. I don't see the need to explicitly focus on SRv6 or SRH here.

ZTR> I think this is a good suggestion. We can generalize the description to all the RH cases.

It's unclear to me how the FlowMonID is set, how it's used, and why the Flow Label isn't sufficient for this purpose. For instance, from the draft: "The FlowMon identifier field is to uniquely identify a monitored flow within the measurement domain." That would seem to mean that monitored flow needs a unique ID assignment. How does this work?
It seems like this will require a pretty complex and stateful control plane protocol. Also, 20 bit identifiers would seem to be on the light side for scaling in even moderately large networks.

ZTR> As said in section 5.3, it can be assigned by the central controller or algorithmically generated by the source node. For large networks and algorithmically generation case, 20 bits is not enough, which may require extended FlowMonID. On the other hand, device cannot set up a very large flow table and use a lot of counters to measure large amount of flows, because of the resource limitation. I.e., may not need so many IDs.

Why not just identify a measured flow by the tuple of (SrcAddr, DstAddr, Flow Label). This guarantees a unique tuple per measured flow. The 3-tuple can easily be hashed 32 or 64 bits to generate a single, relatively unique identifier. The draft raises the concern that flow labels might change, however I believe that argument has been overblown as a concern. Note per RFC 6437:

ZTR> We raised this concern because we see the cases in real deployment. Since RFC 6437 has normalized this behavior, let's remove this concern. :-)

"A forwarding node MUST either leave a non-zero flow label value unchanged or change it only for compelling operational security reasons as described in Section 6.1."

So flow labels really aren't supposed to change in flight. We also learned the lesson at the source host not to change flow labels during the lifetime of a flow as that can cause problems especially when the flow label is used for ECMP and there is a stateful firewall in the path (there was a bug in Linux we fixed where the flow label was not retained when a TCP connection entered CLOSE_WAIT and so firewalls were missing the sent FIN since it was being routed different). AFAIK, Linux servers will not send consistent flow labels for the like of the connection (at least by default).

ZTR> The context of the note in section 2 is about encoding the alternate marking bit in flow label.  As you said  "the source host not to change flow labels during the lifetime". That's one reason that we cannot use flow label. The alternate marking method requires the marking bit change to achieve the measurement.  The reason for using the FlowMonID for flow indication is described in section 5.3. We think it's better than 3 tuples. 

I would note, that if the FlowMonID could be eliminated that the option data would just be a single byte.

Tom

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 19 May 2020 14:39:27 +1200

Tom,

> the Flow Label isn't sufficient for this purpose.

That was proposed by the authors and discarded a long time ago, because
the usage was incompatible with RFC6437. I can't remember the details
without looking up long-expired drafts and old email. But generally:
you can't encode semantics in conformant flow labels, because they are
pseudo-random.

Regards
   Brian

---

From: Tom Herbert <[email-masked]>
Date: Tue, 19 May 2020 08:00:28 -0700

> you can't encode semantics in conformant flow labels, because they are
Brian,

AFAICT the FlowMonID is just another type of flow identifier in the
packet. For the purposes of correlating flows in the network I believe
it is redundant to the flow label and would otherwise add a lot of
complexity to implement. I think what you might be referring to are
earlier proposals to put the loss signal bits in the flow label and
not use an EH-- that idea is indeed infeasible at this point. IMO, we
should be able to eliminate FlowMonID from the option, but still need
the two signaling bits in the option hence the alt marking option
would contain one byte of data.

Tom

---

From: Giuseppe Fioccola <[email-masked]>
Date: Tue, 19 May 2020 15:48:18 +0000

Hi Tom,
Thanks for your support and for your comments.
I understand your point, but, in this regard, we should consider the following paragraph from RFC6437:
   
   "There is no way to verify whether a flow label has been modified en
   route or whether it belongs to a uniform distribution.  Therefore, no
   Internet-wide mechanism can depend mathematically on unmodified and
   uniformly distributed flow labels; they have a "best effort" quality.
   Implementers should be aware that the flow label is an unprotected
   field that could have been accidentally or intentionally changed en
   route (see Section 6)"

This does not suggest to rely fully on the Flow Label.
Maybe we can mention in the next revision of the draft the possibility to use the Flow Label for identification, but we should also highlight this point.
In addition, as listed in Section 5.3 of draft-fz-6man-ipv6-alt-mark-09, FlowMonID adds additional advantages in terms of scalability, flexible flow definition granularity, counters handling simplification...

Regards,

Giuseppe

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Tuesday, May 19, 2020 5:00 PM
To: Brian E Carpenter <[email-masked]>
Cc: 6MAN <[email-masked]>
Subject: Re: Comments on draft-fz-6man-ipv6-alt-mark-09

On Mon, May 18, 2020 at 7:39 PM Brian E Carpenter <[email-masked]> wrote:
> That was proposed by the authors and discarded a long time ago, 
> because the usage was incompatible with RFC6437. I can't remember the 
> details without looking up long-expired drafts and old email. But generally:
Brian,

AFAICT the FlowMonID is just another type of flow identifier in the packet. For the purposes of correlating flows in the network I believe it is redundant to the flow label and would otherwise add a lot of complexity to implement. I think what you might be referring to are earlier proposals to put the loss signal bits in the flow label and not use an EH-- that idea is indeed infeasible at this point. IMO, we should be able to eliminate FlowMonID from the option, but still need the two signaling bits in the option hence the alt marking option would contain one byte of data.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Tue, 19 May 2020 09:32:37 -0700

>    field that could have been accidentally or intentionally changed en

Hi Giuseppe,

Unless AH is present (the common case) the whole IP header and
extension headers, including any HBH and Destination Options, are also
unprotected and best effort delivery. We even have to consider that IP
addresses can change (for instance NAT can change addresses in the
middle of UDP flow lifetime).

> In addition, as listed in Section 5.3 of draft-fz-6man-ipv6-alt-mark-09, FlowMonID adds additional advantages in terms of scalability, flexible flow definition granularity, counters handling simpli…

I don't see any scalability for a 20 bit identifier that is supposed
to be unique throughout the network. As I mentioned this can simply be
combined with the source and destination IP addresses to form a
3-tuple. The 3-tuple can then be hashed to produce a single most
unique flow identifier that is suitable lookups. In this manner the 20
bit field is sufficient and nodes can autonomously set it without any
synchronization. This is how we commonly deal with the flow label
today and other cases of flow classification using a flow hash.

Even if the FlowMonID remains, the likely thing a host implementation
would do is just to set the FlowMonID identically to the flow label.
Do you see any reason why this won't work?

Tom

---

From: Giuseppe Fioccola <[email-masked]>
Date: Wed, 20 May 2020 07:21:00 +0000

Hi Tom,
Please find my answers inline tagged as [GF]

Regards,

Giuseppe

-----Original Message-----
From: Tom Herbert [mailto:[email-masked]] 
Sent: Tuesday, May 19, 2020 6:33 PM
To: Giuseppe Fioccola <[email-masked]>
Cc: Brian E Carpenter <[email-masked]>; 6MAN <[email-masked]>
Subject: Re: Comments on draft-fz-6man-ipv6-alt-mark-09

Hi Giuseppe,

Unless AH is present (the common case) the whole IP header and extension headers, including any HBH and Destination Options, are also unprotected and best effort delivery. We even have to consider that IP addresses can change (for instance NAT can change addresses in the middle of UDP flow lifetime).

> In addition, as listed in Section 5.3 of draft-fz-6man-ipv6-alt-mark-09, FlowMonID adds additional advantages in terms of scalability, flexible flow definition granularity, counters handling simpli…

I don't see any scalability for a 20 bit identifier that is supposed to be unique throughout the network. As I mentioned this can simply be combined with the source and destination IP addresses to form a 3-tuple. The 3-tuple can then be hashed to produce a single most unique flow identifier that is suitable lookups. In this manner the 20 bit field is sufficient and nodes can autonomously set it without any synchronization. This is how we commonly deal with the flow label today and other cases of flow classification using a flow hash.

[GF]: I think we can mention the possibility to use the 3-tuple for identification: Flow Label with source and destination addresses. In addition, we could also consider the hashed 3-tuple to set the FlowMonID algorithmically generated by the source node. 
The only issues with the 3-tuple are:
- in case the IP addresses or Flow Label change en route.
- it allows only layer3/layer4 granularity. We may not have flexible flow identification in order to enable personalized selection rules or multipoint paths monitoring (e.g. draft-ietf-ippm-multipoint-alt-mark)
For this reason we suggest to keep all the possibilities: 3-tuple and FlowMonID.

Even if the FlowMonID remains, the likely thing a host implementation would do is just to set the FlowMonID identically to the flow label.
Do you see any reason why this won't work?

[GF]: We did not specify the way to generate the FlowMonID, we just think that the FlowMonID can identify flows easily and flexibly. So we can surely mention that FlowMonID can be set identically to the Flow Label.

Tom

> > > It's unclear to me how the FlowMonID is set, how it's used, and 
> > > why the Flow Label isn't sufficient for this purpose.
> >
> > because the usage was incompatible with RFC6437. I can't remember 
> > the details without looking up long-expired drafts and old email. But generally:
> > you can't encode semantics in conformant flow labels, because they 
> > are pseudo-random.
> >
> Brian,
>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 20 May 2020 15:31:34 -0700

> - in case the IP addresses or Flow Label change en route.

Giuseppe,

Again, they're not really supposed to change enroute. IP addresses can
only change due to NAT and flow labels can only be changed for a
strong security reason. Given that the marking is only meaningful in a
limited domain which wouldn't have NAT and there is no security issue
that would necessitate rewriting flow labels, for the purposes of this
proposal they can't change in flight.

> - it allows only layer3/layer4 granularity. We may not have flexible flow identification in order to enable personalized selection rules or multipoint paths monitoring (e.g. draft-ietf-ippm-multipo…

The flow label does interact with ECMP, so there may be cases where
they need to be different I suppose.

> For this reason we suggest to keep all the possibilities: 3-tuple and FlowMonID.
I suggest not keep all the possibilities open, pick one that should be
implemented. The cost of sending the FlowMonID is negligible (the cost
of the extension header has already been assumed), to me it's more of
a question of whether it's sufficient for uniquely identifying flows.

> [GF]: We did not specify the way to generate the FlowMonID, we just think that the FlowMonID can identify flows easily and flexibly. So we can surely mention that FlowMonID can be set identically t…

The draft mentions two ways to set the FlowMonID: "The FlowMonID can
be uniformly assigned by the central controller or algorithmically
generated by the source node.".

The use of a centralized control for this is a hard problem. The
protocol and APIs that hosts use to interact with the controller have
to be considered, as do various failure modes, redundancy model,
coherency, etc. If this is suggested, I think there should be some
detail and requirements or reference to a protocol specification for
it.

As the draft states for the automatic generation of flow identifiers
cannot guarantee uniqueness. In this case, we want a mechanism that
provides uniqueness with high probability. The probability of
identifier uniqueness correlates to the amount of entropy of the
inputs. For instance, if the 20 bit FlowMonID is set independently and
pseudo randomly by each node without any additional input entropy,
there is a 50% chance of collision is just 1207 flows. If there were a
32 bit hash with at least 32 bits of input entropy the 50% threshold
jumps to 77,264 flows, for 64 bit hash it's 5.06 billion flows. So as
defined FlowMonID really isn't sufficient for uniqueness. To get more
entropy it can either be combined with other identifying flow
information in a packet (like we do when hashing IP addresses and flow
label) or the FlowMonID size could be increased (presumably the sender
would set the FlowMonID to some sort of pseudo hash with at least 32
bits of entropy for the connection anyway).

One other question, does flow identification need to be symmetric. For
instance, do we need to be able to match packets sent in either
direction of a TCP connection as being part of the same flow?

Tom

Tom

---

From: Giuseppe Fioccola <[email-masked]>
Date: Thu, 21 May 2020 10:03:34 +0000

Hi Tom,
Thanks a lot for the useful inputs.
My answers inline tagged as [GF].

Regards,

Giuseppe

-----Original Message-----
From: Tom Herbert [mailto:[email-masked]] 
Sent: giovedì 21 maggio 2020 00:32
To: Giuseppe Fioccola <[email-masked]>
Cc: Brian E Carpenter <[email-masked]>; 6MAN <[email-masked]>
Subject: Re: Comments on draft-fz-6man-ipv6-alt-mark-09

Giuseppe,

Again, they're not really supposed to change enroute. IP addresses can only change due to NAT and flow labels can only be changed for a strong security reason. Given that the marking is only meaningful in a limited domain which wouldn't have NAT and there is no security issue that would necessitate rewriting flow labels, for the purposes of this proposal they can't change in flight.

[GF]: Yes, agree, it is quite unlikely to happen but it is something to keep in mind.

> - it allows only layer3/layer4 granularity. We may not have flexible 
> flow identification in order to enable personalized selection rules or 
> multipoint paths monitoring (e.g. draft-ietf-ippm-multipoint-alt-mark)

The flow label does interact with ECMP, so there may be cases where they need to be different I suppose.

> For this reason we suggest to keep all the possibilities: 3-tuple and FlowMonID.
I suggest not keep all the possibilities open, pick one that should be implemented. The cost of sending the FlowMonID is negligible (the cost of the extension header has already been assumed), to me it's more of a question of whether it's sufficient for uniquely identifying flows.

[GF]: I understand your point. The two possibilities I have in mind are the centralized control and the auto generation of the FlowMonID. With regards to the auto set, the idea to use the FlowMonID together with the 3 tuple is a possible way to ensure the uniqueness.

> [GF]: We did not specify the way to generate the FlowMonID, we just think that the FlowMonID can identify flows easily and flexibly. So we can surely mention that FlowMonID can be set identically t…

The draft mentions two ways to set the FlowMonID: "The FlowMonID can be uniformly assigned by the central controller or algorithmically generated by the source node.".

The use of a centralized control for this is a hard problem. The protocol and APIs that hosts use to interact with the controller have to be considered, as do various failure modes, redundancy model, coherency, etc. If this is suggested, I think there should be some detail and requirements or reference to a protocol specification for it.

[GF]: If we want to achieve the flexible monitoring like Multipoint Alt Mark, the centralized control is required to zoom in and zoom out. Of course, this requires interaction between controller and the nodes, but we can reuse and extend some existing protocol. Anyway we can further explore this part in a later version after the draft is adopted.

As the draft states for the automatic generation of flow identifiers cannot guarantee uniqueness. In this case, we want a mechanism that provides uniqueness with high probability. The probability of identifier uniqueness correlates to the amount of entropy of the inputs. For instance, if the 20 bit FlowMonID is set independently and pseudo randomly by each node without any additional input entropy, there is a 50% chance of collision is just 1207 flows. If there were a
32 bit hash with at least 32 bits of input entropy the 50% threshold jumps to 77,264 flows, for 64 bit hash it's 5.06 billion flows. So as defined FlowMonID really isn't sufficient for uniqueness. To get more entropy it can either be combined with other identifying flow information in a packet (like we do when hashing IP addresses and flow
label) or the FlowMonID size could be increased (presumably the sender would set the FlowMonID to some sort of pseudo hash with at least 32 bits of entropy for the connection anyway).

[GF]: Yes, fully agree on that. Note that draft-ietf-anima-grasp-15 describes a similar problem and is proposing a pseudo-random 32 bit Session ID that is generated by the initiator. However, because of the finite probability that two nodes might generate the same Session ID value, the receiving node also tags it with the initiator's IP address to allow disambiguation. In a similar way, FlowMonID can be combined with other identifying flow information in a packet (IP addresses together and Flow Label) or the FlowMonID size can also be increased by using the reserved bits.

One other question, does flow identification need to be symmetric. For instance, do we need to be able to match packets sent in either direction of a TCP connection as being part of the same flow?

[GF]: Since we consider the IP layer monitoring, we can focus on unidirectional flow and not bidirectional.

Tom

Tom

> > IETF IPv6 working group mailing list [email-masked] Administrative 
> > Requests: https://www.ietf.org/mailman/listinfo/ipv6