Subject: Re: [tsvwg] WGLC Conclusion: draft-ietf-tsvwg-tunnel-congestion-feedback-03

From: Vincent Roca <[email-masked]>
Date: Thu, 1 Dec 2016 13:37:59 +0100

Hello Xinpeng,

>          Thanks for your comments, please see my reply inline.

You're welcome.
 
> GENERAL COMMENTS
>  
> 1- In Section 4, it is assumed that "all routers in the tunnel support ECN."
>    Isn't it too restrictive? What happens if this is not the case?
>    And 2 paragraphs later, I read: "In case all interior routers support ECN".
>    This is contradictory as no other possibility is considered.
> [Wei] The reason why we assume “all routers in the tunnel support ECN” is that the tunnel congestion level only could be measure based
> on the ratio of CE-marked packet when all routers support ECN. For the contradictory description, I will revise it later.

VR: Okay, I understand. One sentence to justify this assumption could be helpful.

> 2- In Section 4, it is required that measurements be made on "the same span of
>    packets" between ingress and egress nodes. A potential way to do that is
>    suggested in section 5, with cumulative packet counts. However it cannot
>    totally solve the problem IMHO.
>  
>    Let's imagine that a large burst of packets is sent to the tunnel, leading
>    to severe congestion, and that the ingress-to-egress signalling messages are
>    all lost. Later on, only a few packets are sent sporadically and this time
>    the signalling message reaches the egress node. This latter concludes that
>    a certain number of packets have been lost over the **full** period, but it
>    cannot determine if congestion happened recently or not.
>    The period over which the observation is taking place must be small enough
>    and recent enough for the mechanism to work correctly.
>    This is a key point as it means that congestion evaluation (and mitigation)
>    can only take place if signalling messages can be exchanged in a (sufficiently)
>    reliable way. Is there a solution to guaranty this property?
> [Wei] Good question. The possible solution I can imagine now is providing the signaling message with higher transmission priority.

VR: Yes, this could help. In any case, it is essential to have a higher reliability for this signalling traffic.

> 3- Section 6 requires that "congestion management action must be delayed by
>    more than a worst-case global RTT".
>    How is this worst-case global RTT evaluated by the DP (i.e., ingress node or
>    controller)? Since this global RTT is end-to-end, only end hosts can evaluate
>    it. Is it suggested to do some kind of DPI in the ingress node to try to identify
>    request/answer packet pairs in the traffic and therefore measure the current
>    RTT? In that case, the host-to-ingress time is ignored and must be assessed...
>    How to do that? And what about situations where the DP is not on the data 
>    path? Do you have in mind another method?
> [Wei] I think the global RTT mentioned here would be something like an  empirical value, not the value measured through  the traffic by DP.

VR: It can perhaps be empirical, but it should not too far from reality.

> 4- Does the proposed mechanism work in case of recursive uses of tunnels (tunnel
>    in a tunnel)? It's not mentioned IMHO.
> [Wei] The description on this will be added later.

VR: I'm wondering how safe recursive tunnels can be from a congestion control point of view, when
a given from enters a first tunnel and then inside this tunnel, another tunnel, both of them applying DP/AP. 

> 5- Congestion management (actions taken by an AP) is a key aspect and Section 6
>    provides a few hints. I have a few additional comments:
>    - saying that "congestion management needs to avoid the impacts on e2e
>      congestion control" is perhaps inappropriate. I'd rather say that it must
>      not **compromise** e2e congestion control.
> [Wei] This is a question need to be discussed further. Maybe one option is: when  congestion level is severe in tunnel, the network should not compromise e2e congestion control,
> when the congestion level is below a certain threshold, the network  allow the end host to do congestion control before the network take action.
> Besides, for different kind of flow (e.g. TCP flow, UDP flow), based on whether the flow itself will act on network congestion  the network can take different action.

VR:  Why not, but I'm not a CC expert and cannot give any advice from this point of view.

>    - It is said that "Congestion Policing is used in data center to limit...".
>      This sentence suggests that CP is only meaningful in a data center. Is it
>      the case, which would also mean that Circuit Breakers are the solution to
>      use in Internet? Could you clarify in the I-D?
> [Wei] It’s not like what you understanding here. I will clarify it later.

VR: Thanks.

[...]

Best regards,

  Vincent

---

From: "Black, David" <[email-masked]>
Date: Fri, 16 Dec 2016 23:32:04 +0000

Jake,

Thanks for the review, and I have two comments to add - they're inline
and reproduced here to make them easy to find:

> >1. "Faked" ECT, described in section 4, seems like it conflicts with guidance from
> >RFC 6040, and probably should have a more in-depth explanation directly
> >addressing that point and justifying the reasons.
> >
> >relevant quotes from section 7 of RFC 6040:
> >- "The overriding guidance is: Avoid designing alternate ECN tunnelling
> >semantics, if at all possible"
> >- 1. The ECN field of the outer header ought to be cleared to Not-ECT unless it is
> >guaranteed that the corresponding tunnel egress will correctly propagate
> >congestion markings
> >
> >In order to not conflict here, this draft should probably recommend some
> >mechanisms or advice for ensuring that the egress will correctly handle faked
> >ECT, or should have an applicability section that restricts its deployment to
> >environments where there’s some independent guarantee that ingress and
> >egress will be upgraded together. I wasn't clear on which solution was intended.
> [Wei] Yes, the "Faked" ECT seems conflicting with RFC6040. The reason we think
> the "Faked" ECT will work is that we assume the ingress and egress
> are normally controlled under the same network operator, so the upgrade on
> both ingress and egress can be guaranteed. I agree with you that there should
> be some clarification on this solution's applicability in the document.

David> Restricting this to a single operator's network should be clearly stated in
the draft as the scope of applicability as administrative measures are required to
ensure that the tunnel endpoints are associated correctly.  That will also address
the next point on IPFIX security concerns when spanning administrative boundaries.
Both of these points should be noted as potential sources of problems if the tunnel
endpoints are in different networks and/or administrative domains.

----------------------

> >3. It might be nice to explain much earlier in the document that this draft applies
> >to tunnels implementing the rules from section 4 of RFC 6040 for congestion
> >signaling.
> [Wei] ok.
David> Yes, especially as a small modification to those rules is needed:
https://www.ietf.org/mail-archive/web/tsvwg/current/msg14741.html
which would likewise be nice to call out earlier in the draft.

Thanks, --David

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of Weixinpeng (Jackie)
> Sent: Tuesday, November 29, 2016 6:50 AM
> To: Holland, Jake; [email-masked]
> Cc: [email-masked]
> Subject: Re: [tsvwg] WGLC Conclusion: draft-ietf-tsvwg-tunnel-congestion-
> feedback-03
> 
> Hi Jake,
> 	Thanks for your insightful comments, and please see my replies inline.
> 
> Regards,
> -Xinpeng (Jackie)
> 
> >-----Original Message-----
> >From: Holland, Jake [mailto:[email-masked]]
> >Sent: Sunday, November 27, 2016 10:17 AM
> >To: [email-masked]
> >Cc: [email-masked]
> >Subject: Re: [tsvwg] WGLC Conclusion:
> >draft-ietf-tsvwg-tunnel-congestion-feedback-03
> >
> >Here’s the review I promised in the meeting at IETF 97.
> >
> >Overall, I like the idea in this draft a lot and I hope the ideas in it see
> deployment.
> >However, I think there are some problems with the document in its current
> >state.
> >
> >I’m separately sending some editorial comments to the authors and chairs, but
> >this email has the list of content issues I’d like to see addressed before sending
> >it out of the working group.
> >
> >--------
> >
> >Technical Objections:
> >
> >1. "Faked" ECT, described in section 4, seems like it conflicts with guidance from
> >RFC 6040, and probably should have a more in-depth explanation directly
> >addressing that point and justifying the reasons.
> >
> >relevant quotes from section 7 of RFC 6040:
> >- "The overriding guidance is: Avoid designing alternate ECN tunnelling
> >semantics, if at all possible"
> >- 1. The ECN field of the outer header ought to be cleared to Not-ECT unless it is
> >guaranteed that the corresponding tunnel egress will correctly propagate
> >congestion markings
> >
> >In order to not conflict here, this draft should probably recommend some
> >mechanisms or advice for ensuring that the egress will correctly handle faked
> >ECT, or should have an applicability section that restricts its deployment to
> >environments where there’s some independent guarantee that ingress and
> >egress will be upgraded together. I wasn't clear on which solution was intended.
> [Wei] Yes, the "Faked" ECT seems conflicting with RFC6040. The reason we think
> the "Faked" ECT will work is that we assume the ingress and egress
> are normally controlled under the same network operator, so the upgrade on
> both ingress and egress can be guaranteed. I agree with you that there should
> be some clarification on this solution's applicability in the document.

David> Restricting this to a single operator's network should be clearly stated in
the draft as the scope of applicability.  That will also address the next point on
IPFIX security concerns when spanning administrative boundaries.  Both of these
points should be noted as potential sources of problems if the tunnel endpoints
are in different networks and/or administrative domains.

> >
> >
> >2. This draft seems to implicitly require there to be a security association
> >wherever there is an IPFIX connection. If the egress is a different administrative
> >domain than the ingress and/or the controller, it seems this opens some
> >unmentioned security concerns. Like issue #1, this might be a question
> >answerable with an applicability section.
> [Wei] We assume the ingress and egress are in the same administrative domain.
> For the case that egress and ingress belong to
> different domain, there would be a bunch of security issues.
> >
> >
> >3. This draft is much lighter on implementation details than most RFCs I’ve seen
> >that add new global registry entries. I would much prefer to see a section that
> >outlines at least one (preferably more) example algorithms that a controller
> >might use to perform traffic management in response to congestion, with the
> >corresponding example template record that lists the information necessary to
> >support the algorithm. The examples should be sufficient to demonstrate the
> >need for the new IPFIX registry entries.
> [Wei] A good suggestion. I will consider it in the upcoming version.
> >
> >
> >4. I would likewise much prefer to see an explanation of the expected handling
> of
> >the IPFIX connections, especially the expected behavior by the egress and the
> >controller if the template records exchanged are different from what's
> expected.
> >(Or otherwise how to ensure non-harmful fallback behavior if the tunnel
> >endpoints or controller might be incompatible.)
> [Wei] I will add description on this in the upcoming version.
> >
> >
> >5. I think for this draft, a better (but still brief) overview of tunnels is called for
> >than what’s provided in paragraph 3 of the introduction, with references to
> >tunneling protocols that are applicable. Maybe borrow language from RFC 2983
> >sections 1 and/or 2?
> [Wei] Thanks for this advice, I will revise it in the upcoming version.
> >
> >
> >6. I'd like to see a walkthrough of an example of the handling sequence for the
> >IPFIX messaging, even if a full example algorithm is deemed out of scope.
> [Wei] ok.
> >
> >
> >----------
> >
> >This section is not exactly objections, but questions about the doc that I'd like to
> >see explained or reconsidered:
> >
> >
> >1. Why is the direct model a different case from the centralized model with the
> >controller running on the ingress system?  It seems like the descriptions and
> >diagrams could be simplified by defining only one model (the centralized one),
> >but stating that the controller is allowed to be embedded in the ingress system,
> >and if so, that it’s allowed to ensure compatible versioning and secured
> >communication with the ingress by techniques that make use of the shared
> >system.
> [Wei] Ok, good suggestion.
> >
> >
> >2. Why does the IPFIX connection go from ingress to egress to controller, rather
> >than from ingress to controller and separately from egress to controller?
> [Wei] The IPFIX message from ingress to egress is used as an "in-band"
> mechanism to
> convey flow information at the ingress to egress, when egress get the IPFIX
> message it will append  flow information collected
> at egress in the message and send it to the controller/ingress.
> 
> >
> >
> >3. It might be nice to explain much earlier in the document that this draft applies
> >to tunnels implementing the rules from section 4 of RFC 6040 for congestion
> >signaling.
> [Wei] ok.
David> Yes, especially as a small modification to those rules is needed:
https://www.ietf.org/mail-archive/web/tsvwg/current/msg14741.html
which would likewise be nice to call out earlier in the draft.

> >
> >
> >4. In the introduction, the ConEx reference in the introduction was confusing to
> >me. If I'm understanding correctly, it says this is NOT a ConEx protocol, but
> >ConEx does something similar?
> [Wei] Your understanding is right.
> >
> >I would say there are at least 3 options to improve this:
> >a. if this is similar to ConEx but different, add a section that explains why this
> >should NOT be considered a ConEx protocol (what's incompatible between this
> >and ConEx?), or b. if this should be a ConEx protocol, follow the requirements
> >from the list in section 3 of RFC 7713 for what should be detailed when
> >specifying a ConEx protocol, or c. if ConEx this proposal should be considered
> >independently of ConEx and ConEx is irrelevant, just remove the ConEx
> >reference. I think if the intent was to use it as an example, it raises more
> >questions than it answers.
> [Wei] If the ConEx mentioned here seems confusing, I think I should remove it.
> >
> >I read the specs for a few hours, but I couldn't easily tell which was the right
> >option, though I did notice that section 5.4 of RFC7713 says the mechanisms for
> >ConEx shouldn't be flow-specific, but IPFIX (RFC 7011) seems to be a
> >flow-specific carrier of information.
> >
> >(I’m not sure I got this right, but the described behavior seems to be a ConEx
> >protocol, but treating the tunnel as its own end-to-end connection?)
> >
> >Or if the whole purpose of this draft is to sketch an example justification for
> >adding the IPFIX entries, that point should be more explicitly clear, because
> >sections of this seem to be halfway defining a protocol, but not well enough to
> >implement it.
> >
> >
> >5. I think there are some timing issues with trying to infer loss by comparing
> >packet counts at ingress and egress, so I think a discussion of the complications
> >(or at least a reference for further reading) seems like an important point to
> >mention in this draft.
> [Wei] Actually the document has taken the timing issue into consideration, but it
> might
> need more clear description on this.
> >
> >-------
> >
> >Again, the basic idea expressed in this draft seems worthwhile to me and I hope
> >it sees deployment in some form, but it also seems to me this draft still needs
> >some work, and is missing some important parts that are probably necessary.
> >
> >Regards,
> >Jake
> >
> >
> >On 11/23/16, 3:53 PM, "Black, David" <[email-masked]> wrote:
> >
> >    This WGLC has concluded, but more review is needed.
> >
> >    There are two email threads of comments that need attention in a revised
> >draft:
> >    	- Tunnel Congestion Feedback: (non-)update to RFC6040
> >    	- WGLC on Tunnel Congestion Feedback draft - David Black's
> >comments
> >
> >    As noted in the meeting in Seoul, one serious review is not enough for a
> >WGLC - Bob Briscoe and Jake Holland have signed up to do additional reviews.
> >As previously noted, thank you to both - it would be good to see those reviews
> >posted to the list by the end of next week, please.
> >
> >    Thanks, --David
> >
> >    > -----Original Message-----
> >    > From: Black, David
> >    > Sent: Wednesday, October 26, 2016 6:52 PM
> >    > To: [email-masked]
> >    > Cc: Black, David
> >    > Subject: WGLC: draft-ietf-tsvwg-tunnel-congestion-feedback-03
> >    >
> >    > This email announces a TSVWG Working Group Last Call (WGLC) on:
> >    >
> >    >                        Tunnel Congestion Feedback
> >    >              draft-ietf-tsvwg-tunnel-congestion-feedback-03
> >    >
> >https://datatracker.ietf.org/doc/draft-ietf-tsvwg-tunnel-congestion-feedback/
> >    >
> >    > Due to the upcoming Seoul meetings, this WGLC will last for about 2.5
> >weeks
> >    > weeks, ending at midnight Korea Time on Sunday, November 13, in order
> >to
> >    > enable discussion of any issues at TSVWG sessions in Seoul later that
> >week.
> >    >
> >    > Comments should be sent to the [email-masked] list, although purely
> >    > editorial comments may be sent directly to the authors at
> >    > [email-masked].  Please cc: the
> >    > WG chairs at [email-masked]  if you would like the chairs to track
> >    > such editorial comments as part of the WGLC process.
> >    >
> >    > There are two IPR disclosures that relate to this draft:
> >    >
> >https://datatracker.ietf.org/ipr/search/?submit=draft&id=draft-ietf-tsvwg-
> >    > tunnel-congestion-feedback
> >    >
> >    > I will also kick off WGLC by announcing a proposed resolution of the ECN
> >    > codepoint issue for this draft - that issue involves which ECT codepoint(s)
> >    > to use for the "faked" ECN in the outer header of a packet whose inner
> >    > header does not enable ECN.
> >    >
> >    > With some gentle guidance from the ADs, the proposed resolution is to
> >use
> >    > only ECT(0), and state that ECT(1) MUST NOT be used for a couple of
> >reasons:
> >    >
> >    > 1) Tunnel Congestion Feedback (TCF) needs only one ECT codepoint,
> >hence
> >    > RFC 3168 indicates that ECT(0) SHOULD be used for TCF.  This
> >"SHOULD" is
> >    > a general recommendation for all protocols that use only one ECT
> >codepoint.
> >    >
> >    > 2) There's an IETF intent to use ECT(1) with L4S, and TCF interacts badly
> >with
> >    > L4S to the point of being destructive.   This interaction arises within a
> >TCF tunnel
> >    > that uses "faked" ECN, because L4S marks ECT(1) packets as having
> >experienced
> >    > congestion (CE mark) with much higher probability than drop-equivalence,
> >but
> >    > the TCF tunnel egress treats all CE marks as drop equivalent and drops
> >the
> >    > packets.  The resulting sender congestion response (backoff) to the
> >drops
> >    > will be far more severe than it should be and avoiding that response is
> >why
> >    > ECT(1) MUST NOT be used for TCF.  This "MUST NOT" is specific to TCF
> >due
> >    > to the interaction with L4S.
> >    >
> >    > WGLC comments are welcome on this proposed resolution.  I'll work out
> >    > actual text with the TCF draft authors with input from interested parties
> >    > (e.g., Bob Briscoe), as the above rationale does need to be captured in
> >the
> >    > draft.
> >    >
> >    > Thanks, --David
> >    > --------------------------------------------------------
> >    > David L. Black, Distinguished Engineer
> >    > Dell EMC, 176 South St., Hopkinton, MA  01748
> >    > +1 (508) 293-7953     Cell: +1 (978) 394-7754
> >    > [email-masked]  <=== NEW ===
> >    > --------------------------------------------------------
> >
> >

---

From: "Black, David" <[email-masked]>
Date: Fri, 16 Dec 2016 23:42:30 +0000

Vincent - thank you very much for this review.

Xinpeng - While I haven’t seen Bob’s review (hint to Bob), we do now have two more solid reviews of the draft.  Please go ahead and prepare a revised version of the draft that addresses the concerns raised and makes the editorial changes that you think appropriate.   I’m somewhat concerned about Vincent’s first point, as I was under the impression that the scope of applicability of this draft had been expanded/generalized to include some nodes within the tunnel that do not support ECN (although one would prefer that the likely bottleneck nodes support ECN).

Please post that revised version of the draft for discussion (e.g., of whether the concerns have been satisfactorily addressed), and plan on at least one more version before the request for RFC publication of this draft is submitted to our AD and the IESG.

Thanks, --David

From: Vincent Roca [mailto:[email-masked]]
Sent: Thursday, December 01, 2016 7:38 AM
To: Weixinpeng (Jackie)
Cc: Vincent Roca; [email-masked]; Black, David
Subject: Re: [tsvwg] WGLC Conclusion: draft-ietf-tsvwg-tunnel-congestion-feedback-03

Hello Xinpeng,

You're welcome.

GENERAL COMMENTS

VR: Thanks.

[...]

Best regards,

  Vincent