Subject: Re: [tsvwg] 回复：Takeaways for HPCC++

From: Richard Li <[email-masked]>
Date: Mon, 2 Nov 2020 00:43:32 +0000

> it is has specifically taken into account lessons learnt from protocols such as XCP

Would you mind sharing what lessons exactly you learnt from XCP?

Thanks,

Richard

From: tsvwg <[email-masked]> On Behalf Of Scharf, Michael
Sent: Wednesday, October 28, 2020 10:48 AM
To: Rui, Miao <[email-masked]>; tsvwg <[email-masked]>
Cc: Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>
Subject: Re: [tsvwg] 回复：Takeaways for HPCC++

Hi all,
An editorial comment: ICCRG has published RFC 6077 about ten years ago. Obviously, the document may be a bit dated, but it is has specifically taken into account lessons learnt from protocols such as XCP or Quick-Start. Not all content of RFC 6077 may apply to current data centers. Nonetheless, some of the content in RFC 6077 is quite generic, and it could be useful to better explain in a description of HPC++ how this architecture addresses the challenges.
Best regards
Michael (as co-author of RFC 6077)

From: tsvwg <[email-masked]<mailto:[email-masked]>> On Behalf Of Rui, Miao
Sent: Friday, September 11, 2020 10:53 AM
To: Miao, Rui(缪睿) <[email-masked]<mailto:[email-masked]>>; tsvwg <[email-masked]<mailto:[email-masked]>>
Cc: Barak Gafni <[email-masked]<mailto:[email-masked]>>; tsvwg-chairs <[email-masked]<mailto:[email-masked]>>
Subject: [tsvwg] 回复：Takeaways for HPCC++

Dear WG members,

At IETF 108, we presented HPCC++, a new architecture for congestion control designed for high-speed networks. We have modified the draft according to the feedback from the meeting. The newer version is available at https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-02<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pan-tsvwg-hpccplus-02&data=04%7C01%7Crichard.li%40futurewei.com%7C5f95b2910d2b41a59fe208d87b699b9a%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C1%7C637395040817739023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Q%2FDwV%2Bm73NlD3m%2BmYoOIA0OGUUlgtUYCUFz2uwAC3Qk%3D&reserved=0>

Thanks, and please feel free to let us know if you have more comments.

Thanks,
Rui

------------------------------------------------------------------
发件人：Miao, Rui(缪睿) <[email-masked]<mailto:[email-masked]>>
发送时间：2020年8月2日(星期日) 11:21
收件人：tsvwg <[email-masked]<mailto:[email-masked]>>
抄　送：tsvwg-chairs <[email-masked]<mailto:[email-masked]>>; Wesley Eddy <[email-masked]<mailto:[email-masked]>>; "Pan, Rong" <[email-masked]<mailto:[email-masked]>>; Barak Gafni <[email-masked]<mailto:[email-masked]>>
主　题：Takeaways for HPCC++

Hello TSVWG members,

It was our pleasure to present HPCC++ in the IETF-108 meeting this week. We appreciate your valuable feedback. Our current draft is located at https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-01<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pan-tsvwg-hpccplus-01&data=04%7C01%7Crichard.li%40futurewei.com%7C5f95b2910d2b41a59fe208d87b699b9a%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C1%7C637395040817748978%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=mrAKAuPvwwdWCBghjQDoMYqNftztOQtCmGJivkqwgpE%3D&reserved=0>.
Here, we have summarized a list of takeaways to further improve the draft.

1. HPCC++ is a generic congestion control scheme that works for high-speed networks. It requires to have window limit and ACK self-clocking, which is out of the definition of RoCEv2 but should instead work well for TCP/iWRAP. In addition, HPCC++ conforms to the paradigm of existing TCP congestion control, such as switch marking, receiver feedback, and sender enforcement. The major changes are using inband telemetry as the congestion feedback and it is incremental. In the next revision, we describe clearly how it works for TCP/UDP and related protocols.

2. HPCC++ works for both data centers and the Internet. We will add descriptions and results for the Internet-scale size and latency in the next revision.

Thanks again for the comments. Please feel free to let us know if you have more thoughts.

All the best,
Rui, Rong, Barak

---

From: "Scharf, Michael" <[email-masked]>
Date: Mon, 2 Nov 2020 12:53:00 +0000

RFC 6077 Section 3.1 summarizes challenges for all congestion control schemes that rely on network support known at the time of publicatiob.

In addition to the general issues, RFC 6077 Section 3.1 also includes some specific literature pointers to known issues for XCP. As far as I recall, some of the issues in XCP were fairly fundamental. The corresponding papers are referenced in the RFC.

Before publication of RFC 6077, I have myself worked with XCP simulations and XCP hardware implementations. In my PhD thesis I have compared, amonst others, XCP with a simpler solution developed in the IETF (Quick-Start). My personal findings can be found in Sections 4, 5 and 6 of my Phd thesis:

http://content.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf

Specifically, I ran into issues when a router supporting XCP cannot correctly determine the capacity of a link (see section 6.5.3.3 in the thesis).

RFC 6077 explains why it is not trivial to actually know what link capacity in reality (i.e., outside simulations). Examples for such cases include virtualization, tunneling, brownfield deployments with legacy devices, etc.

I don’t understand HPCC++ well enough to know how this problem is solved in actual deployments. If you want to repeat the experiment I did in my PhD thesis for XCP (cf. Section 6.5.3.3), you would have to run e.g. experiments in which the actual link capacity is either by factor 10 larger or by factor 10 smaller than the capacity assumed in the algorithm (I guess it is the variable B_j in the draft). If a congestion control algorithm shall be robust, IMHO it would have to be able to deal with such situations. According to my own results, XCP did not work well in such cases of capacity over- or under-estimation.

BTW, the I-D „draft-irtf-panrg-what-not-to-do“ also discusses related lessons learnt.

Michael

From: Richard Li <[email-masked]>
Sent: Monday, November 2, 2020 1:44 AM
To: Scharf, Michael <[email-masked]>; Rui, Miao <[email-masked]>; tsvwg <[email-masked]>
Cc: Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>
Subject: RE: [tsvwg] 回复：Takeaways for HPCC++

Thanks,

Richard

From: tsvwg <[email-masked]<mailto:[email-masked]>> On Behalf Of Scharf, Michael
Sent: Wednesday, October 28, 2020 10:48 AM
To: Rui, Miao <[email-masked]<mailto:[email-masked]>>; tsvwg <[email-masked]<mailto:[email-masked]>>
Cc: Barak Gafni <[email-masked]<mailto:[email-masked]>>; tsvwg-chairs <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] 回复：Takeaways for HPCC++

Hi all,

Dear WG members,

Thanks,
Rui

Hello TSVWG members,

All the best,
Rui, Rong, Barak

---

From: "Rui, Miao" <[email-masked]>
Date: Wed, 11 Nov 2020 07:37:00 +0800

Thanks, Michael. I have read RFC 6077 and your thesis. So much fun to read them.

Yes, you are right. HPCC++ belongs to the category that relies on network support. Instead of having a single bit indication like ECN, HPCC++ leverages the newly-invented inband telemetry to get link load information for precise the rate update. So it can be viewed as the enhanced/ultimate version of ECN. However, we view HPCC++ is still a primary mode algorithm because it performs rate updates in each end-system in a distributed manner. We have also evaluated through extensive theoretical proofs, simulations, and testbed experiments that HPCC++ is efficient and robust in various settings. Details can be found at

 https://dl.acm.org/doi/10.1145/3341302.3342085

One interesting finding in reading the Quick-Start approach is that to quickly open up the congestion window is essential in today's high-speed network. However, we found our applications intensively use the persistent TCP connections to skip the hand-shaking delay and often have delays in data transmission after the connection is established (e.g., wait for control signal or wait for other connections). Then we turn to rely on congestion control itself to open up the window. Using inband telemetry HPCC++ knows precisely how much it can increase so it can actually use MIMD to obtain the available bandwidth in just one round trip time. (But it still uses AIMD for fairness convergence after iteratiions.)

We have also compared with XCP/RCP in section 3.3 of our paper.  The main takeaway is that those congestion signals are heuristic combinations of different types, so they have scaling parameters to tune their relative significance. In contrast, the key insight of HPCC++ is to limit the ``inflight bytes'' which is essential to resolve congestion. HPCC++ uses inband telemetry with concrete physical meaning to quantify inflight bytes. So you may find that HPCC++ rarely uses parameters (HPCC++ has only three configuration parameters and none of them are reliability-critical). Another issue is that rxRate used in XCP/RCP has no concrete physical meaning and rxRate and qlen used in XCP/RCP overlap with each other. We show an experiment that using txRate in HPCC++ is more robust than rxRate in Figure 6.

I do not quite get it from reading your thesis where you mention the router cannot determine the correct capacity. Here are my preliminary thoughts and please feel free to correct me if I am wrong:
HPCC++ assumes each router knows its physical link capacity. e.g. a link with 100GE. Are you suggesting the available bandwidth for HPCC++/XCP/RCP is dynamically varied due to administration reasons, or queueing management? Can you please elaborate on the reasons in realistic settings? How could it be 10x in over- or under-estimation?
HPCC++ is devised to assume the link capacity should be given not through estimation. Even if we are targeting a situation with bandwidth variation, my takeaway is that HPCC++ should be modified to be adaptive but the design philosophy is still to precisely estimate the available inflight bytes for rate updates, without doing any sorts of heuristics with parameter tuning.
Yes, the congestion feedback can be delayed during the congestion. But the end-system has the window to limit the total inflight bytes, so the congestion situation cannot getting worse.
Yes, there is one RTT delay in getting congestion feedback. In control theory, the utilization can still converge very quickly (we show in theory and experiments that HPCC++ can converge in only one round trip time.)

It is great to talk with you about those algorithm details. I am happy to hear your thoughts.

Thanks,
Rui
------------------------------------------------------------------
发件人：Scharf, Michael <[email-masked]>
发送时间：2020年11月2日(星期一) 04:53
收件人：Richard Li <[email-masked]>; "Miao, Rui(缪睿)" <[email-masked]>; tsvwg <[email-masked]>
抄　送：Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>
主　题：RE: [tsvwg] 回复：Takeaways for HPCC++

Michael

Thanks,

Richard

Hi all,

From: tsvwg <[email-masked]> On Behalf Of Rui, Miao
Sent: Friday, September 11, 2020 10:53 AM
To: Miao, Rui(缪睿) <[email-masked]>; tsvwg <[email-masked]>
Cc: Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>
Subject: [tsvwg] 回复：Takeaways for HPCC++
Dear WG members,

At IETF 108, we presented HPCC++, a new architecture for congestion control designed for high-speed networks. We have modified the draft according to the feedback from the meeting. The newer version is available at https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-02

Thanks,
Rui

------------------------------------------------------------------
发件人：Miao, Rui(缪睿) <[email-masked]>
发送时间：2020年8月2日(星期日) 11:21
收件人：tsvwg <[email-masked]>
抄　送：tsvwg-chairs <[email-masked]>; Wesley Eddy <[email-masked]>; "Pan, Rong" <[email-masked]>; Barak Gafni <[email-masked]>
主　题：Takeaways for HPCC++

Hello TSVWG members,

It was our pleasure to present HPCC++ in the IETF-108 meeting this week. We appreciate your valuable feedback. Our current draft is located at https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-01.
Here, we have summarized a list of takeaways to further improve the draft. 

All the best,
Rui, Rong, Barak

---

From: "Scharf, Michael" <[email-masked]>
Date: Wed, 11 Nov 2020 09:39:17 +0000

RFC 6077, as well as my thesis, deals with *routers*, which can operate over a many different link layer technologies. As far as I know, there are many link layer technologies that do not have a constant physical link capacity (WLAN , 3G/4G/5G, etc.). A typical reason is adaptive encoding and link-layer retransmissions.

But I’ll give you some food for thought for typical data center networks, in order to explain why I don’t buy *any* use of a “rate” parameter in any router algorithm.

All examples below use routers in data centers, and for none of them I understand how a parameter “capacity” or “rate” would be defined in the router.

1/ For instance, I don’t understand how a L3 switch in a datacenter network using IP routing would know the physical capacity associated to a logical IP interface, say, if the L3 switch has multiple logical IP interfaces. At least the L3 switches I use in my lab can assign IP interfaces to VLANs instead of physical ports. In that case, e.g. with VLANs, there may be no 1-to-1 mapping between the logical IP interface and a physical port. A physical port has a given capacity (say, 100 Gbps Ethernet), but that does not imply anything to IP forwarding/routing, which is controlled by the routing table. For instance, the forwarding/routing may use statistical multiplexing across VLANs. From a IP forwarding/routing perspective, several IP interfaces may then share the physical capacity (and possibly also the buffer). Say, if there are 10 VLANs in the 100 Gbps link, which capacity would be associated to one of the logical IP interfaces? If the answer was 10 Gbps, it would be an underestimation by factor 10 if there is no other traffic on the other VLANs.

2/ And note that data center networks can be more complex, e.g., when they get interconnected. In a datacenter interconnect setting, I really don’t know how a router would know the physical capacity on router-to-router interconnects. As one example, please consider a router that is connected by a 100 Gbps interface to an optical OTN switch. The OTN switch could perhaps use 10x 10 Gbps connections inside the optical transport network with some sort of link bundling (modern optical OTN switches have such capabilities, e.g. using LAGs). Now, let’s assume a fiber failure affects 9 out of these 10 connections. The available bandwidth on the router interconnect then drops to 10 Gbps (i.e., 10%), while the router still uses a physical 100 Gbps interface towards the optical switch. It is not clear to me how the router would know that the capacity has dropped to 10% other than by throughput measurements (except if one assumes technologies such as GMPLS UNI).

3/ I also don’t understand how the router would know any “physical capacity” if a data center interconnect was realized e.g. by multi-point-to-multipoint L2VPNs (e.g., VPLS). In that case, the available capacity in the L2VPN is defined by the SLAs, i.e., commercial contracts. It may be difficult for a router to know the terms of a commercial contract… Also, note that many point-to-multipoint VPNs use in their SLAs a total physical capacity into the VPN, i.e., they specify the sum of capacity to multiple routers in the VPN, but no dedicated capacity for any router-to-router link (“hose-model”). The same applies to L3VPNs.

4/ Finally, I think there is an overall industry trend to use overlay networks in data centers (and beyond) with all sorts of tunnels. So, any solution proposed for such settings will have to be able to cope with those tunnels. That might imply that a cross-layer information exchange is needed between the underlay and the overlay… That is already non-trivial for the ECN bits, and for any more complex information exchange a lot of engineering problems may emerge. And note that tunnels can be encrypted.

As far as I can tell, in research it is easy to setup a simulation model with some links between routers having some well-defined capacity. One can also do this with simple hardware setups. Well, I have done such theoretic “toy” setups in my thesis as well. BUT, as far as I can tell, the reality in engineering can be much more complex. BTW, in my thesis, I already run into a lot of challenges when I tried to add Quick-Start support to the IP forwarding/routing in the Linux kernel… Getting a “rate” parameter inside the Linux IP forwarding code was extremely difficult and only doable for trivial network setups. That is the difference between research and engineering…

Unless all these issues (and there are many more) can be solved, IMHO any algorithm inside a router must be able to work without any a-priori knowledge about any “capacity” towards the next hop in a routing table, because there is no notion of a “physical capacity” in many scenarios, and in many other scenarios the information for a next hop routing table entry could be arbitrarily wrong. A router can of course measure the available bandwidth towards the next hop in the routing table by stats, but that adds a lot of complexity in the router and it would be a heuristic like any end-to-end congestion control.

BTW, if there is an assumption that any IP interface in a router has a queue, this could also result in a lot of engineering questions, as router hardware may not be implemented that way. As far as I know, the buffer management in routers can be very complex, e.g., if H-QoS is used. But this is about router hardware design and I am not an expert for that. https://datatracker.ietf.org/meeting/104/materials/slides-104-wgtlgo-forwarding-plane-realities provides some interesting insights into the realization of modern routers.

So, in a nutshell, IMHO an algorithm for routers cannot and must not use any configurated “rate” parameter. As far as I understand the HPCC++ algorithm so far, the parameter “txRate” must therefore be removed from the algorithm, except if the mentioned challenges are all solved (which may be doable – I just don’t know how). Note that this is not about the algorithm itself, but about how to obtain the required input parameters.

The situation may be different for link-layer switches, in particular, if one assumes a greenfield (“clean-slate”), homogeneous link layer network infrastructure. But, as far as I can tell, there may be significant differences between a switch and a router that uses full IP forwarding/routing.
RFC 6077, as well as my thesis, tries to address the challenges in routers. If you want to propose a network supported congestion control scheme that shall work on routers, you’ll have to detail how a router can be engineered. I may be wrong, but I doubt that this is a low-hanging fruit given the functionality of a modern router.

Michael

From: Rui, Miao <[email-masked]>
Sent: Wednesday, November 11, 2020 12:37 AM
To: Scharf, Michael <[email-masked]>; Richard Li <[email-masked]>; tsvwg <[email-masked]>
Cc: Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>; Liu, Hongqiang(洪强) <[email-masked]>; Pan, Rong <[email-masked]>; Lee, Jeongkeun <[email-masked]>
Subject: 回复：[tsvwg] 回复：Takeaways for HPCC++

I do not quite get it from reading your thesis where you mention the router cannot determine the correct capacity. Here are my preliminary thoughts and please feel free to correct me if I am wrong:
·         HPCC++ assumes each router knows its physical link capacity. e.g. a link with 100GE. Are you suggesting the available bandwidth for HPCC++/XCP/RCP is dynamically varied due to administration reasons, or queueing management? Can you please elaborate on the reasons in realistic settings? How could it be 10x in over- or under-estimation?
·         HPCC++ is devised to assume the link capacity should be given not through estimation. Even if we are targeting a situation with bandwidth variation, my takeaway is that HPCC++ should be modified to be adaptive but the design philosophy is still to precisely estimate the available inflight bytes for rate updates, without doing any sorts of heuristics with parameter tuning.
·         Yes, the congestion feedback can be delayed during the congestion. But the end-system has the window to limit the total inflight bytes, so the congestion situation cannot getting worse.
·         Yes, there is one RTT delay in getting congestion feedback. In control theory, the utilization can still converge very quickly (we show in theory and experiments that HPCC++ can converge in only one round trip time.)

Thanks,
Rui
------------------------------------------------------------------
发件人：Scharf, Michael <[email-masked]<mailto:[email-masked]>>
发送时间：2020年11月2日(星期一) 04:53
收件人：Richard Li <[email-masked]<mailto:[email-masked]>>; "Miao, Rui(缪睿)" <[email-masked]<mailto:[email-masked]>>; tsvwg <[email-masked]<mailto:[email-masked]>>
抄　送：Barak Gafni <[email-masked]<mailto:[email-masked]>>; tsvwg-chairs <[email-masked]<mailto:[email-masked]>>
主　题：RE: [tsvwg] 回复：Takeaways for HPCC++

Michael

From: Richard Li <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 2, 2020 1:44 AM
To: Scharf, Michael <[email-masked]<mailto:[email-masked]>>; Rui, Miao <[email-masked]<mailto:[email-masked]>>; tsvwg <[email-masked]<mailto:[email-masked]>>
Cc: Barak Gafni <[email-masked]<mailto:[email-masked]>>; tsvwg-chairs <[email-masked]<mailto:[email-masked]>>
Subject: RE: [tsvwg] 回复：Takeaways for HPCC++

Thanks,

Richard

Hi all,

Dear WG members,

Thanks,
Rui

Hello TSVWG members,

All the best,
Rui, Rong, Barak

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 11 Nov 2020 08:20:57 -0600

Michael,

Top-posting, to thank you for providing this background. It's very helpful
to me.

Best,

Spencer

On Wed, Nov 11, 2020 at 3:39 AM Scharf, Michael <
[email-masked]> wrote:

> RFC 6077, as well as my thesis, deals with **routers**, which can operate
> over a many different link layer technologies. As far as I know, there are
> many link layer technologies that do not have a constant physical link
> capacity (WLAN , 3G/4G/5G, etc.). A typical reason is adaptive encoding and
> link-layer retransmissions.
>
>
>
> But I’ll give you some food for thought for typical data center networks,
> in order to explain why I don’t buy **any** use of a “rate” parameter in
> any router algorithm.
>
>
>
> All examples below use routers in data centers, and for none of them I
> understand how a parameter “capacity” or “rate” would be defined in the
> router.
>
>
>
> 1/ For instance, I don’t understand how a L3 switch in a datacenter
> network using IP routing would know the physical capacity associated to a
> logical IP interface, say, if the L3 switch has multiple logical IP
> interfaces. At least the L3 switches I use in my lab can assign IP
> interfaces to VLANs instead of physical ports. In that case, e.g. with
> VLANs, there may be no 1-to-1 mapping between the logical IP interface and
> a physical port. A physical port has a given capacity (say, 100 Gbps
> Ethernet), but that does not imply anything to IP forwarding/routing, which
> is controlled by the routing table. For instance, the forwarding/routing
> may use statistical multiplexing across VLANs. From a IP forwarding/routing
> perspective, several IP interfaces may then share the physical capacity
> (and possibly also the buffer). Say, if there are 10 VLANs in the 100 Gbps
> link, which capacity would be associated to one of the logical IP
> interfaces? If the answer was 10 Gbps, it would be an underestimation by
> factor 10 if there is no other traffic on the other VLANs.
>
>
>
> 2/ And note that data center networks can be more complex, e.g., when they
> get interconnected. In a datacenter interconnect setting, I really don’t
> know how a router would know the physical capacity on router-to-router
> interconnects. As one example, please consider a router that is connected
> by a 100 Gbps interface to an optical OTN switch. The OTN switch could
> perhaps use 10x 10 Gbps connections inside the optical transport network
> with some sort of link bundling (modern optical OTN switches have such
> capabilities, e.g. using LAGs). Now, let’s assume a fiber failure affects 9
> out of these 10 connections. The available bandwidth on the router
> interconnect then drops to 10 Gbps (i.e., 10%), while the router still uses
> a physical 100 Gbps interface towards the optical switch. It is not clear
> to me how the router would know that the capacity has dropped to 10% other
> than by throughput measurements (except if one assumes technologies such as
> GMPLS UNI).
>
>
>
> 3/ I also don’t understand how the router would know any “physical
> capacity” if a data center interconnect was realized e.g. by
> multi-point-to-multipoint L2VPNs (e.g., VPLS). In that case, the available
> capacity in the L2VPN is defined by the SLAs, i.e., commercial contracts.
> It may be difficult for a router to know the terms of a commercial
> contract… Also, note that many point-to-multipoint VPNs use in their SLAs a
> total physical capacity into the VPN, i.e., they specify the sum of
> capacity to multiple routers in the VPN, but no dedicated capacity for any
> router-to-router link (“hose-model”). The same applies to L3VPNs.
>
>
>
> 4/ Finally, I think there is an overall industry trend to use overlay
> networks in data centers (and beyond) with all sorts of tunnels. So, any
> solution proposed for such settings will have to be able to cope with those
> tunnels. That might imply that a cross-layer information exchange is needed
> between the underlay and the overlay… That is already non-trivial for the
> ECN bits, and for any more complex information exchange a lot of
> engineering problems may emerge. And note that tunnels can be encrypted.
>
>
>
> As far as I can tell, in research it is easy to setup a simulation model
> with some links between routers having some well-defined capacity. One can
> also do this with simple hardware setups. Well, I have done such theoretic
> “toy” setups in my thesis as well. BUT, as far as I can tell, the reality
> in engineering can be much more complex. BTW, in my thesis, I already run
> into a lot of challenges when I tried to add Quick-Start support to the IP
> forwarding/routing in the Linux kernel… Getting a “rate” parameter inside
> the Linux IP forwarding code was extremely difficult and only doable for
> trivial network setups. That is the difference between research and
> engineering…
>
>
>
> Unless all these issues (and there are many more) can be solved, IMHO any
> algorithm inside a router must be able to work without any a-priori
> knowledge about any “capacity” towards the next hop in a routing table,
> because there is no notion of a “physical capacity” in many scenarios, and
> in many other scenarios the information for a next hop routing table entry
> could be arbitrarily wrong. A router can of course measure the available
> bandwidth towards the next hop in the routing table by stats, but that adds
> a lot of complexity in the router and it would be a heuristic like any
> end-to-end congestion control.
>
>
>
> BTW, if there is an assumption that any IP interface in a router has a
> queue, this could also result in a lot of engineering questions, as router
> hardware may not be implemented that way. As far as I know, the buffer
> management in routers can be very complex, e.g., if H-QoS is used. But this
> is about router hardware design and I am not an expert for that.
> https://datatracker.ietf.org/meeting/104/materials/slides-104-wgtlgo-forwarding-plane-realities
> provides some interesting insights into the realization of modern routers.
>
>
>
> So, in a nutshell, IMHO an algorithm for routers cannot and must not use
> any configurated “rate” parameter. As far as I understand the HPCC++
> algorithm so far, the parameter “txRate” must therefore be removed from the
> algorithm, except if the mentioned challenges are all solved (which may be
> doable – I just don’t know how). Note that this is not about the algorithm
> itself, but about how to obtain the required input parameters.
>
>
>
> The situation may be different for link-layer switches, in particular, if
> one assumes a greenfield (“clean-slate”), homogeneous link layer network
> infrastructure. But, as far as I can tell, there may be significant
> differences between a switch and a router that uses full IP
> forwarding/routing.
>
> RFC 6077, as well as my thesis, tries to address the challenges in
> routers. If you want to propose a network supported congestion control
> scheme that shall work on routers, you’ll have to detail how a router can
> be engineered. I may be wrong, but I doubt that this is a low-hanging fruit
> given the functionality of a modern router.
>
>
>
> Michael
>
>
>
>
>
> *From:* Rui, Miao <[email-masked]>
> *Sent:* Wednesday, November 11, 2020 12:37 AM
> *To:* Scharf, Michael <[email-masked]>; Richard Li <
> [email-masked]>; tsvwg <[email-masked]>
> *Cc:* Barak Gafni <[email-masked]>; tsvwg-chairs <
> [email-masked]>; Liu, Hongqiang(洪强) <[email-masked]>;
> Pan, Rong <[email-masked]>; Lee, Jeongkeun <[email-masked]>
> *Subject:* 回复：[tsvwg] 回复：Takeaways for HPCC++
>
>
>
> Yes, you are right. HPCC++ belongs to the category that relies on network
> support. Instead of having a single bit indication like ECN, HPCC++
> leverages the newly-invented inband telemetry to get link load information
> for precise the rate update. So it can be viewed as the enhanced/ultimate
> version of ECN. However, we view HPCC++ is still a primary mode algorithm
> because it performs rate updates in each end-system in a distributed
> manner. We have also evaluated through extensive theoretical proofs,
> simulations, and testbed experiments that HPCC++ is efficient and robust in
> various settings. Details can be found at
>
>
>
> One interesting finding in reading the Quick-Start approach is that to
> quickly open up the congestion window is essential in today's high-speed
> network. However, we found our applications intensively use the persistent
> TCP connections to skip the hand-shaking delay and often have delays in
> data transmission after the connection is established (e.g., wait for
> control signal or wait for other connections). Then we turn to rely on
> congestion control itself to open up the window. Using inband telemetry
> HPCC++ knows precisely how much it can increase so it can actually use MIMD
> to obtain the available bandwidth in just one round trip time. (But it
> still uses AIMD for fairness convergence after iteratiions.)
>
>
>
> We have also compared with XCP/RCP in section 3.3 of our paper.  The main
> takeaway is that those congestion signals are heuristic combinations of
> different types, so they have scaling parameters to tune their relative
> significance. In contrast, the key insight of HPCC++ is to limit the
> ``inflight bytes'' which is essential to resolve congestion. HPCC++ uses
> inband telemetry with concrete physical meaning to quantify inflight bytes.
> So you may find that HPCC++ rarely uses parameters (HPCC++ has only three
> configuration parameters and none of them are reliability-critical).
> Another issue is that *rxRate* used in XCP/RCP has no concrete physical
> meaning and *rxRate* and *qlen* used in XCP/RCP overlap with each other.
> We show an experiment that using txRate in HPCC++ is more robust than
> rxRate in Figure 6.
>
>
>
> I do not quite get it from reading your thesis where you mention the
> router cannot determine the correct capacity. Here are my preliminary
> thoughts and please feel free to correct me if I am wrong:
>
> ·         HPCC++ assumes each router knows its physical link capacity.
> e.g. a link with 100GE. Are you suggesting the available bandwidth for
> HPCC++/XCP/RCP is dynamically varied due to administration reasons, or
> queueing management? Can you please elaborate on the reasons in realistic
> settings? How could it be 10x in over- or under-estimation?
>
> ·         HPCC++ is devised to assume the link capacity should be given
> not through estimation. Even if we are targeting a situation with bandwidth
> variation, my takeaway is that HPCC++ should be modified to be
> adaptive but the design philosophy is still to precisely estimate the
> available inflight bytes for rate updates, without doing any sorts of
> heuristics with parameter tuning.
>
> ·         Yes, the congestion feedback can be delayed during the
> congestion. But the end-system has the window to limit the total inflight
> bytes, so the congestion situation cannot getting worse.
>
> ·         Yes, there is one RTT delay in getting congestion feedback. In
> control theory, the utilization can still converge very quickly (we show in
> theory and experiments that HPCC++ can converge in only one round trip
> time.)
>
>
>
> 收件人：Richard Li <[email-masked]>; "Miao, Rui(缪睿)" <
> [email-masked]>; tsvwg <[email-masked]>
>
> 抄 送：Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]
> >
>
> RFC 6077 Section 3.1 summarizes challenges for all congestion control
> schemes that rely on network support known at the time of publicatiob.
>
>
>
> In addition to the general issues, RFC 6077 Section 3.1 also includes some
> specific literature pointers to known issues for XCP. As far as I recall,
> some of the issues in XCP were fairly fundamental. The corresponding papers
> are referenced in the RFC.
>
>
>
> Before publication of RFC 6077, I have myself worked with XCP simulations
> and XCP hardware implementations. In my PhD thesis I have compared, amonst
> others, XCP with a simpler solution developed in the IETF (Quick-Start). My
> personal findings can be found in Sections 4, 5 and 6 of my Phd thesis:
>
>
>
>
> Specifically, I ran into issues when a router supporting XCP cannot
> correctly determine the capacity of a link (see section 6.5.3.3 in the
> thesis).
>
>
>
> RFC 6077 explains why it is not trivial to actually know what link
> capacity in reality (i.e., outside simulations). Examples for such cases
> include virtualization, tunneling, brownfield deployments with legacy
> devices, etc.
>
>
>
> I don’t understand HPCC++ well enough to know how this problem is solved
> in actual deployments. If you want to repeat the experiment I did in my PhD
> thesis for XCP (cf. Section 6.5.3.3), you would have to run e.g.
> experiments in which the actual link capacity is either by factor 10 larger
> or by factor 10 smaller than the capacity assumed in the algorithm (I guess
> it is the variable B_j in the draft). If a congestion control algorithm
> shall be robust, IMHO it would have to be able to deal with such
> situations. According to my own results, XCP did not work well in such
> cases of capacity over- or under-estimation.
>
>
>
> *From:* Richard Li <[email-masked]>
> *Sent:* Monday, November 2, 2020 1:44 AM
> *To:* Scharf, Michael <[email-masked]>; Rui, Miao <
> [email-masked]>; tsvwg <[email-masked]>
> *Cc:* Barak Gafni <[email-masked]>; tsvwg-chairs <
> [email-masked]>
> *Subject:* RE: [tsvwg] 回复：Takeaways for HPCC++
>
>
>
> *From:* tsvwg <[email-masked]> *On Behalf Of *Scharf, Michael
> *Sent:* Wednesday, October 28, 2020 10:48 AM
> *To:* Rui, Miao <[email-masked]>; tsvwg <[email-masked]>
> *Cc:* Barak Gafni <[email-masked]>; tsvwg-chairs <
> [email-masked]>
> *Subject:* Re: [tsvwg] 回复：Takeaways for HPCC++
>
>
>
> Hi all,
>
> An editorial comment: ICCRG has published RFC 6077 about ten years ago.
> Obviously, the document may be a bit dated, but it is has specifically
> taken into account lessons learnt from protocols such as XCP or
> Quick-Start. Not all content of RFC 6077 may apply to current data centers.
> Nonetheless, some of the content in RFC 6077 is quite generic, and it could
> be useful to better explain in a description of HPC++ how this architecture
> addresses the challenges.
>
> Best regards
>
> *From:* tsvwg <[email-masked]> *On Behalf Of *Rui, Miao
> *Sent:* Friday, September 11, 2020 10:53 AM
> *To:* Miao, Rui(缪睿) <[email-masked]>; tsvwg <[email-masked]>
> *Cc:* Barak Gafni <[email-masked]>; tsvwg-chairs <
> [email-masked]>
> *Subject:* [tsvwg] 回复：Takeaways for HPCC++
>
>
>
> Dear WG members,
>
>
>
> At IETF 108, we presented HPCC++, a new architecture for congestion
> control designed for high-speed networks. We have modified the draft
> according to the feedback from the meeting. The newer version is available
> at https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-02
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pan-tsvwg-hpccplus-02&data=04%7C01%7Crichard.li%40futurewei.com%7C5f95b2910d2b41a59fe208d87b699b9a%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C1%7C637395040817739023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Q%2FDwV%2Bm73NlD3m%2BmYoOIA0OGUUlgtUYCUFz2uwAC3Qk%3D&reserved=0>
>
>
>
> 抄 送：tsvwg-chairs <[email-masked]>; Wesley Eddy <[email-masked]>;
> "Pan, Rong" <[email-masked]>; Barak Gafni <[email-masked]>
>
> 主 题：Takeaways for HPCC++
>
>
>
>
>
> Hello TSVWG members,
>
>
>
> It was our pleasure to present HPCC++ in the IETF-108 meeting this week.
> We appreciate your valuable feedback. Our current draft is located at
> https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-01
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pan-tsvwg-hpccplus-01&data=04%7C01%7Crichard.li%40futurewei.com%7C5f95b2910d2b41a59fe208d87b699b9a%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C1%7C637395040817748978%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=mrAKAuPvwwdWCBghjQDoMYqNftztOQtCmGJivkqwgpE%3D&reserved=0>
> .
>
> 1. HPCC++ is a generic congestion control scheme that works for high-speed
> networks. It requires to have window limit and ACK self-clocking, which is
> out of the definition of RoCEv2 but should instead work well for
> TCP/iWRAP. In addition, HPCC++ conforms to the paradigm of existing TCP
> congestion control, such as switch marking, receiver feedback, and sender
> enforcement. The major changes are using inband telemetry as the congestion
> feedback and it is incremental. In the next revision, we describe clearly
> how it works for TCP/UDP and related protocols.
>
>
>
> 2. HPCC++ works for both data centers and the Internet. We will add
> descriptions and results for the Internet-scale size and latency in the
> next revision.
>
>
>

---

From: "Scharf, Michael" <[email-masked]>
Date: Wed, 11 Nov 2020 20:52:36 +0000

Inline [ms]

From: Barak Gafni <[email-masked]>
Sent: Wednesday, November 11, 2020 8:53 PM
To: Spencer Dawkins at IETF <[email-masked]>; Scharf, Michael <[email-masked]>
Cc: NBU-Contact-Rui Miao <[email-masked]>; Richard Li <[email-masked]>; tsvwg <[email-masked]>; Lee, Jeongkeun <[email-masked]>; Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>; NBU-Contact-Harry Liu <[email-masked]>
Subject: RE: [tsvwg] 回复： 回复：Takeaways for HPCC++

Michael,
Thank you very much for this thorough analysis. Please see below some comments and questions.

Thanks,
Barak

From: Spencer Dawkins at IETF <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, November 11, 2020 6:21 AM
To: Scharf, Michael <[email-masked]<mailto:[email-masked]>>
Cc: NBU-Contact-Rui Miao <[email-masked]<mailto:[email-masked]>>; Richard Li <[email-masked]<mailto:[email-masked]>>; tsvwg <[email-masked]<mailto:[email-masked]>>; Lee, Jeongkeun <[email-masked]<mailto:[email-masked]>>; Barak Gafni <[email-masked]<mailto:[email-masked]>>; tsvwg-chairs <[email-masked]<mailto:[email-masked]>>; NBU-Contact-Harry Liu <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] 回复： 回复：Takeaways for HPCC++

Michael,

Top-posting, to thank you for providing this background. It's very helpful to me.

Best,

Spencer

On Wed, Nov 11, 2020 at 3:39 AM Scharf, Michael <[email-masked]<mailto:[email-masked]>> wrote:
RFC 6077, as well as my thesis, deals with *routers*, which can operate over a many different link layer technologies. As far as I know, there are many link layer technologies that do not have a constant physical link capacity (WLAN , 3G/4G/5G, etc.). A typical reason is adaptive encoding and link-layer retransmissions.

1/ For instance, I don’t understand how a L3 switch in a datacenter network using IP routing would know the physical capacity associated to a logical IP interface, say, if the L3 switch has multiple logical IP interfaces. At least the L3 switches I use in my lab can assign IP interfaces to VLANs instead of physical ports. In that case, e.g. with VLANs, there may be no 1-to-1 mapping between the logical IP interface and a physical port. A physical port has a given capacity (say, 100 Gbps Ethernet), but that does not imply anything to IP forwarding/routing, which is controlled by the routing table. For instance, the forwarding/routing may use statistical multiplexing across VLANs. From a IP forwarding/routing perspective, several IP interfaces may then share the physical capacity (and possibly also the buffer). Say, if there are 10 VLANs in the 100 Gbps link, which capacity would be associated to one of the logical IP interfaces? If the answer was 10 Gbps, it would be an underestimation by factor 10 if there is no other traffic on the other VLANs.
[[BG]] I agree with the general question on how to tie a router interface into a physical interface. For our case here, i am not sure that it is necessary to answer it directly – as what is suggested is to sample a state of an interface at given point in time. This is irespective to the forwarding logic that may or may not determine the outgoing interface of the particular packet under discussion.
 [ms] At least for an I*E*TF document I would expect that the assumptions about the switches or routers are (a) precisely documented and (b) there is running code. And I would be surprised if this does not directly enter an architectural discussion of how the switch or router can actually meet the assumptions. This is basically what I suggest: Explain how the challenges noted in RFC 6077 are solved (e.g., in running code).
2/ And note that data center networks can be more complex, e.g., when they get interconnected. In a datacenter interconnect setting, I really don’t know how a router would know the physical capacity on router-to-router interconnects. As one example, please consider a router that is connected by a 100 Gbps interface to an optical OTN switch. The OTN switch could perhaps use 10x 10 Gbps connections inside the optical transport network with some sort of link bundling (modern optical OTN switches have such capabilities, e.g. using LAGs). Now, let’s assume a fiber failure affects 9 out of these 10 connections. The available bandwidth on the router interconnect then drops to 10 Gbps (i.e., 10%), while the router still uses a physical 100 Gbps interface towards the optical switch. It is not clear to me how the router would know that the capacity has dropped to 10% other than by throughput measurements (except if one assumes technologies such as GMPLS UNI).

4/ Finally, I think there is an overall industry trend to use overlay networks in data centers (and beyond) with all sorts of tunnels. So, any solution proposed for such settings will have to be able to cope with those tunnels. That might imply that a cross-layer information exchange is needed between the underlay and the overlay… That is already non-trivial for the ECN bits, and for any more complex information exchange a lot of engineering problems may emerge. And note that tunnels can be encrypted.
 [[BG]] I agree with this statement. This topic has already started to be discussed in the context of IOAM in the IETF. The way I see it, HPCC++ relies on an underlying measurement technology that should handle that, hence it is not strictly in the scope of this document to solve, but may be relevant to notice about it.
[ms] The IETF is about running code, i.e., I would expect that the document at least documents one “measurement” technology and outlines how the challenges explained in RFC 6077 are all solved in that running code. Operational and deployment experiences would of course be a big plus, albeit maybe not strictly required.
As far as I can tell, in research it is easy to setup a simulation model with some links between routers having some well-defined capacity. One can also do this with simple hardware setups. Well, I have done such theoretic “toy” setups in my thesis as well. BUT, as far as I can tell, the reality in engineering can be much more complex. BTW, in my thesis, I already run into a lot of challenges when I tried to add Quick-Start support to the IP forwarding/routing in the Linux kernel… Getting a “rate” parameter inside the Linux IP forwarding code was extremely difficult and only doable for trivial network setups. That is the difference between research and engineering…

BTW, if there is an assumption that any IP interface in a router has a queue, this could also result in a lot of engineering questions, as router hardware may not be implemented that way. As far as I know, the buffer management in routers can be very complex, e.g., if H-QoS is used. But this is about router hardware design and I am not an expert for that. https://datatracker.ietf.org/meeting/104/materials/slides-104-wgtlgo-forwarding-plane-realities<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F104%2Fmaterials%2Fslides-104-wgtlgo-forwarding-plane-realities&data=04%7C01%7Cgbarak%40nvidia.com%7C9d9276d316f143e3bbd208d8864d187a%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C1%7C637407012977021611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Gcrwf27uKor%2FMeE%2FZ%2Bk9m6nyhx911F2dKwFfJiw%2FZKo%3D&reserved=0> provides some interesting insights into the realization of modern routers.
[[BG]] How would you consider ECN with the context of this comment?
 [ms] It is well-known that many routers (and probably a lot of switches) don’t support ECN. I don’t know all root causes, but I expect that hardware limitations are one of the challenges. That is why I guess looking at the hardware design implications early could make sense.
So, in a nutshell, IMHO an algorithm for routers cannot and must not use any configurated “rate” parameter. As far as I understand the HPCC++ algorithm so far, the parameter “txRate” must therefore be removed from the algorithm, except if the mentioned challenges are all solved (which may be doable – I just don’t know how). Note that this is not about the algorithm itself, but about how to obtain the required input parameters.
[[BG]] Please see my earlier response
 [ms] If the document shall move forward in the IETF, I ask for details how any rate-related parameter will be determined inside a switch or router, in order to explain how the challenges explained in RFC 6077 are solved in running code.
The situation may be different for link-layer switches, in particular, if one assumes a greenfield (“clean-slate”), homogeneous link layer network infrastructure. But, as far as I can tell, there may be significant differences between a switch and a router that uses full IP forwarding/routing.
RFC 6077, as well as my thesis, tries to address the challenges in routers. If you want to propose a network supported congestion control scheme that shall work on routers, you’ll have to detail how a router can be engineered. I may be wrong, but I doubt that this is a low-hanging fruit given the functionality of a modern router.
[[BG]] In general I would say that at least the devices I am familiar with are rather not only implementing an IP stack (router) or MAC (bridge), rather both. They indeed many times called switches these days. A particular packet that is being process may even go through multiple instances of such types of processing. Having said that, there may be one set or egress queues allocated to a particular phyical interface, and this is where congestion control decisions are being taken rather on, for example, a logical interface between a virtual router to a virtual bridge or to another virtual router within the same device. There are corelations beween the two, and the devices should take care of the overall operation.
 [ms] As far as I know, these are the very basic L3 switch forwarding capabilities. But the layer 3 switches I am familiar with can typically also attach a token bucket shaper or other rate policer to a logical interface, possibly in combination with ACL matching and the like. And on more sophisticated routers, a complex H-QoS configuration with several cascaded schedulers may be part of the packet pipeline, possibly in combination with advanced AQM algorithms and the like. I think a network supported congestion control specification would have to describe how the input parameters (such as “txRate”) are picked in such non-trivial cases and how the algorithm ties into the overall QoS implementation of the device. I am not sure whether this is straightforward. At least I don’t understand how that would work, neither theoretically nor practically.
[ms] In a nutshell, I think that the document needs a (possibly long) dedicated section on how nodes can solve the challenges explained in RFC 6077. Given the known failures of previous attempts (including my own work), just assuming that all the challenges listed in RFC 6077 will be solved by some magic doesn’t sound very useful to me.
Michael

From: Rui, Miao <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, November 11, 2020 12:37 AM
To: Scharf, Michael <[email-masked]<mailto:[email-masked]>>; Richard Li <[email-masked]<mailto:[email-masked]>>; tsvwg <[email-masked]<mailto:[email-masked]>>
Cc: Barak Gafni <[email-masked]<mailto:[email-masked]>>; tsvwg-chairs <[email-masked]<mailto:[email-masked]>>; Liu, Hongqiang(洪强) <[email-masked]<mailto:[email-masked]>>; Pan, Rong <[email-masked]<mailto:[email-masked]>>; Lee, Jeongkeun <[email-masked]<mailto:[email-masked]>>
Subject: 回复：[tsvwg] 回复：Takeaways for HPCC++

 https://dl.acm.org/doi/10.1145/3341302.3342085<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdl.acm.org%2Fdoi%2F10.1145%2F3341302.3342085&data=04%7C01%7Cgbarak%40nvidia.com%7C9d9276d316f143e3bbd208d8864d187a%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C1%7C637407012977021611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=w%2BJONFH3IrAuAAEk4QJ4VFkKZy0MoKdoc7Ye%2FyDMXoM%3D&reserved=0>

I do not quite get it from reading your thesis where you mention the router cannot determine the correct capacity. Here are my preliminary thoughts and please feel free to correct me if I am wrong:
•         HPCC++ assumes each router knows its physical link capacity. e.g. a link with 100GE. Are you suggesting the available bandwidth for HPCC++/XCP/RCP is dynamically varied due to administration reasons, or queueing management? Can you please elaborate on the reasons in realistic settings? How could it be 10x in over- or under-estimation?
•         HPCC++ is devised to assume the link capacity should be given not through estimation. Even if we are targeting a situation with bandwidth variation, my takeaway is that HPCC++ should be modified to be adaptive but the design philosophy is still to precisely estimate the available inflight bytes for rate updates, without doing any sorts of heuristics with parameter tuning.
•         Yes, the congestion feedback can be delayed during the congestion. But the end-system has the window to limit the total inflight bytes, so the congestion situation cannot getting worse.
•         Yes, there is one RTT delay in getting congestion feedback. In control theory, the utilization can still converge very quickly (we show in theory and experiments that HPCC++ can converge in only one round trip time.)

Thanks,
Rui

http://content.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcontent.ikr.uni-stuttgart.de%2FContent%2FPublications%2FArchive%2FSf_Diss_40112.pdf&data=04%7C01%7Cgbarak%40nvidia.com%7C9d9276d316f143e3bbd208d8864d187a%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C1%7C637407012977031609%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Zgz38wsLLA8iX3vLcJG%2FY0eX7HLTMQIMFWB2jetC9RI%3D&reserved=0>

Michael

Thanks,

Richard

Hi all,

Dear WG members,

At IETF 108, we presented HPCC++, a new architecture for congestion control designed for high-speed networks. We have modified the draft according to the feedback from the meeting. The newer version is available at https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-02<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pan-tsvwg-hpccplus-02&data=04%7C01%7Cgbarak%40nvidia.com%7C9d9276d316f143e3bbd208d8864d187a%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C1%7C637407012977031609%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=d4ij3tmuBTLB7EEvRqj3rGJSKRjMhrBpvBvfOtOQ4Qw%3D&reserved=0>

Thanks,
Rui

Hello TSVWG members,

It was our pleasure to present HPCC++ in the IETF-108 meeting this week. We appreciate your valuable feedback. Our current draft is located at https://tools.ietf.org/html/draft-pan-tsvwg-hpccplus-01<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pan-tsvwg-hpccplus-01&data=04%7C01%7Cgbarak%40nvidia.com%7C9d9276d316f143e3bbd208d8864d187a%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C1%7C637407012977041607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=GL0rtscqY31CKQbK6Qk4CXwCaMDeXBhSVHWaz7URJtU%3D&reserved=0>.
Here, we have summarized a list of takeaways to further improve the draft.

All the best,
Rui, Rong, Barak

---

From: Barak Gafni <[email-masked]>
Date: Wed, 11 Nov 2020 19:53:29 +0000

Michael,

Thanks,
Barak

From: Spencer Dawkins at IETF <[email-masked]>
Sent: Wednesday, November 11, 2020 6:21 AM
To: Scharf, Michael <[email-masked]>
Cc: NBU-Contact-Rui Miao <[email-masked]>; Richard Li <[email-masked]>; tsvwg <[email-masked]>; Lee, Jeongkeun <[email-masked]>; Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>; NBU-Contact-Harry Liu <[email-masked]>
Subject: Re: [tsvwg] 回复： 回复：Takeaways for HPCC++

Michael,

Best,

Spencer

Michael

Thanks,
Rui

Michael

Thanks,

Richard

Hi all,

Dear WG members,

Thanks,
Rui

Hello TSVWG members,

All the best,
Rui, Rong, Barak

---

From: "Rui, Miao" <[email-masked]>
Date: Fri, 13 Nov 2020 08:05:19 +0800

XCP is a dual algorithm to allocate ``rate'', but HPCC++ is yet a primal algorithm running in a distributed manner. So the rate used in HPCC++ is actually the total traffic aggregate, which we typically called it physical link utilization or inflight bytes (not individual virtual link). Please check our paper for more details.

So, as an example to ``if there are 10 VLANs in the 100 Gbps link, which capacity would be associated to one of the logical IP interfaces? If the answer was 10 Gbps, it would be an underestimation by factor 10 if there is no other traffic on the other VLANs.'',
my answer is that HPCC++ will get the utilization (the estimate of inflight bytes) of the physical 100Gbps link. So the link capacity is through configuration and there is no need to estimate the rate for each individuate virtual interface. I think similar argument can be held for tunnels. Basically, the main design philosophy of congestion control is to ensure utilization convergence and per-flow fairness along a given physical path (a sequence of switches). If you are talking about the bandwidth allocation for virtual links over the physical link, in my opinion, it is the task of network QoS, by leveraging sorts of mechanisms link congestion control, fair queueing, admission control, etc.

We typically use the term router and switch interchangeable in the datacenter environment. Basically, it uses merchant switching ASICs (not software) for packet routing and forwarding. The majority of shares are taken by Broadcom Tomohawk, Mellanox spectrum, Cisco N9K, and a few others.  Those devices all provide link utilization measures via inband telemetry. In a link bundle case, we will use ECMP to select a link and still get queue occupancy and link utilization for each link. 

I agree with your point that simulation-based results lack the level of fidelity in some sense. This is why in our paper we have implemented HPCC++ in SmartNIC and reconfigure switch hardware and conducted extensive experiments in a real testbed. We have also deployed HPCC++ in our datacenter network and observed promising results. This gives us the incentive to standardize it via IETF so we can reach broader benefits in Internet-scale networks. In the IETF draft we submitted, we have documented the detailed algorithm, code, and implementation based on our experience.

Thanks,
Rui

------------------------------------------------------------------
发件人：Scharf, Michael <[email-masked]>
发送时间：2020年11月11日(星期三) 12:52
收件人：Barak Gafni <[email-masked]>; Spencer Dawkins at IETF <[email-masked]>
抄　送："Miao, Rui(缪睿)" <[email-masked]>; Richard Li <[email-masked]>; tsvwg <[email-masked]>; "Lee, Jeongkeun" <[email-masked]>; Barak Gafni <[email-masked]>; tsvwg-chairs <[email-masked]>; "Liu, Hongqiang(洪强)" <[email-masked]>
主　题：RE: [tsvwg] 回复： 回复：Takeaways for HPCC++

Inline [ms]

Thanks,
Barak

Subject: Re: [tsvwg] 回复：回复：Takeaways for HPCC++

Michael, 

Best,

Spencer

On Wed, Nov 11, 2020 at 3:39 AM Scharf, Michael <[email-masked]> wrote:
RFC 6077, as well as my thesis, deals with *routers*, which can operate over a many different link layer technologies. As far as I know, there are many link layer technologies that do not have a constant physical link capacity (WLAN , 3G/4G/5G, etc.). A typical reason is adaptive encoding and link-layer retransmissions.

Thanks,
Rui

Michael

Thanks,

Richard

Hi all,

Dear WG members,

Thanks,
Rui

Hello TSVWG members,

All the best,
Rui, Rong, Barak