Subject: [iccrg] New draft submitted for draft-pan-tsvwg-hpccplus-02.txt

From: "Rui, Miao" <[email-masked]>
Date: Tue, 15 Dec 2020 08:11:00 +0800

Hello ICCRG members,

Alibaba, Intel, and Mellanox have worked on an INT-based High Precision Congestion Control algorithm: HPCC++. We have posted an initial draft that can be found at
https://www.ietf.org/id/draft-pan-tsvwg-hpccplus-02.txt

The key design choice of HPCC++ is to use inband telemetry to provide fine-grained load information, such as queue size and accumulated tx traffic to compute precise flow rates. This has two major benefits:
1. HPCC++ can quickly converge to proper flow rates to highly utilize bandwidth while avoiding congestion;
2. HPCC++ can consistently maintain a close-to-zero queue for low latency.

We would love to hear your comments and feedback.

Best regards,

Rui Miao

---

From: Michael Welzl <[email-masked]>
Date: Tue, 15 Dec 2020 08:24:06 +0100

Hi !

In this world of research, I do want to comment that the signaling protocol looks extremely similar to what I came up with in my Ph.D. thesis. This paper from 2005 describes it:
Michael Welzl: "Router Aided Congestion Avoidance with Scalable Performance Signalling", KiVS 2005 (Kommunikation in verteilten Systemen), Kaiserslautern, Germany, 28 February - 2 March 2005, Springer-Verlag. https://folk.universitetetioslo.no/michawe/research/publications/kivs2005.pdf
I made this overview page in 2006, still alive:  https://folk.universitetetioslo.no/michawe/research/projects/ptp/index.html

Back then, everyone told me that this is completely undeployable  :-)   but this was before data centers became the big thing they are today, and before there were network cards supporting this kind of thing.

Indeed, such mechanisms are simply no fit for the Internet’s end-to-end congestion control - but they should be deployable on path segments, with PEPs at the edge, like here:  https://www.isi.edu/isi-xcp/docs/kapoor-pep-gi2005.final.pdf

Cheers,
Michael

> On Dec 15, 2020, at 1:11 AM, Rui, Miao <[email-masked]> wrote:
> 
> Hello ICCRG members,
> 
> https://www.ietf.org/id/draft-pan-tsvwg-hpccplus-02.txt <https://www.ietf.org/id/draft-pan-tsvwg-hpccplus-02.txt>
> 
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 15 Dec 2020 14:35:28 -0800

Interesting work!

It'd be good to know more precise requirements on INT to help both vendor
supports (beside MLX) and CC evaluation

For example

qlen         | Telemetry info: link j queue length

qlen == instant qlen snapshot at packet ingress or egress, on a
per-port-per-queue basis, or some windowed-avg / aggregate etc.

On Mon, Dec 14, 2020 at 4:11 PM Rui, Miao <[email-masked]> wrote:

> Hello ICCRG members,
>
> Alibaba, Intel, and Mellanox have worked on an INT-based High Precision
> Congestion Control algorithm: HPCC++. We have posted an initial draft that
> can be found at
> The key design choice of HPCC++ is to use inband telemetry to provide
> fine-grained load information, such as queue size and accumulated tx
> traffic to compute precise flow rates. This has two major benefits:

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 15 Dec 2020 15:16:18 -0800

On Tue, Dec 15, 2020 at 3:03 PM Barak Gafni <[email-masked]> wrote:

> Hi,
>
> Thanks for the interest in this work. For your question, at this point the
> draft has been kept open in terms of what is the exact inband telemetry
> technology to be used in order to implement the algorithm. The idea was to
> enable a variety of implementations. With that, one option we are focusing
> on is IOAM which is under work at IPPM WG, and has a data draft specifying
> formats for the communication of these metrics. Alongside this main data
> draft, there is a another draft in its initial work that adds few more
> fields, which may be used for HPCC++.
>
> You are welcome to look here:
>
> https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-11
>
> https://tools.ietf.org/html/draft-gafni-ippm-ioam-additional-data-fields-00
>
Actually the qlen is very specifically defined:)
https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-11#section-5.4.2.7

I understand and agree with the intention to keep telemetry options more
flexible (to get wider HW support). A paragraph explaining what are the key
properties or requirements of these metrics to achieve a precise link load
estimate would provide more guidance. For example the qlen defined in ippm
draft is the "queue length at departure time". Will the algorithm work the
same if qlen is metered at ingress (say some HW can't do egress for some
reason). What if there are hybrid mix of different qlen measurements on the
path.

      includes link load (txBytes, qlen, ts) and link spec (switch_ID,
      port_ID, B) at the egress port.  Note, each switch should record
      all those information at the single snapshot to achieve a precise
      link load estimate."

>
>
> Any further feedback is welcome.
>
>
>
> Thanks,
>
> Barak
>
>
>
> *From:* Yuchung Cheng <[email-masked]>
> *Sent:* Tuesday, December 15, 2020 2:35 PM
> *To:* NBU-Contact-Rui Miao <[email-masked]>
> *Cc:* iccrg <[email-masked]>; Pan, Rong <[email-masked]>;
> NBU-Contact-Harry Liu <[email-masked]>; jri.ietf <
> [email-masked]>; Lee, Jeongkeun <[email-masked]>; Barak Gafni <
> [email-masked]>; Yuval Shpigelman <[email-masked]>
> *Subject:* Re: [iccrg] New draft submitted for
> draft-pan-tsvwg-hpccplus-02.txt
>
>
>
> Interesting work!
>
>
>
> qlen == instant qlen snapshot at packet ingress or egress, on a per-port-per-queue basis, or some windowed-avg / aggregate etc.
>
>
>
>
>
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fid%2Fdraft-pan-tsvwg-hpccplus-02.txt&data=04%7C01%7Cgbarak%40nvidia.com%7Ccaefd0944b4f4a025acd08d8a149dacf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637436685862257347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=WGsD6vSg6JqJ6sNV4dqrE56yXzxmkIf%2FHnWSw742bUE%3D&reserved=0>
>
>
>
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.irtf.org%2Fmailman%2Flistinfo%2Ficcrg&data=04%7C01%7Cgbarak%40nvidia.com%7Ccaefd0944b4f4a025acd08d8a149dacf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637436685862267343%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=R5%2F8N%2FGbsiVrdoWuLBx6TkXzD%2BdZ0EnU107gmLgiDqY%3D&reserved=0>
>
>

---

From: "Pan, Rong" <[email-masked]>
Date: Wed, 16 Dec 2020 02:09:37 +0000

Michael,

I read the paper that you referenced. Indeed, it is very similar to HPCC++.  Time has finally come to do switch-assisted congestion control 😊. I am not clear about your comment on “path segments”. Data center network should be able to support it end-to-end, right?

Thanks,

Rong

From: Michael Welzl <[email-masked]>
Date: Monday, December 14, 2020 at 11:24 PM
To: "Rui, Miao" <[email-masked]>
Cc: iccrg <[email-masked]>, "Pan, Rong" <[email-masked]>, ""Liu, Hongqiang(洪强)"" <[email-masked]>, "jri.ietf" <[email-masked]>, "Lee, Jeongkeun" <[email-masked]>, Barak Gafni <[email-masked]>, Yuval Shpigelman <[email-masked]>
Subject: Re: [iccrg] New draft submitted for draft-pan-tsvwg-hpccplus-02.txt

Hi !

Cheers,
Michael

On Dec 15, 2020, at 1:11 AM, Rui, Miao <[email-masked]<mailto:[email-masked]>> wrote:

Hello ICCRG members,

Best regards,

Rui Miao
_______________________________________________
iccrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/iccrg

---

From: "Pan, Rong" <[email-masked]>
Date: Wed, 16 Dec 2020 02:21:01 +0000

Yuchung,

Thanks for reviewing the draft. Good point about where/how to measure the queue length.  We will look into adding a paragraph to describe the difference between ingress or egress-based qlen or whether they can be mixed. From your experience, what issues do you think we need to look out for?

Best,

Rong

From: Yuchung Cheng <[email-masked]>
Date: Tuesday, December 15, 2020 at 3:17 PM
To: Barak Gafni <[email-masked]>
Cc: NBU-Contact-Rui Miao <[email-masked]>, iccrg <[email-masked]>, "Pan, Rong" <[email-masked]>, NBU-Contact-Harry Liu <[email-masked]>, "jri.ietf" <[email-masked]>, "Lee, Jeongkeun" <[email-masked]>, Barak Gafni <[email-masked]>, Yuval Shpigelman <[email-masked]>
Subject: Re: [iccrg] New draft submitted for draft-pan-tsvwg-hpccplus-02.txt

On Tue, Dec 15, 2020 at 3:03 PM Barak Gafni <[email-masked]<mailto:[email-masked]>> wrote:
Hi,
Thanks for the interest in this work. For your question, at this point the draft has been kept open in terms of what is the exact inband telemetry technology to be used in order to implement the algorithm. The idea was to enable a variety of implementations. With that, one option we are focusing on is IOAM which is under work at IPPM WG, and has a data draft specifying formats for the communication of these metrics. Alongside this main data draft, there is a another draft in its initial work that adds few more fields, which may be used for HPCC++.
You are welcome to look here:
https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-11
https://tools.ietf.org/html/draft-gafni-ippm-ioam-additional-data-fields-00
Actually the qlen is very specifically defined:)
https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-11#section-5.4.2.7

I understand and agree with the intention to keep telemetry options more flexible (to get wider HW support). A paragraph explaining what are the key properties or requirements of these metrics to achieve a precise link load estimate would provide more guidance. For example the qlen defined in ippm draft is the "queue length at departure time". Will the algorithm work the same if qlen is metered at ingress (say some HW can't do egress for some reason). What if there are hybrid mix of different qlen measurements on the path.

      link load estimate."

Thanks,
Barak

From: Yuchung Cheng <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, December 15, 2020 2:35 PM
To: NBU-Contact-Rui Miao <[email-masked]<mailto:[email-masked]>>
Cc: iccrg <[email-masked]<mailto:[email-masked]>>; Pan, Rong <[email-masked]<mailto:[email-masked]>>; NBU-Contact-Harry Liu <[email-masked]<mailto:[email-masked]>>; jri.ietf <[email-masked]<mailto:[email-masked]>>; Lee, Jeongkeun <[email-masked]<mailto:[email-masked]>>; Barak Gafni <[email-masked]<mailto:[email-masked]>>; Yuval Shpigelman <[email-masked]<mailto:[email-masked]>>
Subject: Re: [iccrg] New draft submitted for draft-pan-tsvwg-hpccplus-02.txt

Interesting work!

It'd be good to know more precise requirements on INT to help both vendor supports (beside MLX) and CC evaluation

For example

On Mon, Dec 14, 2020 at 4:11 PM Rui, Miao <[email-masked]<mailto:[email-masked]>> wrote:
Hello ICCRG members,

Alibaba, Intel, and Mellanox have worked on an INT-based High Precision Congestion Control algorithm: HPCC++. We have posted an initial draft that can be found at
https://www.ietf.org/id/draft-pan-tsvwg-hpccplus-02.txt<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fid%2Fdraft-pan-tsvwg-hpccplus-02.txt&data=04%7C01%7Cgbarak%40nvidia.com%7Ccaefd0944b4f4a025acd08d8a149dacf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637436685862257347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=WGsD6vSg6JqJ6sNV4dqrE56yXzxmkIf%2FHnWSw742bUE%3D&reserved=0>

Best regards,
Rui Miao
_______________________________________________
iccrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/iccrg<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.irtf.org%2Fmailman%2Flistinfo%2Ficcrg&data=04%7C01%7Cgbarak%40nvidia.com%7Ccaefd0944b4f4a025acd08d8a149dacf%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637436685862267343%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=R5%2F8N%2FGbsiVrdoWuLBx6TkXzD%2BdZ0EnU107gmLgiDqY%3D&reserved=0>

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 15 Dec 2020 20:44:13 -0800

Hi Rong,

I actually think ingress or egress won't matter as much as qlen vs sojourn
time based telemetry. I have not read the hpcc paper yet but I am curious
its performance with multiple QoS being present -- apologize I have not
read the paper yet:

Let's say a port has two queues for high (H) and low (L) priority packets
separately.

A low-prio packet arrives at (an empty) L. It waits a long time because H
is extremely busy. So the qlen either ingress or egress metered is 0 for
this packet, but its sojourn or actual queuing time is large. The degree
depends on the switch's priority scheduling policy of course. so the packet
indeed experiences some congestion but not visible with qlen-INT metric.

On Tue, Dec 15, 2020 at 6:21 PM Pan, Rong <[email-masked]> wrote:

> Yuchung,
>
>
>
> Thanks for reviewing the draft. Good point about where/how to measure the
> queue length.  We will look into adding a paragraph to describe the
> difference between ingress or egress-based qlen or whether they can be
> mixed. From your experience, what issues do you think we need to look out
> for?
>
>
>
> Best,
>
>
>
> Rong
>
>
>
> *From: *Yuchung Cheng <[email-masked]>
> *Date: *Tuesday, December 15, 2020 at 3:17 PM
> *To: *Barak Gafni <[email-masked]>
> *Cc: *NBU-Contact-Rui Miao <[email-masked]>, iccrg <
> [email-masked]>, "Pan, Rong" <[email-masked]>, NBU-Contact-Harry Liu <
> [email-masked]>, "jri.ietf" <[email-masked]>, "Lee,
> Jeongkeun" <[email-masked]>, Barak Gafni <[email-masked]>, Yuval
> Shpigelman <[email-masked]>
> *Subject: *Re: [iccrg] New draft submitted for

---

From: Michael Welzl <[email-masked]>
Date: Wed, 16 Dec 2020 08:11:18 +0100

Hi Rong,

> On Dec 16, 2020, at 3:09 AM, Pan, Rong <[email-masked]> wrote:
> 
> Michael,
>  
> I read the paper that you referenced.

Thanks!  :-)

> Indeed, it is very similar to HPCC++.  Time has finally come to do switch-assisted congestion control 😊.

Yes, that’s what I thought when I first saw this - now there’s hardware supporting such mechanisms!

> I am not clear about your comment on “path segments”. Data center network should be able to support it end-to-end, right?

Oh sorry, is this the only context here? (It was, in the HPCC paper).  Then yes, of course!  I don’t see a problem with this for data centers.
(except for, possibly, some of the issues with “capacity" that Michael Scharf listed in his interesting email here - I see you were not in cc:
https://mailarchive.ietf.org/arch/msg/tsvwg/37SUp8JSE_tJXzKM6-tDcOhJaeA/   - but I don’t think that these *always* apply, under all conditions…  surely there must be ways around them, at least in datacenters)

Cheers,
Michael

---

From: "Rui, Miao" <[email-masked]>
Date: Thu, 17 Dec 2020 10:35:51 +0800

Hi Michael,

Thanks for your notes. Yes, I was talking about the capacity issue with Michael Scharf in another thread. Having switch-assisted congestion control does not mean that switches need to ``allocate'' bandwidth in a dual-mode. Instead, we can still implement a primal-mode algorithm with precise congestion information provided by switches for fast utilization convergence (though it will take longer for fairness to converge).

Yes, having switch-assisted is possible with newer hardware and many vendors now support this. But we are careful about the primitives supported. 

First, in HPCC++ we only ask the switch to support write-to-packet operation which is already supported in inband telemetry (like IOAM mentioned earlier). However, IMO adding a read-from-packet operation would trigger more interesting research ideas.

Second, there are many choices of load information like tx/rx rate, qlen, timestamp, etc. The key design of HPCC++ is to define ``inflight byte'' to quantify the load, which can be calculated by tx rate, queue length, timestamp. Because inflight byte can tell both overload or underload of the link for us to increase/decrease the rate multiplicatively. On contrary, traditional wisdom of having only the queue-length as the feedback can only guide the rate decrease but not increase. Besides, it subjects to fluctuation unless a certain threshold is held. We also compare RX and TX rate as well in the paper. HPCC++ uses TX rate while XCP uses RX rate. Generally, we view the TX rate is more precise because RX rate and queue_length are overlapped and thus it causes oscillation.

Thanks,
Rui

------------------------------------------------------------------
发件人：Michael Welzl <[email-masked]>
发送时间：2020年12月15日(星期二) 23:11
收件人："Pan, Rong" <[email-masked]>
抄　送："Miao, Rui(缪睿)" <[email-masked]>; "Liu, Hongqiang(洪强)" <[email-masked]>; jri.ietf <[email-masked]>; iccrg <[email-masked]>; "Lee, Jeongkeun" <[email-masked]>; Barak Gafni <[email-masked]>; Yuval Shpigelman <[email-masked]>
主　题：Re: [iccrg] New draft submitted for draft-pan-tsvwg-hpccplus-02.txt

Hi Rong,

Thanks!  :-)

Cheers,
Michael

---

From: "Rui, Miao" <[email-masked]>
Date: Thu, 17 Dec 2020 10:52:36 +0800

Hi Yuchung,

That's a great question! In our current draft and also in the paper, we are focusing on the deployment of uniform transport to discuss the algorithm in a clean way. Actually, many CC algorithms talk in the same as well. 

However, I think you made a good observation that QoS is indeed involved with congestion control in production datacenters. We believe the WRR/QoS in the switch should work friendly with HPCC++. Because in datacenter we mostly use the same priority with different WRR/DWRR weights to avoid starvation from using straight priority(SP). (some small protocol messages like BGP are still using SP though). DWRR/WRR provides a minimum bandwidth guarantee for each queue. In this sense, upon the congestion, HPCC++ should know that at least we are safe to back off to my bandwidth guarantee. So the utilization convergence is still very fast to mitigate the emerging congestion. By contrast, I agree that we should be careful and slow in increasing the rate in this scenario since we are not the only protocol over the link. 

We are happy to work through those details in support QoS in HPCC++ if it is actually an essential concern.

Thanks,
Rui

------------------------------------------------------------------
发件人：Yuchung Cheng <ycheng=[email-masked]>
发送时间：2020年12月15日(星期二) 20:45
收件人："Pan, Rong" <[email-masked]>
抄　送："Miao, Rui(缪睿)" <[email-masked]>; "Liu, Hongqiang(洪强)" <[email-masked]>; jri.ietf <[email-masked]>; iccrg <[email-masked]>; Barak Gafni <[email-masked]>; "Lee, Jeongkeun" <[email-masked]>; Barak Gafni <[email-masked]>; Yuval Shpigelman <[email-masked]>
主　题：Re: [iccrg] New draft submitted for draft-pan-tsvwg-hpccplus-02.txt

Hi Rong,

I actually think ingress or egress won't matter as much as qlen vs sojourn time based telemetry. I have not read the hpcc paper yet but I am curious its performance with multiple QoS being present -- apologize I have not read the paper yet:

Let's say a port has two queues for high (H) and low (L) priority packets separately.

A low-prio packet arrives at (an empty) L. It waits a long time because H is extremely busy. So the qlen either ingress or egress metered is 0 for this packet, but its sojourn or actual queuing time is large. The degree depends on the switch's priority scheduling policy of course. so the packet indeed experiences some congestion but not visible with qlen-INT metric.

 
Best,
 
Rong

 

Thanks,
Barak

From: Yuchung Cheng <[email-masked]> 
Sent: Tuesday, December 15, 2020 2:35 PM
To: NBU-Contact-Rui Miao <[email-masked]>
Cc: iccrg <[email-masked]>; Pan, Rong <[email-masked]>; NBU-Contact-Harry Liu <[email-masked]>; jri.ietf <[email-masked]>; Lee, Jeongkeun <[email-masked]>; Barak Gafni <[email-masked]>; Yuval Shpigelman <[email-masked]>
Subject: Re: [iccrg] New draft submitted for draft-pan-tsvwg-hpccplus-02.txt

Interesting work!

For example

 

 Best regards,
Rui Miao