Subject: [Rdma-cc-interest] Side meeting plans at IETF-106

From: Paul Congdon <[email-masked]>
Date: Mon, 11 Nov 2019 11:11:39 -0800

Fellow rdma-cc-interest members:

I have reserved a side meeting room at IETF-106 to continue our discussions
on congestion control in modern data center networks.  This message is to
solicit feedback on the suggested topics and agenda.

Tuesday, November 19

8:30AM - 9:45AM

Room: VIP A

There has been some discussion about where to take this group and what
activities should spin out from it; should we expand the current focus in
ICCRG or are these topics appropriate for a new research group in IRTF.  At
IETF-106 there are a number of topics that people would like to discuss.
In particular, where can the IETF/IRTF discuss how NICs can be designed for
CC in HPC/RDMA/AI DCN and also how can the network participate to improve
performance.

I'm working on a more detailed agenda for the side meeting, but here are
some current thoughts and feedback is welcomed.

1. How NICs can be designed for better CC in the HPC/RDMA/AI DCN

Discuss feedback on a draft under development on OpenCC:
https://datatracker.ietf.org/doc/draft-zhh-tsvwg-open-architecture/; a
framework for flexible establishment of congestion control algorithms
implemented by NICs and the network.  The expectation is there will be some
experiment results.  The goal is to discuss the ideas with stakeholders
(customers, NIC vendors, switch vendors) and explore what could/should
be standardized.

2. How does the network participate in CC for HPC/RDMA/AI DCN?

There are a few items for discussion.

2.1 AI ECN

Discuss feedback on
https://datatracker.ietf.org/doc/draft-zhuang-tsvwg-ai-ecn-for-dcn/.  The
idea is to use AI for adaptive configuration of the network - a hard
problem.  How is necessary information collected from the devices to form
models and what could/should be standardized here as well?

2.2 Network Fast Feedback

Discuss follow-on feedback on
https://tools.ietf.org/html/draft-even-iccrg-dc-fast-congestion-00 which is
expected to be introduced in ICCRG on Monday.  The draft discusses the
state-of-the-art congestion controllers in use and from research, and poses
a number of questions for discussion. What is to be researched and what
could/should be standardized going forward?

2.3 Mixing RDMA and TCP traffic

These two traffic types with their differing congestion controllers are
known to not play well with one another in the same traffic class.  There
may be some analysis data to share on this topic.  A goal would be to
discuss network approaches for mitigating the impact of the two on each
other.

3. Metrics for HPC/RDMA/AI networks

Are the current metrics and scales appropriate for HPC/RDMA/AI networks?
HPC and Storage networks tend to use IOPS as a key measure and the latency
requirements can be on the order of 10us; much different than Internet
latency and throughput measures.  Should there be a draft on metric
requirements for DCN networks?   Can we work with real customers to define
some well-known scenarios and metrics for HPC/RDMA/AI DCNs.

---

From: Lars Eggert <[email-masked]>
Date: Tue, 12 Nov 2019 13:35:20 +0200

Hi,

> Room: VIP A

I'll try and make the meeting, modulo NomCom duties. (Please all, send your feedback about candidates to NomCom now!)

Because I am not fully sure I can make this, some written feedback on the various agenda items:

> Discuss feedback on a draft under development on OpenCC: https://datatracker.ietf.org/doc/draft-zhh-tsvwg-open-architecture/; a framework for flexible establishment of congestion control algorithms implemented by NICs and the network.  The expectation is there will be some experiment results.  The goal is to discuss the ideas with stakeholders (customers, NIC vendors, switch vendors) and explore what could/should be standardized.

There seem to be three things here:

A. A modular NIC offload interface. Unclear if the IETF is the right home for this? Also, see "Restructuring Endpoint Congestion Control" (Narayan et al.) for another direction.

B. Mixing and matching different CCs in one network over time. Given that at datacenter latencies, you really want to prevent even small-scale hiccups due to interactions between different CCs, I wonder if it would be sufficient to slice the NIC and have different slices where all traffic is handled by one CC? Seems more tractable.

C. The architectural idea to move away from in-band CC signaling from the network to the endpoints. There isn't much in the document that motivates this, and nothing about potential issues (e.g., loss of fate-sharing).

A comment on this document that is really about this entire effort: we should just give up on RoCE. Mellanox has no interest in opening it, and I am therefore unwilling to spend cycles thinking about it.

> Discuss feedback on https://datatracker.ietf.org/doc/draft-zhuang-tsvwg-ai-ecn-for-dcn/.  The idea is to use AI for adaptive configuration of the network - a hard problem.  How is necessary information collected from the devices to form models and what could/should be standardized here as well?

I don't see a proposal here, I don't even see a concrete problem statement? This is yet another "let's throw AI at it" three-pager.

> Discuss follow-on feedback on MailScanner has detected a possible fraud attempt from "tools.ietf.org" claiming to be https://tools.ietf..org/html/draft-even-iccrg-dc-fast-congestion-00 which is expected to be introduced in ICCRG on Monday.  The draft discusses the state-of-the-art congestion controllers in use and from research, and poses a number of questions for discussion. What is to be researched and what could/should be standardized going forward?

This is the beginnings of a survey. It misses a ton of related work esp. from academia though. HOMA, pFabric, HULL, D3, PDQ, pHost, NDP, etc., etc.

> These two traffic types with their differing congestion controllers are known to not play well with one another in the same traffic class.  There may be some analysis data to share on this topic.  A goal would be to discuss network approaches for mitigating the impact of the two on each other.

When you say RDMA, you mean RoCE? Separate RoCE into a slice and move on. It's pointless to try and optimize for coexistence with a protocol that can change willy-nilly.

> Are the current metrics and scales appropriate for HPC/RDMA/AI networks?  HPC and Storage networks tend to use IOPS as a key measure and the latency requirements can be on the order of 10us; much different than Internet latency and throughput measures.  Should there be a draft on metric requirements for DCN networks?   Can we work with real customers to define some well-known scenarios and metrics for HPC/RDMA/AI DCNs.

Whose "current metrics and scales"? Papers on DC mechanisms certainly define appropriate metrics. Obviously Internet scales don't work, but who is using those?

Lars

---

From: "Zhuangyan (Yan)" <[email-masked]>
Date: Tue, 12 Nov 2019 23:28:01 +0000

Hi Lars,

Thank you for the review and detailed comments. Some responses inline at [Y].

Best Regards,

Yan

________________________________
发件人: Rdma-cc-interest <[email-masked]> 代表 Lars Eggert <[email-masked]>
发送时间: 2019年11月12日 19:35
收件人: Paul Congdon
抄送: [email-masked]
主题: Re: [Rdma-cc-interest] Side meeting plans at IETF-106

Hi,

A. A modular NIC offload interface. Unclear if the IETF is the right home for this? Also, see "Restructuring Endpoint Congestion Control" (Narayan et al.) for another direction.
[Y] yes, papers from Sigcomm (or related conferences) have discussed about decoupling congestion controls on NICs a bit, like CCP, AD/DC TCP et al., which we think it might be good for a discussion in the industry for a modular cc design : ), however we might be more focused on the interfaces between layers while the implementation of modules would be open to vendors. It could be discussed further.

B. Mixing and matching different CCs in one network over time. Given that at datacenter latencies, you really want to prevent even small-scale hiccups due to interactions between different CCs, I wonder if it would be sufficient to slice the NIC and have different slices where all traffic is handled by one CC? Seems more tractable.
[Y] Slicing the nic rather than different cc algorithms would also be an option. We also plan on experiments of different configurations of one cc for different traffics which might be in later version. Based on results in this version, 2 cc algorithms performs better than one cc for two types of traffics. It might be due to different feedbacks/reacts in the network.

C. The architectural idea to move away from in-band CC signaling from the network to the endpoints. There isn't much in the document that motivates this, and nothing about potential issues (e.g., loss of fate-sharing).
[Y] The architecture is intended to support both signaling directly from network (rather from receivers--is that what you mean in-band CC signaling?) and signaling from receivers (current practice). Each CC chooses their own ways of signaling and sure it would affect each other somehow if different CCs are used together while their signaling is different. More details will be discussed in later version and inputs/issues that we missed are greatly welcome.

A comment on this document that is really about this entire effort: we should just give up on RoCE. Mellanox has no interest in opening it, and I am therefore unwilling to spend cycles thinking about it.
[Y] :) Actually, the architecture is not binding to RoCE. The thought is to support several transport protocols including TCP. Current experiment results of different ccs are based on TCP (Reno, Cubi, bbr, dctcp), however we don’t want to exclude RoCE either at this point :( ...
Besides, Mellanox provides really good smartNIC design. And very recently, they also announced programmable congestion control on their NICs. At this point, it might be a good timing to discuss with the broader industry on this…

I don't see a proposal here, I don't even see a concrete problem statement? This is yet another "let's throw AI at it" three-pager.
[Y] AI might not be a technical wording nowadays due to ai everywhere…however, we do provide a scene-based ECN reconfiguration to adapt the changes of traffics in data center networks in which scene training and dynamic scene inducing is where AI technologies are applied…and we don't want to limit any specific ai technologies, like deep-forest or et al...
We would share some testing results in the side meeting ...to show it is not another ai paper work...hope it helps.
The problem is stated in background section, however it might not be so obvious… “As stated in [RFC7567], with proper parameters, RED can be an effective algorithm.  However, dynamically predicting the set of parameters (minimum threshold and maximum threshold) is difficult. ” Dynamic configuration of threshold is a problem for network configuration somehow…

When you say RDMA, you mean RoCE? Separate RoCE into a slice and move on. It's pointless to try and optimize for coexistence with a protocol that can change willy-nilly.
[Y] yes, it means RoCE. If the network does not differentiate ROCE and TCP traffics, then they would compete anyhow…L4S might work on a similar work on classic TCP vs. DCTCP.

Whose "current metrics and scales"? Papers on DC mechanisms certainly define appropriate metrics. Obviously Internet scales don't work, but who is using those?
[Y] To my understanding, current metrics in ietf are mostly discussed about Internet. The question is whether it is worthy dicussing and getting consenus on some common metrics for applications in DCN especially HPC. It is an open discussion to seek feedbacks :)

Lars
--
Rdma-cc-interest mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rdma-cc-interest

---

From: "Black, David" <[email-masked]>
Date: Wed, 13 Nov 2019 00:39:59 +0000

A few follow-up comments.

>>> 2.1 AI ECN
>>>
>>> Discuss feedback on https://datatracker.ietf.org/doc/draft-zhuang-tsvwg-ai-ecn-for-dcn/.

[… snip …]

> The problem is stated in background section, however it might not be so obvious… “As stated in [RFC7567], with proper parameters,
> RED can be an effective algorithm.  However, dynamically predicting the set of parameters (minimum threshold and maximum threshold)
> is difficult. ” Dynamic configuration of threshold is a problem for network configuration somehow…

That sounds like a straw-person demolition exercise, as RFC 7567 also states:

   This memo also explicitly obsoletes the recommendation that Random
   Early Detection (RED) be used as the default AQM mechanism for the
   Internet.  This is replaced by a detailed set of recommendations for
   selecting an appropriate AQM algorithm.

The upshot is that RED is not a good AQM algorithm to use as a comparison baseline - I hope that the results to be presented will make comparisons to more recent AQM algorithms, e.g., FQ-CoDel [RFC8290].

>>> These two traffic types with their differing congestion controllers are known to not play well with one another in the same traffic class.

Using protocol names to denote congestion control classes does not work well, even though it’s common (and I’ve done it myself).

We are dealing with two clases of congestion controls.  For lack of better terms the following class names are based on what the transport protocol throughput is proportional to where ‘p’ is the loss and/or congestion marking probability:
               - 1/sqrt(p)-class congestion controls: Includes most existing TCP congestion control algorithms, e.g., NewReno, CUBIC.
               - 1/p-class congestion controls: Includes DCTCP congestion control.
Keep in mind that p is a probability that is usually << 1 when expresed as a decimal, e.g., p=0.01 represents a 1% loss/marking rate.
> [Y] yes, it means RoCE. If the network does not differentiate ROCE and TCP traffics, then they would compete anyhow…L4S might work on a similar work on classic TCP vs. DCTCP.

DCTCP would be a better protocol to focus on than RoCE, as both DCTCP congestion control and the DCQCN congestion control commonly used for RoCE are 1/p-class congestion controls.

TSVWG will be discussing L4S and SCE next week – both of those proposals are intended to enable coexistence of 1/sqrt(p)-class and 1/p-class congestion controls, among other goals.

Thanks, --David

From: Rdma-cc-interest <[email-masked]> On Behalf Of Zhuangyan (Yan)
Sent: Tuesday, November 12, 2019 6:28 PM
To: Lars Eggert; Paul Congdon
Cc: [email-masked]
Subject: [Rdma-cc-interest] 答复: Side meeting plans at IETF-106

[EXTERNAL EMAIL]

Hi Lars,

Best Regards,

Yan

________________________________
发件人: Rdma-cc-interest <[email-masked]<mailto:[email-masked]>> 代表 Lars Eggert <[email-masked]<mailto:[email-masked]>>
发送时间: 2019年11月12日 19:35
收件人: Paul Congdon
抄送: [email-masked]<mailto:[email-masked]>
主题: Re: [Rdma-cc-interest] Side meeting plans at IETF-106

Hi,

Lars
--
Rdma-cc-interest mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rdma-cc-interest

---

From: "Zhuangyan (Yan)" <[email-masked]>
Date: Wed, 13 Nov 2019 01:03:10 +0000

Hi David,

Thank you for the comments. Some responses as below.

Best Regards,

Yan

________________________________
发件人: Black, David <[email-masked]>
发送时间: 2019年11月13日 8:39
收件人: Zhuangyan (Yan); Lars Eggert; Paul Congdon
抄送: [email-masked]; Black, David
主题: RE: [Rdma-cc-interest] 答复: Side meeting plans at IETF-106

[… snip …]

[Y] Since it is a startpoint to see how adaptive/dyamic configuration can help, so we just use RED as the one to see whether it can be improved. Other algorithms can also be tested but would wait for next meeting cycle...the results to be presented are RED only...

[Y] sure, we can see how DCTCP works which might also be used in DCQCN.

Thanks, --David

[EXTERNAL EMAIL]

Hi Lars,

Best Regards,

Yan

Hi,

Lars

---

From: "Black, David" <[email-masked]>
Date: Wed, 13 Nov 2019 03:40:45 +0000

> [Y] Since it is a startpoint to see how adaptive/dyamic configuration can help, so we just use RED as

> the one to see whether it can be improved. Other algorithms can also be tested but would wait for

> next meeting cycle...the results to be presented are RED only...

Doing better than RED is of limited interest, IMHO.

Thanks, --David

From: Zhuangyan (Yan) <[email-masked]>
Sent: Tuesday, November 12, 2019 8:03 PM
To: Black, David; Lars Eggert; Paul Congdon
Cc: [email-masked]
Subject: 答复: [Rdma-cc-interest] 答复: Side meeting plans at IETF-106

[EXTERNAL EMAIL]

Hi David,

Best Regards,

Yan

________________________________
发件人: Black, David <[email-masked]<mailto:[email-masked]>>
发送时间: 2019年11月13日 8:39
收件人: Zhuangyan (Yan); Lars Eggert; Paul Congdon
抄送: [email-masked]<mailto:[email-masked]>; Black, David
主题: RE: [Rdma-cc-interest] 答复: Side meeting plans at IETF-106

[… snip …]

Thanks, --David

From: Rdma-cc-interest <[email-masked]<mailto:[email-masked]>> On Behalf Of Zhuangyan (Yan)
Sent: Tuesday, November 12, 2019 6:28 PM
To: Lars Eggert; Paul Congdon
Cc: [email-masked]<mailto:[email-masked]>
Subject: [Rdma-cc-interest] 答复: Side meeting plans at IETF-106

[EXTERNAL EMAIL]

Hi Lars,

Best Regards,

Yan

Hi,

Lars

---

From: "Zhuangyan (Yan)" <[email-masked]>
Date: Wed, 13 Nov 2019 04:53:25 +0000

Hi David,

The current practice is based on RoCE traffic : ). So we choose RED for ECN marking to avoid packet loss. Not so sure about how to use fq-codel in this case and would be happy to discuss. Alternatively, we can try tcp traffic as well.

Best Regards,

Yan

发送时间: 2019年11月13日 11:40

Thanks, --David

[EXTERNAL EMAIL]

Hi David,

Best Regards,

Yan

[… snip …]

Thanks, --David

[EXTERNAL EMAIL]

Hi Lars,

Best Regards,

Yan

Hi,

Lars

---

From: "Black, David" <[email-masked]>
Date: Wed, 13 Nov 2019 06:05:45 +0000

See Lars’s comments about paying less attention to RoCE – work on that non-IETF protocol belongs in the IBTA (InfiniBand Trade Association), which specifies that protocol.

Thanks, --David

Sent: Tuesday, November 12, 2019 11:53 PM

[EXTERNAL EMAIL]

Hi David,

Best Regards,

Yan

Thanks, --David

From: Zhuangyan (Yan) <[email-masked]<mailto:[email-masked]>>

[EXTERNAL EMAIL]

Hi David,

Best Regards,

Yan

[… snip …]

Thanks, --David

[EXTERNAL EMAIL]

Hi Lars,

Best Regards,

Yan

Hi,

Lars

---

From: "Zhuangyan (Yan)" <[email-masked]>
Date: Wed, 13 Nov 2019 08:13:50 +0000

Thank you, David.

Sure, we will keep those points in following thoughts and try tcp as well.

Since this practice asks for no change of roce, roce also uses ECN defined by IETF as its congestion marking and network management (like netconf/yang et al.) might be used for informaiton collection. Feedbacks and comments in IETF would be greatly appreciated...

Best Regards,

Yan

发送时间: 2019年11月13日 14:05

Thanks, --David

[EXTERNAL EMAIL]

Hi David,

Best Regards,

Yan

Thanks, --David

[EXTERNAL EMAIL]

Hi David,

Best Regards,

Yan

[… snip …]

Thanks, --David

[EXTERNAL EMAIL]

Hi Lars,

Best Regards,

Yan

Hi,

Lars

---

From: Lars Eggert <[email-masked]>
Date: Wed, 13 Nov 2019 10:21:58 +0200

On 2019-11-13, at 10:13, Zhuangyan (Yan) <[email-masked]> wrote:
> Since this practice asks for no change of roce, roce also uses ECN defined by IETF as its congestion marking and network management (like netconf/yang et al.) might be used for informaiton collection.

RoCE might at the moment *react to* ECN as defined by the IETF, but *how* it reacts is *not* defined by the IETF and can change at any time.

It's therefore pointless to try and cater to RoCE, since IBTA/Mellanox can change it at any time however they see fit.

Lars

---

From: "Zhuangyan (Yan)" <[email-masked]>
Date: Wed, 13 Nov 2019 19:09:50 +0000

Thank you, Lars. Sure, totally understand the concern.

It would be risky to work on a protocol that could be changed by others sometime, although in existing defintion there might be some possible ways to improve.

We will indicate this in the draft and try tcp traffics to see how this idea fits.

Thank you for your comments.

Yan

________________________________
From: Rdma-cc-interest <[email-masked]> on behalf of Lars Eggert <[email-masked]>
Sent: Wednesday, November 13, 2019 4:21 PM
To: Zhuangyan (Yan)
Cc: [email-masked]; Black, David; Paul Congdon
Subject: Re: [Rdma-cc-interest] Side meeting plans at IETF-106

Lars