Subject: [Rdma-cc-interest] Minutes of the side meeting at IETF-106

From: Paul Congdon <[email-masked]>
Date: Tue, 3 Dec 2019 11:55:19 -0800

Here are the final minutes as amended by previous comments.  Sorry if this
gets duplicated as some of you are also on the IETF mailing list as well as
the bcc list.  Thank you all for attending and continuing to progress this
work.

IETF-106 Side Meeting:

Data Center Congestion Control – Where’s the best fit in IETF/IRTF?

Tuesday, November 19, 2019

8:30AM – 9:45AM

Attendees (who signed the blue/white sheet or where recognized):

*First*

*Last*

*Affiliation*

Peter

Asai

Preferred Networks / Wide

David

Black

Dell EMC

Roland

Bless

Kit

Zhen

Cao

Huawei

Stuart

Cheshire

Apple

Tim

Costello

BT

Roni

Even

Huawei

Rodney

Grimes

SCE netDEF

Wim

Hendrerickx

Nokia

Jake

Holland

Akamai

Chriatian

Hopps

Lab IU Consulting

Sheng

Jiang

Huawei

Jiao

Kang

Huawei

Mengzhu

Liu

Huawei

David

Melman

Marvell

Tal

Mizrahi

Toga

Mark

Pearson

HPE

Roberto

Peon

Facebook

Jamal Hadi

Salim

Mojatatu

Marcus

Sun

Huawei

Sowmini

Varadhan

Microsoft

Aijun

Wang

China Telecom

Huaru

Yang

Huawei

Yolanda

Yu

Huawei

Yan

Zhaung

Huawei

Ning

Zong

Huawei

Details:

1.    The meeting organizer, Paul Congdon, presented the IETF Note Well
reminder and bashed the agenda.  No changes to the agenda.

2.    The slide material presented at the meeting is available at:
https://mentor.ieee.org/802.1/dcn/19/1-19-0087-00-ICne-ietf-106-sidemeeting.pdf

3.    The slides from IETF-106 HotRFC that announced this side meeting are
available at:
https://datatracker.ietf.org/meeting/106/materials/slides-106-hotrfc-data-center-congestion-control-wheres-the-best-fit-in-ietfirtf

4.    Paul Congdon reviewed the history of previous contributions since
IETF-101.  This is the third side meeting on HPC/RDMA/AI data center
network congestion control.  The last side meeting had mentioned a number
of potential research topics ranging from considering new UDP based
transports for RDMA to ways to improve congestion signal feedback by
providing more inline information or allowing the network to more actively
participate.  The growing number of vendor proprietary solutions hints a
the need for open interoperable standards work.

5.    Roni Even presented on Fast Congestion management for Data Centers
 available at
https://tools.ietf.org/html/draft-even-iccrg-dc-fast-congestion-00

a.    The experimental data was obtained from a simple single switch
topology.  David Black suggested that it will be important to understand
how the solution works in a more complex topology with more than one
bottleneck.  Beware of congestion collapse in this case.

b.    David Black pointed out that ICMP source quench has a bad reputation
in the IETF and it will be very important for this approach to distinguish
itself from that.

c.    David Black also pointed out that many of these environments involve
overlay networks and the solution should not assume that sending a message
back to the source will reliably get the message to the original sender.  A
backward message needs to contain enough information for it to have the
proper impact.

d.    A somewhat related problem is propagating ECN bits from the
encapsulating tunnel to the flows within.  RFC 6040 talks about this as
well as some drafts, but implementation of this has been leisurely.

e.    Jake Holland wanted to understand why ICMP source quench is a
problem.  Is it simply the presence of tunnels?  It wasn’t clear what the
other problems were at the time of the meeting, but this should be
investigated.

f.     Rod Grimes asked if there was any measurement on the number of
control packets were being sent.  The message was modeled after an extended
IBTA CNP (Congestion Notification Packet).  It would be good to understand
the parameters that were used to cause the message to be sent and what is
included in the message.

g.    The summary of recommended actions for this draft include:

                                         i.    Use a more complex topology
with more bottlenecks

                                        ii.    Distinguish the solution
from ICMP source quench

                                       iii.    Consider how this works with
tunnels and encapsulations

                                       iv.    List the parameters that were
used for the test for example threshold time for sending control message.

6.    Yan Zhuang presented An Open Congestion Control Architecture for high
performance fabrics available at:
https://datatracker.ietf.org/doc/draft-zhuang-tsvwg-open-cc-architecture/

a.    Jake Holland indicated the approach was interesting and felt it was
promising.  He asked if there was an existing API or set of operations that
existed to support the experiments.  Yan indicated it was still under
development.  Operations such as pacing and providing an array of flags on
the GRO and more information about packets on the wire would be helpful.
The API will require lots of discussion.

b.    Jamal Hadi Salim pointed out that MIT has a similar approach that was
presented at Sigcomm 2018 and at NetDev in 2018. The NetDev paper was an
engineering paper and discussed APIs. There was a criticism about the
scalability of the approach.

c.    Roberto Peon ask if packet loss evaluated when comparing CCAs?  He
indicated that experiments at 100GbE have shown that bottlenecks exist in
the end-system and without proper feedback they will back off too
aggressively.  Shifting the CC to the NIC could help, but not yet clear.

d.    Roberto Peon stated that shifting decision making to places in the
network where measurements can be made on very short time scales will be
important in the data center.  The longer the delay for a control loop, the
smaller the impact that offloads will have.

e.    The summary of recommended actions for this draft include:

                                         i.    Please provide the details
of the API

7.    Yan Zhuang presented on Artificial Intelligence (AI) based ECN
adaptive reconfiguration for datacenter networks available at
https://datatracker.ietf.org/doc/draft-zhuang-tsvwg-ai-ecn-for-dcn

a.    Stuart Cheshire asked about the use of RED in the test cases and
whether there was an assumption that RED prevented packet loss and CoDel
does not.  If ECN is used correctly, you can prevent packet loss regardless
of using RED or CoDel.  The experiments used RED because that was available
in the commodity switches used in the test.

b.    David Black points out that it is important to use an FQ or a partial
FQ solution here because the experiments are likely showing the
interference between flows.   It is suggested this analysis should be done
using a more modern AQM.  It could be that the optimization is not as
effective if a modern AQM is used.

c.    It was pointed out that CoDel is more burst friendly.  RED is
obsolete.

d.    Jake Holland had a concern about how this works at larger scale.  He
agreed that CoDel would have a different outcome.  There was a question
about the AI model and what constitutes ‘normal’.

e.    The summary of recommended actions for this draft include:

                                         i.    The analysis should be
re-run using a modern AQM – not RED

                                        ii.    Provide details about the
model’s definition of ‘normal’ behavior I

8.    Yolanda Yu presented on The impact of mixing TCP and RoCEv2.  The
slides are in the complete set of slide material at
https://mentor.ieee.org/802.1/dcn/19/1-19-0087-00-ICne-ietf-106-sidemeeting.pdf

a.    David Black made a positive comment about the work, indicating that
more of this type is needed.  He requested that all the settings and
configuration in the test environment be documented in extreme detail.

b.    The summary of recommended actions for this work:

                                         i.    Provide excruciating detail
about the configuration of the test environment

9.    Paul Congdon took the final minutes to ask people how to proceed
within the IETF.  David Black felt the side meeting approach appears to be
working and suggested to continue.  Paul pointed out that, while it is
great that IETF is making the time available for side meetings, there is a
challenge assuring a time is available to meet that doesn’t conflict and
can draw critical attendees.   The meetings are not being recorded and
there isn’t a place to upload the contributions.  This would be an
improvement that IETF could make to the process.

10. The side meeting was adjourned at approximately 9:45AM

---

From: Lars Eggert <[email-masked]>
Date: Wed, 4 Dec 2019 08:37:25 +0200

Hi,

On 2019-12-3, at 21:55, Paul Congdon <[email-masked]> wrote:
> e.    Jake Holland wanted to understand why ICMP source quench is a problem.  Is it simply the presence of tunnels?  It wasn’t clear what the other problems were at the time of the meeting, but this should be investigated.

It can't be authenticated, i.e., it's trivial to spoof. In other words, any node can quench any other node.

> d.    Roberto Peon stated that shifting decision making to places in the network where measurements can be made on very short time scales will be important in the data center.  The longer the delay for a control loop, the smaller the impact that offloads will have.

+1

> 9.    Paul Congdon took the final minutes to ask people how to proceed within the IETF.  David Black felt the side meeting approach appears to be working and suggested to continue.

+1

>  Paul pointed out that, while it is great that IETF is making the time available for side meetings, there is a challenge assuring a time is available to meet that doesn’t conflict and can draw critical attendees.   The meetings are not being recorded and there isn’t a place to upload the contributions.  This would be an improvement that IETF could make to the process.

Any organizer of a side meeting can minute the meeting (as you did here) and make materials available, e.g., on GitHub.

Lars