Subject: [iccrg] CCID5 - BBR for DCCP

From: <[email-masked]>
Date: Tue, 9 Nov 2021 15:02:59 +0000

Dear all,

As discussed yesterday, at the ICCRG session I would like to bring here 2 topics to discussion.

In a first instance, we would like to start the process to bring BBR as a new CCID profile for DCCP. As an initial step, we have submitted the draft https://www.ietf.org/id/draft-romo-iccrg-ccid5-00.txt and we would be happy to receive any feedback or comment on it. It is worth to mention that our goal is not to propose definitions related to the Congestion Control algorithm itself, which is defined protocol agnostic at https://datatracker.ietf.org/doc/draft-cardwell-iccrg-bbr-congestion-control/, but to outline the necessary requirements to use it within DCCP.

The second topic I would like to discuss, is related to the problem found during the implementation of BBRv1 for DCCP. To summarize it, BBR requires the restoration of the cwnd when leaving the probeRTT phase, that restoration implies a big change (from a low to a high value) of the cwnd, and therefore a previous synchronization of the DCCP sequence and acknowledgement validity windows (RFC4340<https://datatracker.ietf.org/doc/html/rfc4340> section 7.5). Such synchronization is per DCCP RFC4340 a reliable negotiation to be confirmed and therefore takes at least one RTT, increasing the duration of the probeRTT phase. In the presentation slides<https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-ccid5-an-implementation-of-bbr-congestion-control-algorithm-for-dccp-00> you can find a more detailed description of the problem as well as the temporary solution that has been implemented. Here we would like to bring to consideration the possibility of enhancing or modifying the DCCP Sequence window negotiation process.

Best Regards
Nathalie Romo Moreno

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 9 Nov 2021 11:12:05 -0500

On Tue, Nov 9, 2021 at 10:03 AM <[email-masked]> wrote:

> In a first instance, we would like to start the process to bring BBR as a
> new CCID profile for DCCP. As an initial step, we have submitted the draft
> https://www.ietf.org/id/draft-romo-iccrg-ccid5-00.txt and we would be
> happy to receive any feedback or comment on it. It is worth to mention that
> our goal is not to propose definitions related to the Congestion Control
> algorithm itself, which is defined protocol agnostic at
> https://datatracker.ietf.org/doc/draft-cardwell-iccrg-bbr-congestion-control/,
> but to outline the necessary requirements to use it within DCCP.
>

Thanks for this nice work! It's great to see BBR implementations in a wider
variety of transports.

We have proposed to add the following text to the next rev of the BBR draft
to reference your work, alongside the descriptions of TCP and QUIC
implementations:
---
An open source implementation of BBR is also available for DCCP [RFC4340]
[draft-romo-iccrg-ccid5].
...
[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion
Control Protocol (DCCP)", RFC 4340, DOI 10.17487/RFC4340, March 2006, <
https://www.rfc-editor.org/info/rfc4340>.
...
[draft-romo-iccrg-ccid5] Romo, N., Kim, J., and M. Amend, "Profile for
Datagram Congestion Control Protocol (DCCP) Congestion Control ID 5", Work
in Progress, Internet-Draft, draft-romo-iccrg-ccid5, 25 October 2021, <
https://tools.ietf.org/html/draft-romo-iccrg-ccid5>.
---
Please let us know if you see any issues with that.

As you note, the intent with the BBR draft is to keep the description of
the algorithm protocol-agnostic. And I think it would be good to keep BBR
protocol-agnostic for as long as possible, to improve the odds of keeping
implementations for various transports in sync.

> The second topic I would like to discuss, is related to the problem found
> during the implementation of BBRv1 for DCCP. To summarize it, BBR requires
> the restoration of the cwnd when leaving the probeRTT phase, that
> restoration implies a big change (from a low to a high value) of the cwnd,
> and therefore a previous synchronization of the DCCP sequence and
> acknowledgement validity windows (RFC4340
> <https://datatracker.ietf.org/doc/html/rfc4340> section 7.5). Such
> synchronization is per DCCP RFC4340 a reliable negotiation to be confirmed
> and therefore takes at least one RTT, increasing the duration of the
> probeRTT phase. In the presentation slides
> <https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-ccid5-an-implementation-of-bbr-congestion-control-algorithm-for-dccp-00>
> you can find a more detailed description of the problem as well as the
> temporary solution that has been implemented. Here we would like to bring
> to consideration the possibility of enhancing or modifying the DCCP
> Sequence window negotiation process.
>

Thanks for highlighting this issue. One possible idea for addressing the
interactions between the DCCP Sequence window negotiation process and the
BBR ProbeRTT mechanism would be as follows: instead of basing the sender's
requested DCCP Sequence Number validity window strictly on the cwnd value,
the sender could try to continually maintain a Sequence Number validity
window  big enough to handle the anticipated volume of in-flight data in
the "near future" For BBR in ProbeRTT, this would be something like the
following, which would take into account both the current cwnd and the fact
that upon exiting ProbeRTT the flow will expect to raise its cwnd back to
something in the neighborhood of its previous cwnd before ProbeRTT:

In Internet Draft notation:
   anticipated_volume = max(cwnd, BBR.prior_cwnd)

In Linux TCP notation:
   anticipated_volume = max(tp->snd_cwnd, bbr->prior_cwnd)

That could avoid the approach mentioned in the slides, of a "Pro-active
update of local values even if the confirmation has not been received yet
(feature negotiation not finished)".

Does that sound like a plausible approach?

best regards,
neal

---

From: <[email-masked]>
Date: Wed, 10 Nov 2021 14:32:32 +0000

Hello Neal,

Thanks for the feedback and the prompt response. Please, see my comments in line

On Tue, Nov 9, 2021 at 10:03 AM <[email-masked]<mailto:[email-masked]>> wrote:

Dear all,

Thanks for this nice work! It's great to see BBR implementations in a wider variety of transports.

We have proposed to add the following text to the next rev of the BBR draft to reference your work, alongside the descriptions of TCP and QUIC implementations:
---
An open source implementation of BBR is also available for DCCP [RFC4340] [draft-romo-iccrg-ccid5].
...
[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, DOI 10.17487/RFC4340, March 2006, <https://www.rfc-editor.org/info/rfc4340>.
...
[draft-romo-iccrg-ccid5] Romo, N., Kim, J., and M. Amend, "Profile for Datagram Congestion Control Protocol (DCCP) Congestion Control ID 5", Work in Progress, Internet-Draft, draft-romo-iccrg-ccid5, 25 October 2021, <https://tools.ietf.org/html/draft-romo-iccrg-ccid5>.
---
Please let us know if you see any issues with that.

NR:// no issue, that looks good to me.

As you note, the intent with the BBR draft is to keep the description of the algorithm protocol-agnostic. And I think it would be good to keep BBR protocol-agnostic for as long as possible, to improve the odds of keeping implementations for various transports in sync.

Thanks for highlighting this issue. One possible idea for addressing the interactions between the DCCP Sequence window negotiation process and the BBR ProbeRTT mechanism would be as follows: instead of basing the sender's requested DCCP Sequence Number validity window strictly on the cwnd value, the sender could try to continually maintain a Sequence Number validity window  big enough to handle the anticipated volume of in-flight data in the "near future" For BBR in ProbeRTT, this would be something like the following, which would take into account both the current cwnd and the fact that upon exiting ProbeRTT the flow will expect to raise its cwnd back to something in the neighborhood of its previous cwnd before ProbeRTT:

In Linux TCP notation:

That could avoid the approach mentioned in the slides, of a "Pro-active update of local values even if the confirmation has not been received yet (feature negotiation not finished)".

Does that sound like a plausible approach?
NR:// I think it is a good solution. However, I would like to bring to consideration that according to  RFC4340 section 7.5.2<https://datatracker.ietf.org/doc/html/rfc4340#section-7.5.2>, keeping the Sequence number validity window too large, increases the risk of attacks. On the other hand, I think that the short duration of the probeRTT phase reduces significantly that risk, but I would like to hear other opinions about this potential security problem.

NR:// Apart from these two topics, I would like to bring another question related to the probeRTT phase.  In the CCID5 draft (section 7.1)<https://datatracker.ietf.org/doc/html/draft-romo-iccrg-ccid5#section-7.1>  I have also described that during the transition to and from the probeRTT phase, it is also required the synchronization of the ACK ratio feature. Right now, we are considering waiting for the confirmation of the ACK ratio negotiation before applying the reduction of the cwnd,  which would delay the start of the probeRTT phase (again, by at least one RTT). Unlike the problem that I had previously found with the Synchronization of the Sequence number validity window, I don’t see here an impact in the performance, but I would like to know if you see any problem with this approach.

Best regards,

Nathalie Romo Moreno

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 10 Nov 2021 09:56:27 -0500

On Wed, Nov 10, 2021 at 9:32 AM <[email-masked]> wrote:

> We have proposed to add the following text to the next rev of the BBR
> draft to reference your work, alongside the descriptions of TCP and QUIC
> implementations:
>
> ---
>
> NR:// I think it is a good solution. However, I would like to bring to
> consideration that according to  RFC4340 section 7.5.2
> <https://datatracker.ietf.org/doc/html/rfc4340#section-7.5.2>, keeping
> the Sequence number validity window too large, increases the risk of
> attacks. On the other hand, I think that the short duration of the probeRTT
> phase reduces significantly that risk, but I would like to hear other
> opinions about this potential security problem.
>

Since it's very common for applications to be application-limited, sending
small chunks of data much smaller than the cwnd, it will often be the case
that a sequence number validity window will be much larger than what is
used. Thus any additional risk from ProbeRTT mode leaving sequence number
validity window underutilized is miniscule relative to this effect,
particularly in BBRv2, where ProbeRTT cuts cwnd to 0.5x the estimated BDP
(applications routinely use less than 0x5x the BDP). In any case,
developers and users who care about security (which should be all of them)
should be using TLS/SSL/etc.

> NR:// Apart from these two topics, I would like to bring another question
> related to the probeRTT phase.  In the CCID5 draft (section 7.1)
> <https://datatracker.ietf.org/doc/html/draft-romo-iccrg-ccid5#section-7.1>
> I have also described that during the transition to and from the probeRTT
> phase, it is also required the synchronization of the ACK ratio feature.
> Right now, we are considering waiting for the confirmation of the ACK ratio
> negotiation before applying the reduction of the cwnd,  which would delay
> the start of the probeRTT phase (again, by at least one RTT). Unlike the
> problem that I had previously found with the Synchronization of the
> Sequence number validity window, I don’t see here an impact in the
> performance, but I would like to know if you see any problem with this
> approach.
>

In ProbeRTT the goal is to quickly reduce the volume of inflight data in
order to greatly increase the odds of measuring the unloaded two-way
propagation delay. As such, it's important to cut the cwnd quickly, so my
sense is that it's not a good idea to wait the extra round trip to get the
confirmation. It seems better to request the new ACK ratio, optimistically
presume that 99% of the time the new ACK ratio will reach the receiver and
be accepted, then cut the cwnd immediately. If the ACK ratio does not make
it to the other side, then that's likely to be due to congestion, which
just increases the importance of cutting the cwnd ASAP rather than waiting.

I see  "Change options are retransmitted until some response is received." i
n:
  https://datatracker.ietf.org/doc/html/rfc4340#section-6.1
For my education, is it thus correct that a DCCP/BBR sender could just send
an ACK ratio request with every data segment sent in ProbeRTT, to
practically ensure that the reduced volume of data is properly ACKed in
ProbeRTT, without waiting for confirmation?

best regards,
neal

---

From: <[email-masked]>
Date: Thu, 11 Nov 2021 11:13:15 +0000

Hello Neal,

I wrote just one comment at the end to clarify better the problem.

On Wed, Nov 10, 2021 at 9:32 AM <[email-masked]<mailto:[email-masked]>> wrote:
Hello Neal,

Dear all,

In Linux TCP notation:

Since it's very common for applications to be application-limited, sending small chunks of data much smaller than the cwnd, it will often be the case that a sequence number validity window will be much larger than what is used. Thus any additional risk from ProbeRTT mode leaving sequence number validity window underutilized is miniscule relative to this effect, particularly in BBRv2, where ProbeRTT cuts cwnd to 0.5x the estimated BDP (applications routinely use less than 0x5x the BDP). In any case, developers and users who care about security (which should be all of them) should be using TLS/SSL/etc.

In ProbeRTT the goal is to quickly reduce the volume of inflight data in order to greatly increase the odds of measuring the unloaded two-way propagation delay. As such, it's important to cut the cwnd quickly, so my sense is that it's not a good idea to wait the extra round trip to get the confirmation. It seems better to request the new ACK ratio, optimistically presume that 99% of the time the new ACK ratio will reach the receiver and be accepted, then cut the cwnd immediately. If the ACK ratio does not make it to the other side, then that's likely to be due to congestion, which just increases the importance of cutting the cwnd ASAP rather than waiting.

I see  "Change options are retransmitted until some response is received." in:
  https://datatracker.ietf.org/doc/html/rfc4340#section-6.1
For my education, is it thus correct that a DCCP/BBR sender could just send an ACK ratio request with every data segment sent in ProbeRTT, to practically ensure that the reduced volume of data is properly ACKed in ProbeRTT, without waiting for confirmation?

NR:// That is true. The problem is, at the beginning of the probeRTT phase when the cwnd is cut, we have to wait until the amount of packets in flight is reduced to be able to send new packets with the ACK ratio update.  The amount of packets in flight will only be reduced if we receive an ACK, which will not happen if the ACK ratio is too high. In that scenario, the new packets with the ACK ratio update will only be sent after the sender RTO timer expires. Our initial solution to this problem was to send at least one packet with the ACK ratio update before the cwnd is cut, but if this packet doesn’t reach the receiver, then we have to wait again for the RTO timer to expire, before any new data can be transmitted. That’s why we consider that ensure that the ACK ratio value has been updated at the receiver before the cut of the cwnd, could be a better approach. This will not change the duration or the principle of the probeRTT phase, but just delay its start by one RTT.

Best regards,

Nathalie Romo Moreno