Subject: Re: [aqm] Follow-up: PIE performance in cable modem environments

From: Greg White <[email-masked]>
Date: Fri, 3 May 2013 22:55:02 +0000

Preethi,

I did a diff on the new version of the PIE code vs the earlier release and found, as you stated, that the basic algorithm hasn't changed.  I do see some other improvements/changes though.  One which, I think, bears discussion is the weighting of packet drops based on size.

In the new code there is the option (turned on in the simulations I reported on in my paper) to drop packets according to p*pkt_size/mean_pkt_size where p is the drop probability calculated by the PIE control law. This serves to significantly decrease the drop probability of the small VoIP and gaming packets, which may be sensitive to loss (from a QoE perspective) and non-responsive to loss (from a congestion control perspective).  This is evidenced by the low packet loss rate for gaming traffic that I reported in my paper.

However, I worry that the unintended consequence may be that this weighting incentivizes application developers toward the use of small packets, which seems ill advised.

Thoughts?

-Greg

From: Preethi Natarajan <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 30, 2013 5:48 PM
To: Greg White <[email-masked]<mailto:[email-masked]>>, Preethi Natarajan <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: "Rong Pan (ropan)" <[email-masked]<mailto:[email-masked]>>, "Bill Ver Steeg (versteb)" <[email-masked]<mailto:[email-masked]>>, "Chiara Piglione (cpiglion)" <[email-masked]<mailto:[email-masked]>>, "Mythili Suryanarayana Prabhu (mysuryan)" <[email-masked]<mailto:[email-masked]>>, "Fred Baker (fred)" <[email-masked]<mailto:[email-masked]>>, Dan Rice <[email-masked]<mailto:[email-masked]>>
Subject: Re: [aqm] Follow-up: PIE performance in cable modem environments

Hi,

Thank you Greg for the update and the link to the white paper.

We wanted to quickly clarify about how we tuned PIE for DOCSIS.

The basic PIE algorithm has not changed. We updated the simulation code with the missing line mentioned below (which was a bug). The DOCSIS MAC layer has this special nature of stop-and-go with 5ms-6ms request and grant delay. This requires adjustment of any algorithm: for example  CoDel has to increase its target delay from 5ms to a higher value. Similarly, our new parameters are to make PIE adjust faster for the DOCSIS stop-and-go behavior. Please note that eventually all these design parameters will be automatically set, users of the PIE algorithm would not be required to set any design parameters.

Again, many thanks for your update.
PIE team

From: Greg White <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 30, 2013 3:54 PM
To: Preethi Natarajan <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: "Rong Pan (ropan)" <[email-masked]<mailto:[email-masked]>>, "Bill Ver Steeg (versteb)" <[email-masked]<mailto:[email-masked]>>, "Chiara Piglione (cpiglion)" <[email-masked]<mailto:[email-masked]>>, "Mythili Suryanarayana Prabhu (mysuryan)" <[email-masked]<mailto:[email-masked]>>, "Fred Baker (fred)" <[email-masked]<mailto:[email-masked]>>, Daniel Rice <[email-masked]<mailto:[email-masked]>>
Subject: Re: [aqm] Follow-up: PIE performance in cable modem environments

Additionally, I've re-run my suite of simulations using the updated PIE code from Cisco.  The results (in much more detail than I presented at ICCRG) are documented in a white paper available here:
Active Queue Management Algorithms for DOCSIS 3.0<http://www.cablelabs.com/downloads/pubs/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf>

Thanks to Preethi, Rong, et al. for debugging and tuning PIE to work well in the cable environment, and for sharing the resulting code.

Best Regards,
Greg

From: Preethi Natarajan <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 23, 2013 5:18 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: "Rong Pan (ropan)" <[email-masked]<mailto:[email-masked]>>, "Bill Ver Steeg (versteb)" <[email-masked]<mailto:[email-masked]>>, "Chiara Piglione (cpiglion)" <[email-masked]<mailto:[email-masked]>>, "Mythili Suryanarayana Prabhu (mysuryan)" <[email-masked]<mailto:[email-masked]>>, "Fred Baker (fred)" <[email-masked]<mailto:[email-masked]>>
Subject: [aqm] Follow-up: PIE performance in cable modem environments

Hello,

This is a follow-up to Greg White's (from Cable Labs) talk at the recent ICCRG meeting on PIE's performance in cable modem environments.

Post the meeting, Greg was kind to share his ns-2 DOCSIS model with us. We investigated PIE's performance using this model. The key items from this investigation:

  1.  Bug in PIE code: The previous PIE release (that Greg used for evaluations) was missing a line of code. This missing line brings down drop probability under certain conditions and turns out to be critical for the cable modem scenario. Without this line of code, the drop probability remains high and takes longer to come down even when the queue delay has remained lower than the reference. The updated ns-2 PIE code can be found here — ftp://ftpeng.cisco.com/pie/.
  2.  Bug in ns-2 TCP/Linux: Greg's cable modem simulations used the TCP Cubic variant. We discovered a serious bug in ns-2 TCP/Linux Agent (confirmed by Dr. Injong Rhee's team) that makes TCP/Cubic senders very aggressive and unresponsive to packet drops/notifications, pretty much like UDP traffic. Please find more details about the bug here -- http://sourceforge.net/tracker/?func=detail&aid=3608750&group_id=149743&atid=775392.

We are working with Cable Labs to verify the cable modem results, they'll soon be available on our FTP site along with the PIE code.

A technical paper about PIE was recently accepted at the IEEE Conference on High Performance Switching and Routing 2013. A copy of the paper is attached here.

The Linux PIE implementation is expected to be ready by next week and we'll follow-up on that as well.

Many thanks,
Preethi on behalf of PIE team.

---

From: John Leslie <[email-masked]>
Date: Fri, 3 May 2013 19:53:45 -0400

Greg White <[email-masked]> wrote:
> 
> I did a diff on the new version of the PIE code vs the earlier release
> and found, as you stated, that the basic algorithm hasn't changed.
> I do see some other improvements/changes though. One which, I think,
> bears discussion is the weighting of packet drops based on size.
> 
> In the new code there is the option (turned on in the simulations I
> reported on in my paper) to drop packets according to
> p*pkt_size/mean_pkt_size where p is the drop probability calculated by
> the PIE control law. This serves to significantly decrease the drop
> probability of the small VoIP and gaming packets, which may be
> sensitive to loss (from a QoE perspective) and non-responsive to loss
> (from a congestion control perspective).  This is evidenced by the low
> packet loss rate for gaming traffic that I reported in my paper.
> 
> However, I worry that the unintended consequence may be that this
> weighting incentivizes application developers toward the use of small
> packets, which seems ill advised.

   The IESG has approved for publication as an RFC:

http://datatracker.ietf.org/doc/draft-ietf-tsvwg-byte-pkt-congest/

which, basically, says "Don't do that".

--
John Leslie<[email-masked]>

---

From: Dave Taht <[email-masked]>
Date: Sat, 4 May 2013 02:09:43 -0700

On Fri, May 3, 2013 at 3:55 PM, Greg White <[email-masked]> wrote:
> Preethi,
>
> I did a diff on the new version of the PIE code vs the earlier release and
> found, as you stated, that the basic algorithm hasn't changed.  I do see
> some other improvements/changes though.  One which, I think, bears
> discussion is the weighting of packet drops based on size.
>
> In the new code there is the option (turned on in the simulations I reported
> on in my paper) to drop packets according to p*pkt_size/mean_pkt_size where
> p is the drop probability calculated by the PIE control law. This serves to
> significantly decrease the drop probability of the small VoIP and gaming
> packets, which may be sensitive to loss (from a QoE perspective) and
> non-responsive to loss (from a congestion control perspective).  This is
> evidenced by the low packet loss rate for gaming traffic that I reported in
> my paper.
>
> However, I worry that the unintended consequence may be that this weighting
> incentivizes application developers toward the use of small packets, which
> seems ill advised.
>
> Thoughts?

Optimizing a little bit for smaller packets is an old trick that goes
back to wondershaper, and most likely prior.

Personally, I'd have liked it if the internet MTU had stayed below 580
until the world was running universally at above 4Mbits.

As for how PIE does it, I haven't got the linux sources yet, so I
haven't looked at it.

the *fq_codel settings used in your study used a quantum of 300, which
has much the same effect as however PIE is doing it, most likely.

You can quibble as to how much to de-optimize for big packets, but it
is generally a fairly reliable indicator of bulk vs ack or interactive
traffic,
and optimizing for the smaller packets is very useful on asymmetric
low speed uplinks. It seems like an increasingly less useful
optimization as speeds get higher and higher.

There are some newer interactive protocols (videoconferencing comes to
mind) where packet sizes of larger than 300 bytes seem possible, and
DNSSEC bloats up dns packets - but for the vast majority of other
sorts of traffic, acks, syns, normal dns, dhcp, voip, gaming, etc,
optimizing for size a little bit is a win.

As for disincentivizing the use of big packets, there are plenty of
reasons for bulk transfers to try and send the biggest packet
possible, and even if an app did start sending smaller packets the
per-packet overhead of IP is increasingly trivial above 300 bytes, and
in general smaller packets are better for the sharing of the network,
anyway...

>From wikipedia:

"When data is formatted into packets, the bitrate of the communication
medium can be better shared among users than if the network were
circuit switched."

>
> -Greg
>
>
>
> From: Preethi Natarajan <[email-masked]>
> To: Greg White <[email-masked]>, Preethi Natarajan
> <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]"
> <[email-masked]>, "[email-masked]" <[email-masked]>
> Cc: "Rong Pan (ropan)" <[email-masked]>, "Bill Ver Steeg (versteb)"
> <[email-masked]>, "Chiara Piglione (cpiglion)" <[email-masked]>,
> "Mythili Suryanarayana Prabhu (mysuryan)" <[email-masked]>, "Fred Baker
> (fred)" <[email-masked]>, Dan Rice <[email-masked]>
>
> The basic PIE algorithm has not changed. We updated the simulation code with
> the missing line mentioned below (which was a bug). The DOCSIS MAC layer has
> this special nature of stop-and-go with 5ms-6ms request and grant delay.
> This requires adjustment of any algorithm: for example  CoDel has to
> increase its target delay from 5ms to a higher value. Similarly, our new
> parameters are to make PIE adjust faster for the DOCSIS stop-and-go
> behavior. Please note that eventually all these design parameters will be
> automatically set, users of the PIE algorithm would not be required to set
> any design parameters.
>
> From: Greg White <[email-masked]>
> To: Preethi Natarajan <[email-masked]>, "[email-masked]"
> <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]"
> <[email-masked]>
> Cc: "Rong Pan (ropan)" <[email-masked]>, "Bill Ver Steeg (versteb)"
> <[email-masked]>, "Chiara Piglione (cpiglion)" <[email-masked]>,
> "Mythili Suryanarayana Prabhu (mysuryan)" <[email-masked]>, "Fred Baker
> (fred)" <[email-masked]>, Daniel Rice <[email-masked]>
> Additionally, I've re-run my suite of simulations using the updated PIE code
> from Cisco.  The results (in much more detail than I presented at ICCRG) are
> documented in a white paper available here:
> Active Queue Management Algorithms for DOCSIS 3.0
>
> Thanks to Preethi, Rong, et al. for debugging and tuning PIE to work well in
> the cable environment, and for sharing the resulting code.
>
> Best Regards,
> Greg
>
> From: Preethi Natarajan <[email-masked]>
> To: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>,
> "[email-masked]" <[email-masked]>
> Cc: "Rong Pan (ropan)" <[email-masked]>, "Bill Ver Steeg (versteb)"
> <[email-masked]>, "Chiara Piglione (cpiglion)" <[email-masked]>,
> "Mythili Suryanarayana Prabhu (mysuryan)" <[email-masked]>, "Fred Baker
> (fred)" <[email-masked]>
> This is a follow-up to Greg White's (from Cable Labs) talk at the recent
> ICCRG meeting on PIE's performance in cable modem environments.
>
> Post the meeting, Greg was kind to share his ns-2 DOCSIS model with us. We
> investigated PIE's performance using this model. The key items from this
> investigation:
>
> Bug in PIE code: The previous PIE release (that Greg used for evaluations)
> was missing a line of code. This missing line brings down drop probability
> under certain conditions and turns out to be critical for the cable modem
> scenario. Without this line of code, the drop probability remains high and
> takes longer to come down even when the queue delay has remained lower than
> the reference. The updated ns-2 PIE code can be found here —
> ftp://ftpeng.cisco.com/pie/.
> Bug in ns-2 TCP/Linux: Greg's cable modem simulations used the TCP Cubic
> variant. We discovered a serious bug in ns-2 TCP/Linux Agent (confirmed by
> Dr. Injong Rhee's team) that makes TCP/Cubic senders very aggressive and
> unresponsive to packet drops/notifications, pretty much like UDP traffic.
> Please find more details about the bug here --
> http://sourceforge.net/tracker/?func=detail&aid=3608750&group_id=149743&atid=775392.
>
> We are working with Cable Labs to verify the cable modem results, they'll
> soon be available on our FTP site along with the PIE code.
>
> A technical paper about PIE was recently accepted at the IEEE Conference on
> High Performance Switching and Routing 2013. A copy of the paper is attached
> here.
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: grenville armitage <[email-masked]>
Date: Sat, 04 May 2013 19:45:02 +1000

On 05/04/2013 19:09, Dave Taht wrote:
	[..]
> until the world was running universally at above 4Mbits.

IP over ATM, FTW ;)

cheers,
gja

---

From: Greg White <[email-masked]>
Date: Tue, 30 Apr 2013 22:54:44 +0000

Best Regards,
Greg

Hello,

Many thanks,

---

From: Preethi Natarajan <[email-masked]>
Date: Tue, 30 Apr 2013 16:48:04 -0700

Hi,

<http://www.cablelabs.com/downloads/pubs/Active_Queue_Management_Algorithms_
DOCSIS_3_0.pdf> 

Best Regards,
Greg

Hello,

Post the meeting, Greg was kind to share his ns-2 DOCSIS model with us. We
investigated PIE's performance using this model. The key items from this
investigation: 
1. Bug in PIE code: The previous PIE release (that Greg used for
evaluations) was missing a line of code. This missing line brings down drop
probability under certain conditions and turns out to be critical for the
cable modem scenario. Without this line of code, the drop probability
remains high and takes longer to come down even when the queue delay has
remained lower than the reference. The updated ns-2 PIE code can be found
here ‹ ftp://ftpeng.cisco.com/pie/.
2. Bug in ns-2 TCP/Linux: Greg's cable modem simulations used the TCP Cubic
variant. We discovered a serious bug in ns-2 TCP/Linux Agent (confirmed by
Dr. Injong Rhee's team) that makes TCP/Cubic senders very aggressive and
unresponsive to packet drops/notifications, pretty much like UDP traffic.
Please find more details about the bug here --
http://sourceforge.net/tracker/?func=detail&aid=3608750&group_id=149743&atid
=775392.
We are working with Cable Labs to verify the cable modem results, they'll
soon be available on our FTP site along with the PIE code.

The Linux PIE implementation is expected to be ready by next week and we'll
follow-up on that as well.

Many thanks,

---

From: Preethi Natarajan <[email-masked]>
Date: Tue, 07 May 2013 16:28:21 -0700

Hi Greg,

As other posts point out, the byte-mode operation has been around for a
while and it is not unique/new to PIE.

We weren't aware of Bob Briscoe's BCP draft until recently. We understand
the concerns mentioned here. In fact, we encountered the DropTail's
preferential treatment for small packets in our simulations (mentioned in
the BCP draft) and that was our motivation to  experiment with byte-mode.

We think that a good alternative to byte-mode would be ToS/DiffServe-field
based differential dropping, similar to WRED, since that would give similar
benefits as byte-mode. That way, we don't have to stick to byte-mode based
approach for PIE.

Regards,
The PIE Team

From:  Greg White <[email-masked]>
Date:  Friday, May 3, 2013 3:55 PM
To:  Preethi Natarajan <[email-masked]>, "[email-masked]"
<[email-masked]>, "[email-masked]" <[email-masked]>
Subject:  Re: [aqm] Follow-up: PIE performance in cable modem environments

Preethi,

Thoughts?

-Greg

Hi,

Best Regards,
Greg

Hello,

Many thanks,