Subject: [rmcat] Congestion Control Behavior Suggestion Thread

From: Dan Weber <[email-masked]>
Date: Wed, 13 Mar 2013 22:20:46 -0600

Hi there,

This is the  first and example behavior test case based on the concept I
proposed in "RMCAT Evaluation Ideas".  Please add your own following the
same format.  I seek to aggregate all of these test cases onto a wiki and
establish categories for them.

Given: A video media stream sends at 512kbit/s on a link with an upstream
capacity of 512 kbit/s
When: an FTP Transfer of 1GB file begins uploading
Then: Packet loss occurs.  Throughput should be diminished.  RTPCC should
notify application of developments.

Given: A link with capacity limited at 512 kbit/s
When: A video conference call starts sending Mulaw audio (ptime=20,
81.92kbit/s) and Video @ max kbitrate of 384 kbit/s
Then:  All packets should be received.

...

This may need more work, but it can get us going in the right direction.

Thanks,
Dan

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 14 Mar 2013 11:34:11 +0000

Hi,

> This may need more work, but it can get us going in the right direction.

thanks for starting this! Yes, there needs to be more detail. In case folks haven't see it yet, we tried to specify such scenarios for doing performance measurements for TCP extensions a few years ago: http://eggert.org/papers/2008-pfldnet.pdf

The intent with that paper, and I guess the intent now, would be to nail down all constants sufficiently and describe the parameterization of each test sufficiently so that someone could set up a simulation or experiment based only on the text. (For the TCP scenarios described in the paper, someone - from NEC? - actually did provide an ns2 script.)

Lars

---

From: Varun Singh <[email-masked]>
Date: Thu, 14 Mar 2013 09:34:44 -0400

Hi Dan,

> Then:  All packets should be received.

The "Then" part may depend also on what other network characteristics
may be applied (path loss and/or delay).

-- 
http://www.netlab.tkk.fi/~varun/

---

From: Dan Weber <[email-masked]>
Date: Thu, 14 Mar 2013 11:35:43 -0600

Hi Varun,

On Thu, Mar 14, 2013 at 7:34 AM, Varun Singh <[email-masked]> wrote:

> I concur.  Even putting together these surprisingly simple behavior cases
is difficult.  Every single one of these cases needs to be tangible and
easily implementable.

Your specific note carries relevance.  How do you define the network
environment?  Linux's Network Emulation (netem,
http://www.linuxfoundation.org/collaborate/workgroups/networking/netem)
provides a very valuable toolset for this sort of thing.

Do you think it would make sense to use its capabilities for defining our
'given' case?

I'm looking forward to reading Lars's paper to see if it may provide some
more insight into this matter.

Thanks,
Dan

---

From: Varun Singh <[email-masked]>
Date: Thu, 14 Mar 2013 14:00:30 -0400

On 14.3.2013, at 13.35, Dan Weber <[email-masked]> wrote:

Hi Varun,

My take for the cc-eval was to define some of these characteristics as
variables. Then the implementer can pick values for a variable from a set.

This will create a combinatorial explosion of tests for each scenario, but
I hope we can attempt to constrain the number of variables in each
scenario.

I have used both netem, and dummynet.

However there is netkit, AutoNetkit. Both of these have been recommended
 but I haven't used either for RTP.

Typically the tools allow setting delay, throughput, packet loss rate,
queue length. (The list may be longer).

However, packet loss rate is random, which doesn't work well for some of
our tests.

For dummynet, I have a wrapper written up for changing some of these
params.
See
https://github.com/vr000m/Research-Scripts/tree/master/dummynet

Thanks,
Dan

---

From: "Bill Ver Steeg (versteb)" <[email-masked]>
Date: Thu, 14 Mar 2013 18:04:33 +0000

netem is a very good emulation tool, and we often use it to get close to the right answer. However, I will caution that there is a wide variety of packet schedulers in use in the wild. Not all of them the exhibit flat, deterministic scheduling that netem defaults to.

One (of many) example of such a packet scheduling/limiting algorithm was described in yesterday's congestion control research group. HFC CPE (cable modems) use a leaky bucket algorithm that provides a weighted average to enforce an SLA, and this can interact with higher level control loops. In other words, if a sender is slightly under the provisioned link rate for a while, it can then can send at slightly over the link rate for a while. If you are not careful, a variable bitrate sender can infer an incorrect SLA rate based on such scheduler behavior. I do not imply that this is the definitive case, just that there can be interactions. Flow control algorithms that work very well for a given scheduler can have unexpected entanglements with another (chain of) schedulers.

Fortunately, there are ways to introduce these behaviors in test rigs.

So, if we define a reference test environment, we either need to have a few scheduler variations or recognize that the test rig is only a first-order approximation and live testing on a variety of network topologies is required.

Short version-  I agree that netem is a very good tool, but you need to be careful how you use it.

Bvs

On Mar 14, 2013, at 1:35 PM, "Dan Weber" <[email-masked]<mailto:[email-masked]>> wrote:

Hi Varun,

On Thu, Mar 14, 2013 at 7:34 AM, Varun Singh <[email-masked]<mailto:[email-masked]>> wrote:
Hi Dan,

I concur.  Even putting together these surprisingly simple behavior cases is difficult.  Every single one of these cases needs to be tangible and easily implementable.

Your specific note carries relevance.  How do you define the network environment?  Linux's Network Emulation (netem, http://www.linuxfoundation.org/collaborate/workgroups/networking/netem) provides a very valuable toolset for this sort of thing.

Do you think it would make sense to use its capabilities for defining our 'given' case?

I'm looking forward to reading Lars's paper to see if it may provide some more insight into this matter.

Thanks,
Dan

---

From: Dan Weber <[email-masked]>
Date: Thu, 14 Mar 2013 12:15:30 -0600

Hi Bill,

I understand your concerns about netem.  It's no panacea.

I'm shooting to define a table of configurable parameters to go with each
test case.

E.g.
Ploss 1% Correlation (90%)
Delay 80ms Jitter 50ms
Bandwidth 2mbit/s
etc.

It seems to me that this is required regardless of the tools we use.  If we
can standardize on which parameters are adjustable, we can include them in
every test case and use a variety of tools.  As for defining a reference
test environment, does anyone have access to some systems we could open up
for the purpose of this testing?  I may be able to call around and see if
we can get one provided for us if no one else has resources available.

Thanks,
Dan

On Thu, Mar 14, 2013 at 12:04 PM, Bill Ver Steeg (versteb) <
[email-masked]> wrote:

>  netem is a very good emulation tool, and we often use it to get close to
> the right answer. However, I will caution that there is a wide variety of
> packet schedulers in use in the wild. Not all of them the exhibit flat,
> deterministic scheduling that netem defaults to.
>
>  One (of many) example of such a packet scheduling/limiting algorithm was
> described in yesterday's congestion control research group. HFC CPE (cable
> modems) use a leaky bucket algorithm that provides a weighted average to
> enforce an SLA, and this can interact with higher level control loops. In
> other words, if a sender is slightly under the provisioned link rate for a
> while, it can then can send at slightly over the link rate for a while. If
> you are not careful, a variable bitrate sender can infer an incorrect SLA
> rate based on such scheduler behavior. I do not imply that this is the
> definitive case, just that there can be interactions. Flow control
> algorithms that work very well for a given scheduler can have unexpected
> entanglements with another (chain of) schedulers.
>
>  So, if we define a reference test environment, we either need to have a
> few scheduler variations or recognize that the test rig is only a
> first-order approximation and live testing on a variety of network
> topologies is required.
>

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 14 Mar 2013 18:21:36 +0000

Hi,

one thing to keep in mind is that we don't have to describe a fully exhaustive list of tests in eval-criteria, esp. initially.

If we describe a few and the folks pushing solution candidates ran these, that'd be great. They should also run other tests, and some of them in turn should make it into a future revision of eval-criteria.

I do (again) want to encourage the folks that are proposing one solution candidate to also try to compare theirs against the others directly in one measurement setup, instead of everyone only bringing their set of measurements and then we all trust that the setups were really fully comparable. Esp. great would be if folks actually implemented each other's candidates instead of just running the other's code. This way, we'd have way ore confidence that the respective specs were complete and correct.

Lars

---

From: Dan Weber <[email-masked]>
Date: Thu, 14 Mar 2013 12:24:27 -0600

Hi Varun,

On Thu, Mar 14, 2013 at 12:00 PM, Varun Singh <[email-masked]> wrote:

> On Thu, Mar 14, 2013 at 7:34 AM, Varun Singh <[email-masked]>wrote:
>
>> Hi Dan,
>>

I'm not sure I understand exactly what you mean by this.  I do understand
how a combinatorial explosion can occur with countless parameters.
However, this problem mirrors ones used in other fields.  I'm not sure if
you're familiar with Genichi Taguchi but he defined the methods that are
the backbone of most popular total quality management systems.  His
solution to this particular problem is the use of an orthogonal array.  You
might enjoy reading his article on their use: "Robust Quality"
http://offermaxima.com/pdfs/RobustQuality.pdf

> I have not used dummynet, netkit, or autonetkit, but I'm always interested
in trying new things.  Is there a basic set of parameters that apply to all
of them?

> However, packet loss rate is random, which doesn't work well for some of
Netem does allow for packet loss with correlation so that you can generate
burst packet loss, although the distribution is still primarily random.
However, that may be reasonably good enough if we get the rest of the
packet loss from actually limiting the link capacity.

> I'll check this out when I get a chance.

Thanks,
Dan

---

From: "Bill Ver Steeg (versteb)" <[email-masked]>
Date: Thu, 14 Mar 2013 18:30:37 +0000

In an ns2 environment, having a physical test rig is mute. Quite a bit of progress can be made in this domain. A simple tar file of the complete ns2 test rig would make for results that can be verified, tweaked and retested.

Once we move to a real device, it seems that Linux sources and sinks with netem-like impairments would work. I used to have a pretty good test rig, but I loaned it to GaTech last year.......

I do think that a well defined test rig would help in this effort, and also in the broader congestion work underway at IETF. Getting gear is easier than staff to run it. If I can get a coop student this summer, maybe I can cobble some gear together and stand up a test rig somewhere that is publicly accessible.

Bvs

On Mar 14, 2013, at 2:15 PM, "Dan Weber" <[email-masked]<mailto:[email-masked]>> wrote:

Hi Bill,

I'm shooting to define a table of configurable parameters to go with each test case.

E.g.

It seems to me that this is required regardless of the tools we use.  If we can standardize on which parameters are adjustable, we can include them in every test case and use a variety of tools.  As for defining a reference test environment, does anyone have access to some systems we could open up for the purpose of this testing?  I may be able to call around and see if we can get one provided for us if no one else has resources available.

Thanks,
Dan

On Thu, Mar 14, 2013 at 12:04 PM, Bill Ver Steeg (versteb) <[email-masked]<mailto:[email-masked]>> wrote:
netem is a very good emulation tool, and we often use it to get close to the right answer. However, I will caution that there is a wide variety of packet schedulers in use in the wild. Not all of them the exhibit flat, deterministic scheduling that netem defaults to.

Bvs

---

From: Dan Weber <[email-masked]>
Date: Thu, 14 Mar 2013 12:38:53 -0600

Hey Bill,

I'm not familiar with ns2, but I am in favor of having a complete test rig
that everyone can download.

Does it work with existing applications?

Thanks,
Dan

On Thu, Mar 14, 2013 at 12:30 PM, Bill Ver Steeg (versteb) <
[email-masked]> wrote:

>  In an ns2 environment, having a physical test rig is mute. Quite a bit
> of progress can be made in this domain. A simple tar file of the complete
> ns2 test rig would make for results that can be verified, tweaked and
> retested.
>
>  Once we move to a real device, it seems that Linux sources and sinks
> with netem-like impairments would work. I used to have a pretty good test
> rig, but I loaned it to GaTech last year.......
>
>  I do think that a well defined test rig would help in this effort, and
> also in the broader congestion work underway at IETF. Getting gear is
> easier than staff to run it. If I can get a coop student this summer, maybe
> I can cobble some gear together and stand up a test rig somewhere that is
> publicly accessible.
>
>
>  Bvs
>
> On Mar 14, 2013, at 2:15 PM, "Dan Weber" <[email-masked]> wrote:
>
>  Hi Bill,
>
> It seems to me that this is required regardless of the tools we use.  If
> we can standardize on which parameters are adjustable, we can include them
> in every test case and use a variety of tools.  As for defining a reference
>>  netem is a very good emulation tool, and we often use it to get close
>> to the right answer. However, I will caution that there is a wide variety
>> of packet schedulers in use in the wild. Not all of them the exhibit flat,
>>  One (of many) example of such a packet scheduling/limiting algorithm
>> was described in yesterday's congestion control research group. HFC CPE
>> (cable modems) use a leaky bucket algorithm that provides a weighted
>> average to enforce an SLA, and this can interact with higher level control
>> loops. In other words, if a sender is slightly under the provisioned link
>> rate for a while, it can then can send at slightly over the link rate for a
>> while. If you are not careful, a variable bitrate sender can infer an
>> incorrect SLA rate based on such scheduler behavior. I do not imply that
>> this is the definitive case, just that there can be interactions. Flow
>> control algorithms that work very well for a given scheduler can have
>> unexpected entanglements with another (chain of) schedulers.
>>

---

From: Dan Weber <[email-masked]>
Date: Thu, 14 Mar 2013 12:43:14 -0600

This paper is excellent.  Thanks for sharing.

Do you have a copy of the NS2 script we could use for inspiration?

Thanks,
Dan

On Thu, Mar 14, 2013 at 5:34 AM, Eggert, Lars <[email-masked]> wrote:

> thanks for starting this! Yes, there needs to be more detail. In case
> folks haven't see it yet, we tried to specify such scenarios for doing
> performance measurements for TCP extensions a few years ago:
> http://eggert.org/papers/2008-pfldnet.pdf
>
> The intent with that paper, and I guess the intent now, would be to nail
> down all constants sufficiently and describe the parameterization of each
> test sufficiently so that someone could set up a simulation or experiment
> based only on the text. (For the TCP scenarios described in the paper,
> someone - from NEC? - actually did provide an ns2 script.)
>
> Lars

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 14 Mar 2013 18:46:36 +0000

> Do you have a copy of the NS2 script we could use for inspiration?

http://sourceforge.net/projects/tcpeval/

Lars

---

From: Jim Gettys <[email-masked]>
Date: Thu, 14 Mar 2013 20:08:36 -0400

And note that netem needs to be run on a separate system.

Jim
On Mar 14, 2013 2:30 PM, "Bill Ver Steeg (versteb)" <[email-masked]>

---

From: Fu Jiantao <[email-masked]>
Date: Fri, 15 Mar 2013 15:14:15 +0800

We use tc extensively in our test, I think this may be helpful, see here:
http://www.slideshare.net/fuji246/network-emulator