Subject: [rmcat] RMCAT Evaluation Ideas

From: Dan Weber <[email-masked]>
Date: Wed, 13 Mar 2013 21:57:49 -0600

Hi everyone,

While I sat on the airplane today, it became clear to me that there is a
reasonable way to do some evaluation of congestion control algorithms that
can be both indicative of real use cases (e.g. live streaming video) while
also being easy to test (e.g. fixed rate packet generators).

Three ideas here:

1. Test cases
We will never agree on what a definition of fairness is, and I think it is
a lost cause.  However, we can definitely reach consensus on expected
behavior given various test cases.

Behavioral driven design gives us a great method for defining these.  It
follows the form (given the facts (concrete verifiable constants), when
action or event occurs, then I expect it to do  XYZ.).  I will start a
separate thread kicking this off.

2. Packet Generator Model
After re-reading Varun's draft and speaking with him on the evaluation
scenarios, I think it makes enough sense to build a packet generator to
implement what he suggests.  The time involved isn't substantial, and it
can help us get a unified starting point for testing even if we decide to
change it later.  It's worth at least this much.

My proposed implementation of this will be able to simulate real video
traffic as well as fixed packet generator traffic.  For the real video
traffic, it can be fed a CSV file filled with frame sizes and RTP
timestamps.  For the fixed generator traffic, ladder steps can be used.
 I'm thinking I'll start a project on Google Code, and make any interested
participant a developer.

3. Evaluation Metrics

We need an initial set of metrics for testing any congestion control
algorithm.  To evaluate any congestion control algorithm, we'll want enough
data to build our favorite line charts.  For simplicity, all metrics should
be sampled per second (maybe per millisecond?).  This also allows us to
work with real test environments easily.

Here is my initial and non-exhaustive list of metrics (based off of the
draft):

[Received] In Kbit/s
[Sent] Out Kbit/s
Packets sent/second,
Packets received/second,
Packets lost/second [Determined by using standard RTP packet loss
calculations]
RTT/s
Discards per second [ use a range of buffer calculations, 20ms 40ms, 100ms,
200ms, 500ms]

These metrics should be built into an RTP stack that we standardize for
testing.  My two preferences are jrtplib (C++) and libjitsi (Java).  At
least with both of these, we can contact the authors and get any bugs fixed
if we need them relatively easily.

Thanks,
Dan

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Thu, 14 Mar 2013 10:04:26 +0100

Hi Dan,

I made a testing tool to test my rate adaptation algorithm 
implementation in my MCU. You can find the source code here:

http://sourceforge.net/p/mcumediaserver/code/745/tree/trunk/cca/

It consist on a random packet generator (which can be parametrized to 
emulate certain video behaviors, like different sizes for I/P frames) 
and a receiever. I implementes a full RTP/RTCP stack with preliminary 
support for NACK and FEC (which I think it is essential for evaluating 
the behavior of the algorithm).

The current algorithm implemented is based on google's congestion 
control algorithm, but it could be also interesting to create a comon 
interface so we could test different algorithms and compare between them.

Best regards
Sergio

El 14/03/2013 4:57, Dan Weber escribió:
> Hi everyone,
>
> While I sat on the airplane today, it became clear to me that there is 
> a reasonable way to do some evaluation of congestion control 
> algorithms that can be both indicative of real use cases (e.g. live 
> streaming video) while also being easy to test (e.g. fixed rate packet 
> generators).
>
> Three ideas here:
>
> 1. Test cases
> We will never agree on what a definition of fairness is, and I think 
> it is a lost cause.  However, we can definitely reach consensus on 
> expected behavior given various test cases.
>
> Behavioral driven design gives us a great method for defining these. 
>  It follows the form (given the facts (concrete verifiable constants), 
> when action or event occurs, then I expect it to do  XYZ.).  I will 
> start a separate thread kicking this off.
>
>
> scenarios, I think it makes enough sense to build a packet generator 
> to implement what he suggests.  The time involved isn't substantial, 
> and it can help us get a unified starting point for testing even if we 
> decide to change it later.  It's worth at least this much.
>
> timestamps.  For the fixed generator traffic, ladder steps can be 
> used.  I'm thinking I'll start a project on Google Code, and make any 
> interested participant a developer.
>
>
> 3. Evaluation Metrics
>
> algorithm.  To evaluate any congestion control algorithm, we'll want 
> enough data to build our favorite line charts.  For simplicity, all 
> metrics should be sampled per second (maybe per millisecond?).  This 
> also allows us to work with real test environments easily.
>
> Here is my initial and non-exhaustive list of metrics (based off of 
> the draft):
>
> [Received] In Kbit/s
> [Sent] Out Kbit/s
> Packets sent/second,
> Packets received/second,
> Discards per second [ use a range of buffer calculations, 20ms 40ms, 
> 100ms, 200ms, 500ms]
>
>
> These metrics should be built into an RTP stack that we standardize 
> for testing.  My two preferences are jrtplib (C++) and libjitsi 
> (Java).  At least with both of these, we can contact the authors and 
> get any bugs fixed if we need them relatively easily.
>
> Thanks,
> Dan

---

From: Stefan Holmer <[email-masked]>
Date: Thu, 14 Mar 2013 10:32:19 +0100

Another option could be to develop something based on the RTP stack on
http://webrtc.googlecode.com. We have some legacy
code<https://code.google.com/p/webrtc/source/browse/#svn%2Ftrunk%2Fwebrtc%2Fmodules%2Frtp_rtcp%2Ftest%2FBWEStandAlone>
which
contains, among other things, various load generators, but it hasn't been
used for quite some time, so I think it would need a bit of work to get
back to a working state.

/Stefan

On Thu, Mar 14, 2013 at 10:04 AM, Sergio Garcia Murillo <
[email-masked]> wrote:

> Hi Dan,
>
> I made a testing tool to test my rate adaptation algorithm implementation
> in my MCU. You can find the source code here:
>
> http://sourceforge.net/p/**mcumediaserver/code/745/tree/**trunk/cca/<http://sourceforge.net/p/mcumediaserver/code/745/tree/trunk/cca/>
>
> emulate certain video behaviors, like different sizes for I/P frames) and a
> receiever. I implementes a full RTP/RTCP stack with preliminary support for
> NACK and FEC (which I think it is essential for evaluating the behavior of
> the algorithm).
>
> The current algorithm implemented is based on google's congestion control
> algorithm, but it could be also interesting to create a comon interface so
> we could test different algorithms and compare between them.
>
> Best regards
> Sergio
>
>> We will never agree on what a definition of fairness is, and I think it
>> is a lost cause.  However, we can definitely reach consensus on expected

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 14 Mar 2013 14:07:38 +0000

2. Packet Generator Model
After re-reading Varun's draft and speaking with him on the evaluation scenarios, I think it makes enough sense to build a packet generator to implement what he suggests.  The time involved isn't substantial, and it can help us get a unified starting point for testing even if we decide to change it later.  It's worth at least this much.

My proposed implementation of this will be able to simulate real video traffic as well as fixed packet generator traffic.  For the real video traffic, it can be fed a CSV file filled with frame sizes and RTP timestamps.
[Zaheduzzaman Sarker] we already use some sort of packet generator in our simulations, right? And it might be also not that hard to come up with a packet generator. However, the hard part is to realize how representative those packet generator are for our usecases at hand? IMHO, we will need to verify the generate packet trace a with some reference trace. Also remember we might have QoE as evaluation  metric too and I hear people are also talking about subjective quality related to fairness. It will be hard to achieve that with just simple packet generators.
 For the fixed generator traffic, ladder steps can be used.  I'm thinking I'll start a project on Google Code, and make any interested participant a developer

BR

Zahed

---

From: Varun Singh <[email-masked]>
Date: Thu, 14 Mar 2013 10:09:20 -0400

Hi Stefan, Sergio,

Is it possible for either of you to characterize the behaviour of the
packet generator?
I am mainly looking for plots related to the different video parameters.

Thanks,
Varun

On Thu, Mar 14, 2013 at 5:32 AM, Stefan Holmer <[email-masked]> wrote:
> http://webrtc.googlecode.com. We have some legacy code which contains, among
> other things, various load generators, but it hasn't been used for quite
> some time, so I think it would need a bit of work to get back to a working
> state.
>
> /Stefan
>
>
> On Thu, Mar 14, 2013 at 10:04 AM, Sergio Garcia Murillo
> <[email-masked]> wrote:
>>
>> Hi Dan,
>>
>>> implement what he suggests.  The time involved isn't substantial, and it can
>>> help us get a unified starting point for testing even if we decide to change
>>> it later.  It's worth at least this much.
>>>
>>> timestamps.  For the fixed generator traffic, ladder steps can be used.  I'm
>>> thinking I'll start a project on Google Code, and make any interested
>>> participant a developer.
>>>
>>>
>>> 3. Evaluation Metrics
>>>
>>> be sampled per second (maybe per millisecond?).  This also allows us to work
>>> with real test environments easily.
>>>

-- 
http://www.netlab.tkk.fi/~varun/

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 14 Mar 2013 14:13:50 +0000

> For the fixed generator traffic, ladder steps can be used.  I'm thinking I'll start a project on Google Code, and make any interested participant a developer

Suggestion: take a look at if an existing generator can be extended, rather than starting from scratch? E.g. flowgrind

Lars

---

From: "Bill Ver Steeg (versteb)" <[email-masked]>
Date: Thu, 14 Mar 2013 14:21:02 +0000

Iperf can also generate a wide variety of traffic. I have also recently modified it  to track one-way delay, and can help with changing the code to send data patterns and track delay. 

I will try to make tomorrow's 8am meeting, and we can compare notes on what tools are available. 

Bill VerSteeg

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 14 Mar 2013 16:12:02 +0100

Hi,

just to name one more option:

we have a Java-based, event-driven simulation framework which we recently 
extended to integrate a virtual machine (using QEMU). That means we can use 
traffic generated by a real operating system in simulations. Currently we 
only have a small relay program running in the virtual machine that can be 
controlled by the simulation program to send chunks of data. Thus a specific 
generator could be implemented in simulation program. The relay program sends 
only TCP traffic at the moment. Moreover, we can also run a real application 
in the virtual machine. At the moment we are still working on the framework. 
Before we will publish the code, we need to find some more bug fixes. But the 
simulations are working and we are using this for all our current TCP 
simulations.

Mirja

On Thursday 14 March 2013 15:07:38 Zaheduzzaman Sarker wrote:
> timestamps. [Zaheduzzaman Sarker] we already use some sort of packet
> generator in our simulations, right? And it might be also not that hard to
> come up with a packet generator. However, the hard part is to realize how
> representative those packet generator are for our usecases at hand? IMHO,
> we will need to verify the generate packet trace a with some reference
> trace. Also remember we might have QoE as evaluation  metric too and I hear
> people are also talking about subjective quality related to fairness. It
> will be hard to achieve that with just simple packet generators. For the
> fixed generator traffic, ladder steps can be used.  I'm thinking I'll start
> a project on Google Code, and make any interested participant a developer
>
>
>
> BR
>
> Zahed

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Thu, 14 Mar 2013 15:41:58 +0000

Hi, 

Great idea on the packet generator model. We will also be happy to contribute on this piece as well. 

In the past we have put together some some ns-2 code for via in two approaches: 

- trace-driven:  where the packet sizes and intervals are taken from h.264 encoding result of video sequences. one could also switch between different rate versions of the same video content at GOP boundaries. 

- statistical modeling: where the encoder rate control behavior is modeled statistically, with tunable parameters on burstiness, range of deviation from target rate, and lag in reaction (i.e., how long it takes the encoder to react to a change in target rate). 

Thanks,
Xiaoqing

---

From: Fu Jiantao <[email-masked]>
Date: Fri, 15 Mar 2013 17:59:24 +0800

Testcase can be the following combinations

    - typical network(bw, rtt, qdelay, loss, token_burst)

    - with/without cross traffic(CBR, TCP/UDP etc)

    - avail-bw change(capcity change or cross traffic change)

    - inherit loss network

    - route change

Jeromy