Subject: [rmcat] review of draft-ietf-rmcat-eval-criteria-02

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 13 Oct 2014 17:58:02 +0200

Hi Varun,

I reviewed the draft-ietf-rmcat-eval-criteria-02 document as an individual 
contributor (not w-g- chair). I have a couple of general comments and a set of 
more detailed comments below.

For me it is currently not very clear what the purpose of this document is (in 
relation to the reqs doc and the test scenarios doc). Form my point of view the 
requirement document is (mainly) written for people who would like to develop a 
new rmcat algorithm. I think this document should address people who want to 
evaluate an (existing) proposal by not only implementing the initial test 
scenarios but e.g. perform additional tests to check if the initial requirements 
can be fulfilled. Therefore this document should relate closely to the reqs 
document (while not stating new requirements!).

Therefore I would see the following topics as content of this document:

1. Summary of the requirements (of draft-ietf-rmcat-cc-requirements):
This should only be a very brief summary mainly pointing to the other document. 
If you look at draft-ietf-rmcat-cc-requirements there might be even some 
requirements that do not need (simulative/testbed) evaluation e.g. the amount of 
overhead. Further you could add to the list some of the 'classical requirements' 
as convergence/stability (and provide a respective reference) but I believe 
these requirements are all somehow covered by draft-ietf-rmcat-cc-requirements...?

2. List of metric and parameters:
Currently section 3 has 2 lists which overlap to some extend. Further there is 
another list in draft-ietf-rmcat-eval-test. I believe there should be ONE list 
in the eval document and the test document should relate to this. However, the 
description of which data should be traced i would rather see as a part of 
draft-ietf-rmcat-eval-test. More detailed comments further below.
Further I have the feeling that the metrics described in RFC5166 are not 
comprehensive, e.g. minimizing oscillations is rather a requirement than a 
metric from my perspective. Therefore we should define our own list here (of 
course this referring RFC5166) and also focus on metrics that are related to 
media congestion control (where we e.g. might have application-limited traffic).

Further, I would list parameters that can be varied in the evaluation and 
potentially even discuss a sensible range for evaluation, e.g. there is a 
maximum (base) RTT as we talk about the Internet and usually avoid to route the 
traffic several times around the world.

3. Guideline for evaluation (scenarios):
Here is would like to see all three things from above to be set into relation: 
To evaluate a certain requirement (or set of related requirements), you can vary 
these parameter in this range and then look at this metric and you would expect 
a results/value of the metric in this range.

I know, this more or less proposes a completely different document but this is 
just a proposal. Again for me the purpose of the currently document is not very 
clear anymore. Otherwise you would need to find a way to further explain in the 
existing document what the purpose and goals of this document are. If you plan a 
wider revision and would like to get some more input, I'm also willing to 
contribute text. Just let me know. Or we can sit together in Honolulu if you 
think that's helpful.

Here some more detailed comments:

Section 3: Metrics
------------------
- 'minimizing oscillations' is a requirement for me; e.g. oscillation size would 
be a metric defined as e.g. the difference between the minimum sending 
rate/window and the next maximum or the difference between the min and max 
within a given time frame..

- 'Reactivity': I'm used to the term responsiveness instead; again this is a 
requirement for me while convergence time (as given in RFC5166) is the 
respective metric for evaluation

- 'packet loss and discards': What's the difference between a loss and a discard 
and why is this listed separately? Is this in number of packets or bytes? Or do 
you mean the loss rate (=total number of pkt/bytes divided by the lost 
pkts/bytes over a certain time interval)?

- 'Section 2.1 of [RFC5166] discusses the tradeoff between
       throughput, delay and loss.' is not a metric

- 'Thoughput',  'Sending rate, Receiver rate, Goodput' and 'Bandwidth 
Utilization' are all very closely connected might not need to be treated 
separately. I think the metric that is of interest is utilization while you 
usually need to measure the sending rate to actually get the utilization. As I 
stated above I would see this part, what exactly to measure (and also where!), 
as part of draft-ietf-rmcat-eval-test. It could also be a part of this document 
but might make the test document more complicated. In any case which data to 
trace in the measurement and metrics for statistical evaluation should be 
discussed separately. And one more comment regarding the sending/receiving rate: 
these are usually bytes over a certain time frame and the length of the time 
frame should be defined (together with the evaluation metric that is using this 
data); I usually use a time interval of two RTTs.

- The discussion on fairness/unfairness does not belong in this section but in 
the next. In this section there are two metrics which might be of interest: 
jains fairness index and/or the ratio of the average throughput of one flow to 
another (or the ratio of groups of flows) over a certain time interval

- In point 7 'stable rate' needs to be defined. I usually use e.g. time to reach 
80%/95% of the equal share (or a desired/given rate in media transmissions...?) 
There is something similar done in draft-irtf-iccrg-tcpeval section 5.3.3.

- Regarding your question on Jains fairness index: I think this is a valuable 
metric (as mentioned above) only the term fairness is wrong. This metric give an 
impression if the capacity is shared equally between competing flows or groups 
of flows. There are evaluation scenarios where this is desired, e.g. when two 
flow compete using the same congestion control algorithm and the same traffic 
model (in our case, might even send the same video)

- Section 3.1. would go into the test document from my current point of view.

- And regarding tracing (which might go into the test doc): wouldn't it be 
useful for some evaluations to trace the congestion window (if possible) because 
this might be larger than the sending rate...?

Section 4: Guidelines
---------------------
This section currently describes some requirements (Avoiding Congestion Collapse 
and Stability) as well as part of the traffic model and network model.
I would propose to have the parameter discussion as mentioned above. Further the 
parameter discussion should be separated into traffic and network parameter. 
This would integrate some text from section 4.3., 4.8. and 4.9. (traffic model) 
as well as 4.4., 4.5. 4.6. and 4.7. (network model). And then use this section 
to give guidance how to evaluate the requirements in 
draft-ietf-rmcat-cc-requirements.

Further this document should rely on e.g. draft-irtf-iccrg-tcpeval-01 and (only) 
discuss additional things needed to evaluate media congestion control (as the 
requirements are also slightly different). So one big difference is the traffic 
model. So there should be cases in the evaluation where the traffic is 
application limited. If there is e.g. in a certain scenario enough bandwidth 
capacity available to change the video coding to a higher rate, how long does it 
take until this is indicated to the application (assuming there is some kind of 
interface)? Or how often is the sending rate lower than the application rate 
(which could cause starvation)? This of course needs a traffic model as well as 
an interface which are currently both in separate documents (and work in progress).

What I really miss in this section is any relation to the metrics. I would 
expect something like: convergence is reached if the convergence time is never 
longer than X sec (as a very general example) or (more media specific) quality 
is sufficient if the sending rate is never X% lower than the application rate 
and never for more than % seconds.... and so on.

- section 4.1. and 4.2 it this actually the same kind of evaluation? And this 
states only a requirement up-to now. How would this be evaluated?

- section 4.4. 'start-rates': what do you mean? The start rate of the 
application traffic or the link bandwidth?

- section 4.5.: I believe most lower layer protocol nowadays have a very good 
recovery implemented such that there are usually e.g. in wifi not much 
errors/loss. therefore form the perceptive of the transport only the delay is 
varying.
'Furhermore packet loss may induce additional delay': this reads a little weird: 
make more clear that there is packet loss at the lower layer but only delay 
visible to the higher layer...

- Section 4.6. 'The main motivation for the previous and current criteria is to
    identify situations in which the proposed congestion control is less
    performant.' Not sure why this is written here... isn't this always the 
intention?

- section 4.4. also describes some king of evaluation on an transient effect 
which should be part of 4.7. then...?

- Section 4.9. discussion on short flow is again more a requirement but short 
flow traffic should be configurable in the traffic model

- section 4.10. this again is a requirement. I don't think this can be evaluated 
in simulation/testbeds; so I believe that does not belong in this document.

Section 5 and 6
---------------
Not sure if this belong into this document. At least section 6 is partly 
documented in our charter already. If you see this document as a frame document 
that will introduce the other documents (e.g. reqs, tests, interface, traffic 
model) and explain the relation between those documents than this text can stay 
but should move into the introduction.

Appendix A
-----------
- I don't understand why this tradeoff is discussed here.

- Appendix A.1: I really, really think that some kind of 'quality metrics' must 
be discussed in this document because this is the different to other congestion 
control evaluation. I don't think we should discuss any kind of quality of 
experience though. This metric should rather be e.g. % of times where the 
sending rate was lower than the application rate; % of times where the link was 
underutilizated and a higher coding could have been used. Again this also needs 
some kind of video traffic model; but I believe a simple interface which 
indicated 'go one step up/down' and the respective model that switches to 
certain points of time one step up or down would be sufficient for initial 
evaluation; though I'm not a video expert...

All in all I would recommend to relate this document in any case stronger to 
draft-ietf-rmcat-cc-requirements and mainly discuss things that are specific to 
MEDIA congestion control while only referring or summarizing other aspects.

I hope this is helpful. Let me know if you have questions or would like further 
input.

Mirja

---

From: Varun Singh <[email-masked]>
Date: Thu, 23 Oct 2014 14:55:00 +0300

Hi Mirja,

Thank you for the feedback and apologies for the tardiness, I needed
to resync with the other drafts. Comments inline.

On Mon, Oct 13, 2014 at 6:58 PM, Mirja Kühlewind
<[email-masked]> wrote:
> contributor (not w-g- chair). I have a couple of general comments and a set
> of more detailed comments below.
>
> For me it is currently not very clear what the purpose of this document is
> (in relation to the reqs doc and the test scenarios doc). Form my point of
> view the requirement document is (mainly) written for people who would like
> to develop a new rmcat algorithm. I think this document should address
> people who want to evaluate an (existing) proposal by not only implementing

The idea of the criteria document was to create guidelines that the
test documents could either follow or fulfill.

> the initial test scenarios but e.g. perform additional tests to check if the
> initial requirements can be fulfilled. Therefore this document should relate
> closely to the reqs document (while not stating new requirements!).
>

True, this document should not generate new congestion control
requirements. The motivation behind the guidelines section was to have
a set of requirements for the test cases. i.e., the congestion control
behaviour is described by the requirements document, while the
test-cases documents attempt to fulfil the guidelines in the criteria
document.

However, the text currently in the document may appear as the new
congestion control requirements.
More discussion below.

> This should only be a very brief summary mainly pointing to the other
> document. If you look at draft-ietf-rmcat-cc-requirements there might be
> even some requirements that do not need (simulative/testbed) evaluation e.g.
> the amount of overhead. Further you could add to the list some of the
> 'classical requirements' as convergence/stability (and provide a respective
> reference) but I believe these requirements are all somehow covered by
> draft-ietf-rmcat-cc-requirements...?
>
> Currently section 3 has 2 lists which overlap to some extend. Further there
> is another list in draft-ietf-rmcat-eval-test. I believe there should be ONE
> list in the eval document and the test document should relate to this.
> However, the description of which data should be traced i would rather see
> as a part of draft-ietf-rmcat-eval-test. More detailed comments further
> below.

Yes, RFC5166 was a starting point. I was hoping that the metrics
section would firm up when we have more proposals/evaluations. Perhaps
after Honolulu, we'll have a better list of metrics.

> maximum (base) RTT as we talk about the Internet and usually avoid to route
> the traffic several times around the world.
>

Sure, I agree with the general guidance and am happy to add values
that make sense. Earlier version carried RTT and loss tables, which
have since moved to the appendix of eval-test -- in an attempt to make
this document more generic.

> Here is would like to see all three things from above to be set into
> relation: To evaluate a certain requirement (or set of related
> requirements), you can vary these parameter in this range and then look at
> this metric and you would expect a results/value of the metric in this
> range.
>
> I know, this more or less proposes a completely different document but this
> is just a proposal. Again for me the purpose of the currently document is
> not very clear anymore. Otherwise you would need to find a way to further

I think there are at least multiple ways to do this:
1. To test the follow requirement vary these parameters, or doing this
test fulfils a set of requirements.
2. These kind of network/traffic behaviour exists, check how the
proposal performs in these scenarios.

I suppose we want to find a way to express both in this document.

> explain in the existing document what the purpose and goals of this document
> are. If you plan a wider revision and would like to get some more input, I'm
> also willing to contribute text. Just let me know. Or we can sit together in
> Honolulu if you think that's helpful.
>
>
> - 'minimizing oscillations' is a requirement for me; e.g. oscillation size
> would be a metric defined as e.g. the difference between the minimum sending

There needs to be component of frequency of oscillations too, i.e.,
number of times
it oscillates between a max and min.

Because it could just be ramping up or ramping down. The timescales to
measure this
are important, partly because the media is data limited and it may
generate video which
is bursty.

> respective metric for evaluation
I agree.

> - 'packet loss and discards': What's the difference between a loss and a
> discard and why is this listed separately? Is this in number of packets or

Discards, unlike lost packets are received by the RTP endpoint but
discarded by the dejitter buffer because it was arrived too late to be
played back.

> bytes? Or do you mean the loss rate (=total number of pkt/bytes divided by

Here it is packets. But we can have Packets sent, packets lost,
packets discarded.
Similarly Bytes sent, bytes lost, bytes discarded.

> the lost pkts/bytes over a certain time interval)?
>

There was some discussion in the eval-test to do 200ms or 500ms.

>       throughput, delay and loss.' is not a metric

I suppose this is an open issue of sort, we need a metric for
comparing performance of a proposal when it can be tuned to give
higher throughput at expense of some loss and delay.

Or, when comparing across proposals: when there are no losses or
comparative delay, we would prefer the one with higher throughput.

For the oscillations, responsiveness, and the trade-off metric: I am
hoping that we would be able to get better insight into measuring
these metrics more accurately when we have more comparative
evaluations.

> Utilization' are all very closely connected might not need to be treated

Goodput is the bytes in an interval fed into the decoder. This removes
the packet overheads and any packets that were discarded by the
dejitter buffer.

> usually need to measure the sending rate to actually get the utilization. As

True we need the sending rate, easily possible in controlled
simulation environments and testbeds.

> I stated above I would see this part, what exactly to measure (and also
> where!), as part of draft-ietf-rmcat-eval-test. It could also be a part of
> this document but might make the test document more complicated. In any case
> which data to trace in the measurement and metrics for statistical
> evaluation should be discussed separately. And one more comment regarding
> the sending/receiving rate: these are usually bytes over a certain time
> frame and the length of the time frame should be defined (together with the
> evaluation metric that is using this data); I usually use a time interval of
> two RTTs.
>

I think during the tests we were thinking about 200 or 500ms. If the
RTTs are low and too few video frames are generated, we might get 0
frames in that interval. I dont have a strong opinion about this, but
may be Zahed can weigh in, as his proposal sometimes drops a few video
frames to test (unloaded) delay?

> - The discussion on fairness/unfairness does not belong in this section but
> in the next. In this section there are two metrics which might be of
> interest: jains fairness index and/or the ratio of the average throughput of
> one flow to another (or the ratio of groups of flows) over a certain time
> interval
>
> - In point 7 'stable rate' needs to be defined. I usually use e.g. time to
> reach 80%/95% of the equal share (or a desired/given rate in media
> transmissions...?) There is something similar done in
> draft-irtf-iccrg-tcpeval section 5.3.3.
>

It makes senses to check stability in a few simple scenarios:
when there no cross traffic, fixed network characteristics (i think we
have one of these in the test-cases), but is unlikely going to apply
to an LTE scenario.

Will look in to this, thanks for the pointer.

> - Regarding your question on Jains fairness index: I think this is a
> valuable metric (as mentioned above) only the term fairness is wrong. This
> metric give an impression if the capacity is shared equally between
> competing flows or groups of flows. There are evaluation scenarios where
> this is desired, e.g. when two flow compete using the same congestion
> control algorithm and the same traffic model (in our case, might even send
> the same video)
>

Will fold it in.

> useful for some evaluations to trace the congestion window (if possible)
> because this might be larger than the sending rate...?
>

This is something that the proposals should point out. In my opinion,
I have not made any significant changes to the metrics section since
the first version, mainly because we haven't discussed the proposal
evaluations in much detail. I think this will be fixed after the
upcoming IETF meeting and we'll see more metrics.

>
> Section 4: Guidelines
> ---------------------
> This section currently describes some requirements (Avoiding Congestion
> Collapse and Stability) as well as part of the traffic model and network
> model.
> I would propose to have the parameter discussion as mentioned above. Further
> the parameter discussion should be separated into traffic and network
> parameter. This would integrate some text from section 4.3., 4.8. and 4.9.
> (traffic model) as well as 4.4., 4.5. 4.6. and 4.7. (network model). And
> then use this section to give guidance how to evaluate the requirements in
> Further this document should rely on e.g. draft-irtf-iccrg-tcpeval-01 and
> (only) discuss additional things needed to evaluate media congestion control
> (as the requirements are also slightly different). So one big difference is
> the traffic model. So there should be cases in the evaluation where the
> traffic is application limited. If there is e.g. in a certain scenario
> enough bandwidth capacity available to change the video coding to a higher
> rate, how long does it take until this is indicated to the application
> (assuming there is some kind of interface)? Or how often is the sending rate
> lower than the application rate (which could cause starvation)? This of
> course needs a traffic model as well as an interface which are currently
> both in separate documents (and work in progress).
>
> expect something like: convergence is reached if the convergence time is
> never longer than X sec (as a very general example) or (more media specific)
> quality is sufficient if the sending rate is never X% lower than the
> application rate and never for more than % seconds.... and so on.
>
> - section 4.1. and 4.2 it this actually the same kind of evaluation? And
> this states only a requirement up-to now. How would this be evaluated?
>

I think 4.1 could be removed or be reworded into something like:
Simulate conditions that will not immediately trigger the circuit breaker.

> application traffic or the link bandwidth?

application traffic, i.e., the starting video media bit rate.

> - section 4.5.: I believe most lower layer protocol nowadays have a very
> good recovery implemented such that there are usually e.g. in wifi not much
> errors/loss. therefore form the perceptive of the transport only the delay
> is varying.

True, but it is also possible to turn off link layer retransmissions,
I think the cellular test cases want them to be turned off.

> 'Furhermore packet loss may induce additional delay': this reads a little
> weird: make more clear that there is packet loss at the lower layer but only
> delay visible to the higher layer...
>

Will fix language.

>    performant.' Not sure why this is written here... isn't this always the

Yes. Depending on the variation of the link capacity, this test would
really be brutal for some congestion control algorithms.

> which should be part of 4.7. then...?

Yes. 4.4 is transience at the start of the session, and 4.7 is
transience mid session. This can be merged.

> - Section 4.9. discussion on short flow is again more a requirement but
> short flow traffic should be configurable in the traffic model
>

This can be removed entirely or folded into a single guideline about
measuring performance with cross-traffic.

> - section 4.10. this again is a requirement. I don't think this can be
> evaluated in simulation/testbeds; so I believe that does not belong in this
> document.
>

I think it should be covered under metrics at the very least, because
it contributes to overhead. Additionally, the performance would vary
for example on the frequency of feedback packets and what

> documented in our charter already. If you see this document as a frame
> document that will introduce the other documents (e.g. reqs, tests,
> interface, traffic model) and explain the relation between those documents
> than this text can stay but should move into the introduction.
>

Sections 5 and 6 can be removed.

> Appendix A

I think this is the remnants of the discussion for the trade-off
between Throughput, loss, and delay.

> - Appendix A.1: I really, really think that some kind of 'quality metrics'
> must be discussed in this document because this is the different to other
> congestion control evaluation. I don't think we should discuss any kind of
> quality of experience though. This metric should rather be e.g. % of times
> where the sending rate was lower than the application rate; % of times where

There are many strategies here, in the end a lot depends on what was rendered.
A sender can burst a large frame, pace the same frame over a short interval
or over a large interval, etc. All these have various short-term
impacts, I don't know
if we are there yet in terms of quantifying it.

Or if we should attempt to model this based on raw transport metrics.
(delays, throughput, etc.)

> the link was underutilizated and a higher coding could have been used. Again
> this also needs some kind of video traffic model; but I believe a simple
> interface which indicated 'go one step up/down' and the respective model
> that switches to certain points of time one step up or down would be
> sufficient for initial evaluation; though I'm not a video expert...
>

It is not in the main section because it was unclear what quality
meant. The agreement was we want the highest throughput possible with
the least amount of delay :) but anything less was unclear. Since the
proposals are now being evaluated, I am hoping that we'll come
together on some metric in the near future.

> draft-ietf-rmcat-cc-requirements and mainly discuss things that are specific
> to MEDIA congestion control while only referring or summarizing other
> aspects.
>
> I hope this is helpful. Let me know if you have questions or would like
> further input.
>

I will hold my updates until after Honolulu, based on the evaluation
results, I'll have more insight into the metrics.

-Varun

-- 
http://www.netlab.tkk.fi/~varun/