Subject: [rmcat] Comments on eval draft (draft-singh-rmcat-cc-eval-02)

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 7 Mar 2013 18:30:14 +0100

Hi,

Now that I have finished reading the evaluation draft, I think the 
recent version look much better than the previous one however I have 
several comments, reflections, questions related to different sections 
of the draft.

Warning !!!!! long email ahead. Sorry, but I could not help it.

Metrics:

·I understand that the RFC 5166 is referred for the description of the 
metrics mentioned in this draft. However, my take is that RFC 5166 was 
written to address a wider scope than what we are trying to do in RMCAT 
hence IMHO it is best to scope the definition and description down to 
the particular scope of RMCAT- interactive real-time media over RTP. 
This means this draft should actually include proper definition of the 
metrics, methodologies to realize them within this document (or in 
additional documents).

·Barely mentioning the metrics does not really help. We need to define 
proper methodologies to use them. Some of the metrics are easy to 
visualize but some of them are not (see more below). We need to 
describe/define a common understanding of all the metrics and how to use 
them in evaluation.

·Delay: this version excludes "Delay" as metric. Could you explain why?

·Minimum oscillation in encoding rate: This is related to QoE and 
minimizing oscillationis an wanted behavior. How do we measure 
oscillation? RFC 5166 gives some ref. but how well do they represent 
interactive real-time communication?  We have variable bitrate encoder 
for video whose output will oscillate depending on scenes to encode. 
What do we do there? We perhaps will need to measure the oscillation in 
the target bitrate set bythe Congestion Control (CC) algorithm but very 
often there will be mismatch between the target rate and actual encoding 
rate.

·Reactivity to transient events: We canreact to transient event only if 
we know that it is a transient event. In testbeds, perhaps we can do 
this by setting up particular test for a particular transient event and 
very soon we will find that there is no "silver bullet" to dodge all 
favorite transient events. What can we do then? one way to handle this 
could be to setup a sequence tests of transient events and weight them 
to evaluate the reactivity to the transient events. Please let me know 
if I am missing something here.

·Packet loss and discard rate: We have three categories here- 1) loss at 
transport layer (TL), 2) loss due to late arrival (LL) and 3) residual 
loss  (RL = TL+LL).  RL will also have effects of application depended 
techniques used for countering the packet loss (for example, FEC, RTP 
retransmission). Where TL perhaps a good metric for network, RL is 
perhaps what the application care about most. We will need to define a 
relation here. Also this metric is not about how the CC algorithms 
calculate the packet loss rate (--  different algorithm may calculate 
loss event/rate differently- per RTCP Send Report/Receiver Report, 
fractional loss over a time period, per RTT etc. etc.) but about how we 
want to measure the loss, right? then, lets define it concretely.

·Bandwidth utilization: This is a metric I have hard time to grasp. 
Whatdo we actuallywant to achieve/measure with this metric? I perhaps 
understand it if I have a fixed capacity link but in real-time 
communication the input data itself can starve*//*,resulting  inless 
utilization of the available bandwidth. And for cellular networks where 
the users have different capacity view even within a cell depending on 
their radio condition, speed, distance etc. etc. ; even in a aggregate 
level (per cell measurement) will depend on what is going on other 
cells. Moreover, the view of available bandwidth changes all the time, I 
don't see how we can visualize the available bandwidth utilization. I am 
not aware of any model that actually does it. Please let me know if I 
have missed something here. The thing is if we can't really measure a 
metric in a proper/straight forward way then there is not point of 
including it.

·Convergence time: I would like add this metric in the preferred metric 
list. This is also related to fairness metric as it gives a view about 
how fast the algorithm can reach a stable bitrate when competing with 
flows with same CC algorithm.

Guideline:

·4.3 : It seems that it is necessary to have some reference cases to 
verify against.

·4.4 : Is this the plan to test against all possible access types?

·4.7 : any idea on how to "highlight the bottleneck capacity share of 
each RTP flow" for wireless/cellular networks?

·This also says "However, long-lived TCP flows may starve out the RTP 
flow depending on router queue length. In the latter case the proposed 
congestion control for RTP should be as aggressive as standard TCP"- how 
realistic is this? at some point (if there are buffer-bloats) the RMCAT 
will have to give up as it has certain delay budget to meet.

·missing guidelines

·Congested feedback channel : the CC algorithm should be evaluated 
against congested feedback channel. Could be included as separate 
guideline or a part of 4.5.

·Different start up values: as the requirement draft mentioned several 
options for start-up values, the algorithms should be tested. One can 
argue that different CC algorithm will have different strategies but 
then it should be judged against fairness and convergence time.

·Fairness to other RMCAT proposed algorithms: as part of fairness guideline.

Eval scenarios:

·This section should really list a "must be" evaluated test scenarios 
not example scenarios. Those "must be" evaluated scenarios should take 
the guides lines into account and includetherelatedmetrics so that 
people have a common view on how to evaluate their algorithms. Any kind 
of effort to write down example scenarios should be waste of time and 
energy. Of course, there could be optional test cases and people can go 
an add their scenarios and if some scenarios become very useful then 
include those in the "must-be" evaluated sets.

·Video starting rate is 128 kbps. Feels like the default behavior is 
slow start, why?

·Yes, I would like to get a value(s) for queue length when AQM is used 
along with the AQM behavior.

·I have hard time fitting a radio scenario into dumbbell topology with 
the current definition in the draft.

BR

-- 

Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Multimedia Technologies
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com

==================================================