Subject: [rmcat] RMCAT Testing Topology and Initial Testing

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Fri, 5 Apr 2013 16:54:19 +0000

RMCAT Design Team,

At the RMCAT Design Team meeting on 28 March, I volunteered to document an initial network topology model and an initial testing methodology model for our RMCAT work. The goal is to have a topology and an associated testing methodology so that we  could determine if the RMCAT requirements are being met.

Attached is my first draft for such a proposal. It is a work in progress.

Comments are welcome, but please be sure to raise a technology-based objection and associated rationale with any item within it you find objectionable.

We have our work cut out for us, as designing an acceptable control system (i.e. rate adaptation algorithm) to meet all of our requirements will be tough. As can be seen, my initial testing proposal only addresses the simplest part of our work: RMCAT self-friendliness.

I would like to poll this audience, when should the next RMCAT design meeting be held?

I would be willing to review various aspects of my proposal at the next meeting.

Comments? Fire away!

Best Regards,

Michael Ramalho

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 11 Apr 2013 17:09:50 +0200

Hi Michael,

Thanks for putting up such a well thought document. I have some comments 
and questions, please find them in the attached document.

BR

Zahed

On 2013-04-05 18:54, Michael Ramalho (mramalho) wrote:
>
> RMCAT Design Team,
>
> At the RMCAT Design Team meeting on 28 March, I volunteered to 
> document an initial network topology model and an initial testing 
> methodology model for our RMCAT work. The goal is to have a topology 
> and an associated testing methodology so that we  could determine if 
> the RMCAT requirements are being met.
>
> Comments are welcome, but please be sure to raise a technology-based 
> objection and associated rationale with any item within it you find 
> objectionable.
>
> We have our work cut out for us, as designing an acceptable control 
> system (i.e. rate adaptation algorithm) to meet all of our 
> requirements will be tough. As can be seen, my initial testing 
> proposal only addresses the simplest part of our work: RMCAT 
> self-friendliness.
>

-- 

Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Multimedia Technologies
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com

==================================================

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Fri, 12 Apr 2013 21:05:32 +0000

Zahed: Thank you for your through review of the initial RMCAT network topology model and testing methodology document.

***

RMCAT Design Team,

Attached are two files.

The first file is my response and commentary to Zahed's review of the original initial RMCAT network topology model and testing methodology document (*_MAR_reply*).

The second file is an update to the original document based on Zahed's input and other inputs (*_April_12*).

Mo Zanaty and I propose that the RMCAT design team meet next Wednesday, April 17 at 16:00 GMT. Look for an invite to this meeting after this email from Mo.

At that meeting I will review the document and Mo and I will seek direction on what tasks to perform next.

Best Regards,

Michael Ramalho

From: Zaheduzzaman Sarker [mailto:[email-masked]]
Sent: Thursday, April 11, 2013 11:10 AM
To: Michael Ramalho (mramalho)
Cc: [email-masked]
Subject: Re: [rmcat] RMCAT Testing Topology and Initial Testing

Hi Michael,

Thanks for putting up such a well thought document. I have some comments and questions, please find them in the attached document.

BR

Zahed

Comments? Fire away!

Best Regards,

Michael Ramalho

--

Zahed

ANM ZAHEDUZZAMAN SARKER

Ericsson AB

Multimedia Technologies

Laboratoriegränd 11

97128 Luleå, Sweden

Phone +46 10 717 37 43

Fax +46 920 996 21

SMS/MMS +46 76 115 37 43

[email-masked]<mailto:[email-masked]>

www.ericsson.com<http://www.ericsson.com>

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 13 Apr 2013 13:51:53 +0000

Hi,

thanks guys, for working on this!

I wanted to point out that in the past in other WGs, participants have complained if content was shared in proprietary formats (e.g., MS Office). So maybe using a wiki would be a more appropriate place for this work. Or, this WG can decide that we are OK with this and continue to use what's easiest to get the work done. (I have no preference.)

Lars

---

From: Varun Singh <[email-masked]>
Date: Wed, 17 Apr 2013 16:33:30 +0300

Hi Michael,

I reviewed the latest document, and also read the discussion in ZS_MAR but
my comments may still overlap that discussion.

So is the meeting today?

On Sat, Apr 13, 2013 at 12:05 AM, Michael Ramalho (mramalho) <
[email-masked]> wrote:

>  Zahed: Thank you for your through review of the initial RMCAT network
> topology model and testing methodology document.****
>
> ** **
>
> *******
>
> ** **
>
> RMCAT Design Team,****
>
> ** **
>
> Attached are two files.****
>
> ** **
>
> The first file is my response and commentary to Zahed’s review of the
> original initial RMCAT network topology model and testing methodology
> document (*_MAR_reply*).****
>
> ** **
>
> The second file is an update to the original document based on Zahed’s
> input and other inputs (*_April_12*).****
>
> ** **
>
> Mo Zanaty and I propose that the RMCAT design team meet next Wednesday,
> April 17 at 16:00 GMT. Look for an invite to this meeting after this email
> from Mo.****
>
> ** **
>
> At that meeting I will review the document and Mo and I will seek
> direction on what tasks to perform next.****
>
> ** **
>
> Best Regards,****
>
> ** **
>
> Michael Ramalho****
>
> ** **
>
> *From:* Zaheduzzaman Sarker [mailto:[email-masked]]
> *Sent:* Thursday, April 11, 2013 11:10 AM
> *To:* Michael Ramalho (mramalho)
> *Cc:* [email-masked]
> *Subject:* Re: [rmcat] RMCAT Testing Topology and Initial Testing****
>
> ** **
>
> Hi Michael,
>
> ****
> On 2013-04-05 18:54, Michael Ramalho (mramalho) wrote:****
>
> RMCAT Design Team,****
>
>  ****
>
> At the RMCAT Design Team meeting on 28 March, I volunteered to document an
> initial network topology model and an initial testing methodology model for
> our RMCAT work. The goal is to have a topology and an associated testing
> methodology so that we  could determine if the RMCAT requirements are being
> met.****
>
>  ****
>
> Attached is my first draft for such a proposal. It is a work in progress.*
> ***
>
>  ****
>
> objectionable.****
>  ****
> We have our work cut out for us, as designing an acceptable control system
> (i.e. rate adaptation algorithm) to meet all of our requirements will be
> tough. As can be seen, my initial testing proposal only addresses the
> simplest part of our work: RMCAT self-friendliness.****
>
>  ****
>
> I would like to poll this audience, when should the next RMCAT design
> meeting be held?****
>
>  ****
>
> I would be willing to review various aspects of my proposal at the next
> meeting.****
>
>  ****
>
> Comments? Fire away!****
>
>  ****
>
> Best Regards,****
>
>  ****
>
> Michael Ramalho****
>
>
>
> ****
>
> -- ****
>
> ** **
>
> Zahed****
>
> ** **
>
> ==================================================****
>
> ANM ZAHEDUZZAMAN SARKER ****
>
> ** **
>
> ** **
>
> Ericsson AB****
>
> Multimedia Technologies****
>
> Laboratoriegränd 11****
>
> 97128 Luleå, Sweden****
>
> Phone +46 10 717 37 43****
>
> Fax +46 920 996 21****
>
> SMS/MMS +46 76 115 37 43****
>
> [email-masked]****
>
> www.ericsson.com****
>
> ** **
>
> ==================================================****
>
>

-- 
http://www.netlab.tkk.fi/~varun/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 17 Apr 2013 14:33:39 +0000

I made the webex recur daily (even weekends) at 16:00 London (summer time), still using the same meeting number, password, link, etc. as before (repeated below). I didn't send out any calendar invite since no one replied about having a meeting today. I suggest we meet next week on Wed Apr 24, if enough people reply. Or maybe another doodle with multiple days/times again.

Topic: RMCAT Eval Design Team
Date: Every 1 day, from Wednesday, April 17, 2013 to no end date
Time: 4:00 pm, GMT Summer Time (London, GMT+01:00)
Meeting Number: 203 451 310
Meeting Password: rmcat
Host Key: 412564
Link: https://cisco.webex.com/ciscosales/j.php?ED=221585972&UID=484666302&PW=NNmNhZTJjYTI1&RT=MiMyMQ%3D%3D
Global Access Numbers: http://cisco.com/en/US/about/doing_business/conferencing/index.html

Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Varun Singh
Sent: Wednesday, April 17, 2013 9:34 AM
To: Michael Ramalho (mramalho)
Cc: [email-masked]; Zaheduzzaman Sarker
Subject: Re: [rmcat] RMCAT Testing Topology and Initial Testing

Hi Michael,

I reviewed the latest document, and also read the discussion in ZS_MAR but my comments may still overlap that discussion.

So is the meeting today?

On Sat, Apr 13, 2013 at 12:05 AM, Michael Ramalho (mramalho) <[email-masked]<mailto:[email-masked]>> wrote:
Zahed: Thank you for your through review of the initial RMCAT network topology model and testing methodology document.

***

RMCAT Design Team,

Attached are two files.

Best Regards,

Michael Ramalho

From: Zaheduzzaman Sarker [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, April 11, 2013 11:10 AM
To: Michael Ramalho (mramalho)
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [rmcat] RMCAT Testing Topology and Initial Testing

Hi Michael,

BR

Zahed

Comments? Fire away!

Best Regards,

Michael Ramalho

--

Zahed

ANM ZAHEDUZZAMAN SARKER

Ericsson AB

Multimedia Technologies

Laboratoriegränd 11

97128 Luleå, Sweden

Phone +46 10 717 37 43<tel:%2B46%2010%20717%2037%2043>

Fax +46 920 996 21<tel:%2B46%20920%20996%2021>

SMS/MMS +46 76 115 37 43<tel:%2B46%2076%20115%2037%2043>

---

From: "Bill Ver Steeg (versteb)" <[email-masked]>
Date: Wed, 17 Apr 2013 14:56:45 +0000

I am good for next week

Bvs

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mo Zanaty (mzanaty)
Sent: Wednesday, April 17, 2013 10:34 AM
To: Varun Singh; Michael Ramalho (mramalho)
Cc: [email-masked]; Zaheduzzaman Sarker
Subject: Re: [rmcat] RMCAT Testing Topology and Initial Testing

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Varun Singh
Sent: Wednesday, April 17, 2013 9:34 AM
To: Michael Ramalho (mramalho)
Cc: [email-masked]<mailto:[email-masked]>; Zaheduzzaman Sarker
Subject: Re: [rmcat] RMCAT Testing Topology and Initial Testing

Hi Michael,

So is the meeting today?

***

RMCAT Design Team,

Attached are two files.

Best Regards,

Michael Ramalho

Hi Michael,

BR

Zahed

Comments? Fire away!

Best Regards,

Michael Ramalho

--

Zahed

ANM ZAHEDUZZAMAN SARKER

Ericsson AB

Multimedia Technologies

Laboratoriegränd 11

97128 Luleå, Sweden

---

From: Kevin Gross <[email-masked]>
Date: Sun, 28 Apr 2013 09:48:17 -0600

This looks like a good start.

I have a few comments on this proposal.

   1. You specify sources with infinite amount of data running "flat out".
   It sounds like such a source, in isolation, would consume all offered
   bandwidth. This is not how real media sources behave. Many have
   fixed bandwidth. All have some maximum bandwidth.
   2. You specify a transmission interval based on packet size. Again, this
   is not how real media sources behave. Most have a fixed transmission
   interval (packet time). If MTU is exceeded, back-to-back MTU packets are
   emitted each interval.
   3. It is fairly clear from the text what you're trying to do with
   measurement points A and B but the diagram is unclear. The router after the
   bottleneck would have to have different behavior on each of its egress
   ports to create the behavior you're describing. You can clarify by
   inserting a "WAN emulator" block between the output of the post-bottleneck
   router and measurement point B.

Kevin Gross
+1-303-447-0517
Media Network Consultant
AVA Networks - www.AVAnw.com <http://www.avanw.com/>, www.X192.org

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Mon, 29 Apr 2013 13:37:53 +0000

Kevin,

Thanks for your email. I hope to address your points in the design team meeting today.

In advance of that, let me try to answer here (with "MAR:" below) - and then we can further discuss during the conference.

Thanks in advance,

Michael Ramalho

From: [email-masked] [mailto:[email-masked]] On Behalf Of Kevin Gross
Sent: Sunday, April 28, 2013 11:48 AM
To: Michael Ramalho (mramalho)
Cc: Cullen Jennings (fluffy); Toerless Eckert (eckert); Mo Zanaty (mzanaty); [email-masked]; Paul Jones (paulej); James Polk (jmpolk)
Subject: Re: [rmcat] RMCAT Testing Topology and Initial Testing

  1.  You specify sources with infinite amount of data running "flat out". It sounds like such a source, in isolation, would consume all offered bandwidth. This is not how real media sources behave. Many have fixed bandwidth. All have some maximum bandwidth.
MAR: You understand what I intended. That is, the sources run similarly to so-called "longed lived TCP sources" (i.e., TCP that has an infinite amount of data to send).
MAR: So exactly what types of sources do we admit? Surely real-time audio and video ... but might we want holographic, multi-dimensional applications in twenty years (with much, higher BW requirements)?
MAR: And shouldn't the transport layer - as much as feasible - decouple itself from the actual application requirements?
MAR: We are in the middle ground here (almost a layer violation) - we desire to build a transport with congestion control USEABLE for real-time traffic.
MAR: So while we should keep real-time requirements front of mind, we really need to build a "real-time suitable" transport with congestion control that can span the whole range of possible bandwidths (including the holodeck).
MAR: That being said, I do think we must include some real-time constraints (e.g., my proposal of packets no FEWER than one per 100 ms) - but let's not constrain the bandwidths of the application.
MAR: The idea captured in the document was to help ensure that the transport is stable - thus constraining the actual generation of data by the sources. Then, once we are convinced that the transport protocol has all the desirable attributes (self-friendly, stable, etc.) then we can see what happens in the "real-world" when the burstiness of real audio/video applications are run on top of the transport.
MAR: There is nothing that stops us from building some limit to the behavior of the sources. Indeed, the adaptive sources I am building need to have the min/max rates provided to them. If their algorithms say they can run at higher than the max rate - the sources limit themselves to the max rate. At that point, sources in that state (i.e., est. avail. BW > than their max) appear like CBR sources to the remaining sources in the bottleneck.

  1.  You specify a transmission interval based on packet size. Again, this is not how real media sources behave. Most have a fixed transmission interval (packet time). If MTU is exceeded, back-to-back MTU packets are emitted each interval.
MAR: There is nothing stopping us from such a definition in other test scenarios. The reason why I specified it that way is that that it represents a BEST CASE (smooth) injection of data into the bottleneck. We know the situation only gets worse from there. The impact of the burstiness of a given source is highly dependent on bottleneck service rate and the instantaneous work provided to the bottleneck by of the bursty source. Proposing a "smooth source" takes out yet another degree of uncertainty. I agree it is not reality.

  1.  It is fairly clear from the text what you're trying to do with measurement points A and B but the diagram is unclear. The router after the bottleneck would have to have different behavior on each of its egress ports to create the behavior you're describing. You can clarify by inserting a "WAN emulator" block between the output of the post-bottleneck router and measurement point B.
MAR: Agree. The diagram should be changed. In my lab I presently only inject pure delay or loss at where the routers are shown in the figure with routers (simple command line for our routers) ... but in general you should think of the routers on leg A and B as impairment boxes (loss, jitter, delay in either direction).
MAR: Thanks Kevin.
Kevin Gross
+1-303-447-0517
Media Network Consultant
AVA Networks - www.AVAnw.com<http://www.avanw.com/>, www.X192.org<http://www.X192.org>

On Fri, Apr 5, 2013 at 10:54 AM, Michael Ramalho (mramalho) <[email-masked]<mailto:[email-masked]>> wrote:
RMCAT Design Team,

Comments? Fire away!

Best Regards,

Michael Ramalho

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Mon, 29 Apr 2013 16:53:40 +0200

On 2013-04-29 15:37, Michael Ramalho (mramalho) wrote:

>
> MAR: There is nothing that stops us from building some limit to the 
> behavior of the sources. Indeed, the adaptive sources I am building 
> need to have the min/max rates provided to them. If their algorithms 
> say they can run at higher than the max rate -- the sources limit 
> themselves to the max rate. At that point, sources in that state 
> (i.e., est. avail. BW > than their max) appear like CBR sources to the 
> remaining sources in the bottleneck.
>
you mean "appear like CBR sources" on average, right? If we consider 
video with a VBR encoder then as far as I know it will fulfill the 
target rate on a average so it can overshoot or undershoot the target 
rate at any given instance of time. This does not matters for a "flat 
out" data running kind of view but it might matter for VBR sources if 
the adaptation module is not aware of the VBR nature.
>
>  2. You specify a transmission interval based on packet size. Again,
>     this is not how real media sources behave. Most have a fixed
>     transmission interval (packet time). If MTU is exceeded,
>     back-to-back MTU packets are emitted each interval.
>
> MAR: There is nothing stopping us from such a definition in other test 
> scenarios. The reason why I specified it that way is that that it 
> represents a BEST CASE (smooth) injection of data into the bottleneck. 
> We know the situation only gets worse from there. The impact of the 
> burstiness of a given source is highly dependent on bottleneck service 
> rate and the instantaneous work provided to the bottleneck by of the 
> bursty source.
>
I see a need for another test scenario to represent the "impact of the 
burstiness of a given source".
To me, in Michael's document he wants to get rid of number of 
uncertainties to achieve a simplistic view of the whole system. This is 
fine as long as we remember what we are missing and try to address those 
in later versions or other similar documents. The outcome of such a 
document is the first test that the adaptation algorithm should pass. if 
any algorithm fails to do so there is no point of wasting time on that 
for more complicated testcases.

-- 

Zahed

Ericsson AB
Multimedia Technologies
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43

---

From: Kevin Gross <[email-masked]>
Date: Mon, 29 Apr 2013 11:38:24 -0600

We discussed 1 during the call and I think the resolution was that we
accept this unlimited traffic source as a first cut. Other traffic sources
discussed were fixed rate and variable rate. Fixed rate was deemed not
forward-looking enough. Variable rate was too difficult to quantify.

The conversation brought up an additional question about this document. The
objective here is to start with something simple and build from there. I
support this approach. But, I can think of ways to start even simpler and I
think we should. First off, a two-node network is simpler than the
three-node configuration proposed. On a two-node network, tests could
explore the following scenarios:

   1. No bottleneck and minimum RTT
   2. No bottleneck and various RTT
   3. Bottleneck with infinite queue space and various RTT
   4. Bottleneck with various queue space and various RTT

 I think we all would agree that rmcat should not impede traffic if there's
no contention. The hope is that many of these tests will be uneventful but
that doesn't mean we can skip them.

Kevin Gross
+1-303-447-0517
Media Network Consultant

On Mon, Apr 29, 2013 at 7:37 AM, Michael Ramalho (mramalho) <
[email-masked]> wrote:

>  Kevin,****
>
> ** **
>
> Thanks for your email. I hope to address your points in the design team
> meeting today.****
>
> ** **
>
> In advance of that, let me try to answer here (with “MAR:” below) – and
> then we can further discuss during the conference.****
>
> ** **
>
> Thanks in advance,****
>
> ** **
>
> Michael Ramalho****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Kevin Gross
> *Sent:* Sunday, April 28, 2013 11:48 AM
> *Cc:* Cullen Jennings (fluffy); Toerless Eckert (eckert); Mo Zanaty
> (mzanaty); [email-masked]; Paul Jones (paulej); James Polk (jmpolk)
>
> This looks like a good start.****
>
> ** **
>
> I have a few comments on this proposal.****
>
>    1. You specify sources with infinite amount of data running "flat
>    out". It sounds like such a source, in isolation, would consume all offered
>    fixed bandwidth. All have some maximum bandwidth.****
>
> MAR: You understand what I intended. That is, the sources run similarly to
> so-called “longed lived TCP sources” (i.e., TCP that has an infinite amount
> of data to send).****
>
> MAR: So exactly what types of sources do we admit? Surely real-time audio
> and video … but might we want holographic, multi-dimensional applications
> in twenty years (with much, higher BW requirements)?****
>
> MAR: And shouldn’t the transport layer – as much as feasible – decouple
> itself from the actual application requirements?****
>
> MAR: We are in the middle ground here (almost a layer violation) – we
> desire to build a transport with congestion control USEABLE for real-time
> traffic.****
>
> MAR: So while we should keep real-time requirements front of mind, we
> really need to build a “real-time suitable” transport with congestion
> control that can span the whole range of possible bandwidths (including the
> holodeck).****
>
> MAR: That being said, I do think we must include some real-time
> constraints (e.g., my proposal of packets no FEWER than one per 100 ms) –
> but let’s not constrain the bandwidths of the application.****
>
> MAR: The idea captured in the document was to help ensure that the
> transport is stable – thus constraining the actual generation of data by
> the sources. Then, once we are convinced that the transport protocol has
> all the desirable attributes (self-friendly, stable, etc.) then we can see
> what happens in the “real-world” when the burstiness of real audio/video
> applications are run on top of the transport.****
>
> behavior of the sources. Indeed, the adaptive sources I am building need to
> have the min/max rates provided to them. If their algorithms say they can
> run at higher than the max rate – the sources limit themselves to the max
> rate. At that point, sources in that state (i.e., est. avail. BW > than
> their max) appear like CBR sources to the remaining sources in the
> bottleneck.****
>
>    1. You specify a transmission interval based on packet size. Again,
>    this is not how real media sources behave. Most have a fixed transmission
>    emitted each interval.****
>
> represents a BEST CASE (smooth) injection of data into the bottleneck. We
> know the situation only gets worse from there. The impact of the burstiness
> of a given source is highly dependent on bottleneck service rate and the
> instantaneous work provided to the bottleneck by of the bursty source.
> Proposing a “smooth source” takes out yet another degree of uncertainty. I
> agree it is not reality.****
>
>    1. It is fairly clear from the text what you're trying to do with
>    router and measurement point B.****
>
> MAR: Agree. The diagram should be changed. In my lab I presently only
> inject pure delay or loss at where the routers are shown in the figure with
> routers (simple command line for our routers) … but in general you should
> think of the routers on leg A and B as impairment boxes (loss, jitter,
> delay in either direction).****
>
> MAR: Thanks Kevin.
> ****
>
> Kevin Gross****
>
> +1-303-447-0517****
>
> Media Network Consultant****
>
> AVA Networks - www.AVAnw.com <http://www.avanw.com/>, www.X192.org****
>
> ** **
>
> [email-masked]> wrote:****
>
> RMCAT Design Team,****
>
>  ****
>