Subject: [ippm] Framework for Model Based Measurements needed?

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 24 Apr 2015 17:42:54 +0200

Hi Matt, hi Al,

I'm in the process of reviewing draft-ietf-ippm-model-based-metrics-04... and I 
have to say I'm a little lost... :-(

I understand the idea but it's really hard to follow what the document is giving 
me and what not. In the abstract it says that the doc will define a class of 
metrics...not sure the document is actually defining metrics... than later (2. 
half of intro) it says it "describes a framework for deriving traffic patterns 
and delivery statistics for model based metrics" but don't really think you can 
talk about a framework here. So from my point of view the document describes (as 
state above) traffic pattern and to some extent statistics plus gives a little 
bit of example and a lot of motivational text (which I believe could maybe be 
organized better).

To make my mind up what I'm actually reading I came up with the following drawing:

                    ___________         _______________
                    |         |         |             |
Input Parameter -->|  Model  |         | Measurement |<-- Traffic generator
                    |         |         |             |
                    -----------         ---------------
                         |        Test         |
           target values |       /---\         | IP metrics
                         ------->| + |<---------
                                 \---/
                                   |
                                   \/
                                pass/fail

So in this picture, your document talks about the target values, a little bit 
about the model (which is simply the TCP model) and about the traffic generation 
but leaves out a lot of the stuff in the middle. Matt, if you have understood 
you correctly, you still plan to write a document that describes something 
similar to my drawing above...? If you actually plan to do this, I think that 
would be very helpful and should be published before this draft 
(draft-ietf-ippm-model-based-metrics-04); also because most of the motivation 
and background text should rather go in that new document from my point of view. 
And just for the sake of readability, I would even think about splitting up the 
current draft into potentially 2 documents (model and measurements/tests).

I think the most important (actually quite simple) message for this work is: 
Don't use TCP for measurement traffic generator! However having everything in on 
document, I afraid that this message simply gets lost... :-(

Let me know what you think and if the diagram above is at all similar to what 
you have in mind!

Mirja

---

From: Matt Mathis <[email-masked]>
Date: Fri, 24 Apr 2015 12:49:32 -0700

Thanks. Good comments.   If you find it hard to follow, it has serious
problems.... :-( :-(

A revised version of your diagram would help quite a bit as would other
fixes as you suggest. (and very much highlights things that are not clear),

The other document would be a step up to the larger problem.   E.g. how to
wrap MBM in a framework to support independently verifiable large scale
measurement that can estimate the probability  that users can access the
content they want, at the performance needed to make effective use of the
content.

Re: not using TCP: the real problem is that all throughput maximizing
behaviors, by design cause self inflicted congestion which is extremely
non-linear and conflated with many other signals.     TCP can be used for
measurement if you somehow inhibit congestion control and throughput
maximizing behaviors, for example by clamping cwnd.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

On Fri, Apr 24, 2015 at 8:42 AM, Mirja Kühlewind <
[email-masked]> wrote:

> Hi Matt, hi Al,
>
> I'm in the process of reviewing draft-ietf-ippm-model-based-metrics-04...
> and I have to say I'm a little lost... :-(
>
> I understand the idea but it's really hard to follow what the document is
> giving me and what not. In the abstract it says that the doc will define a
> class of metrics...not sure the document is actually defining metrics...
> than later (2. half of intro) it says it "describes a framework for
> deriving traffic patterns and delivery statistics for model based metrics"
> but don't really think you can talk about a framework here. So from my
> point of view the document describes (as state above) traffic pattern and
> to some extent statistics plus gives a little bit of example and a lot of
> motivational text (which I believe could maybe be organized better).
>
> So in this picture, your document talks about the target values, a little
> bit about the model (which is simply the TCP model) and about the traffic
> generation but leaves out a lot of the stuff in the middle. Matt, if you
> have understood you correctly, you still plan to write a document that
> describes something similar to my drawing above...? If you actually plan to
> do this, I think that would be very helpful and should be published before
> this draft (draft-ietf-ippm-model-based-metrics-04); also because most of
> the motivation and background text should rather go in that new document
> from my point of view. And just for the sake of readability, I would even
> think about splitting up the current draft into potentially 2 documents
> (model and measurements/tests).
>
> I think the most important (actually quite simple) message for this work
> is: Don't use TCP for measurement traffic generator! However having
> everything in on document, I afraid that this message simply gets lost...
> :-(
>
> Let me know what you think and if the diagram above is at all similar to
> what you have in mind!
>
> Mirja
>
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 28 Apr 2015 19:42:19 +0200

Hi Matt,

okay, how should we continue on this? Should I finish my review and provide more 
detailed feedback or you do want to make another revision first? If this is 
going to be a larger revision, I'd definitely prefer the second option! :-)

If you do a revision now, one more comment: I'd recommend to move the 
introductional part of section 4 as well section 4.2 (everything except the 
subsection 4.1) to the beginning of the doc because it a nice read and motivates 
the problem well.

Re-Re: not using TCP: Jup, the message is not use a full TCP. And also if you 
don't use TCP, bandwidth is not the only metric you need to look at! These 
points should also be stated at the very beginning of the doc!

Mirja

On 24.04.2015 21:49, Matt Mathis wrote:
> A revised version of your diagram would help quite a bit as would other fixes as
> you suggest. (and very much highlights things that are not clear),
>
> The other document would be a step up to the larger problem.   E.g. how to wrap
> MBM in a framework to support independently verifiable large scale measurement
> that can estimate the probability  that users can access the content they want,
> at the performance needed to make effective use of the content.
>
> Re: not using TCP: the real problem is that all throughput maximizing behaviors,
> by design cause self inflicted congestion which is extremely non-linear and
> conflated with many other signals.     TCP can be used for measurement if you
> somehow inhibit congestion control and throughput maximizing behaviors, for
> example by clamping cwnd.
>
> Thanks,
> --MM--
> Privacy matters!  We know from recent events that people are using our services
> to speak in defiance of unjust governments.   We treat privacy and security as
> matters of life and death, because for some users, they are.
>
> On Fri, Apr 24, 2015 at 8:42 AM, Mirja Kühlewind
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hi Matt, hi Al,
>
>     than later (2. half of intro) it says it "describes a framework for deriving
>     traffic patterns and delivery statistics for model based metrics" but don't
>     really think you can talk about a framework here. So from my point of view
>     the document describes (as state above) traffic pattern and to some extent
>     statistics plus gives a little bit of example and a lot of motivational text
>     (which I believe could maybe be organized better).
>
>     Don't use TCP for measurement traffic generator! However having everything
>     in on document, I afraid that this message simply gets lost... :-(
>

-- 
------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Communication Systems Group
Institute TIK, ETH Zürich
Gloriastrasse 35, 8092 Zürich, Switzerland

Room ETZ G93
phone: +41 44 63 26932
email: [email-masked]
------------------------------------------

---

From: Matt Mathis <[email-masked]>
Date: Wed, 29 Apr 2015 08:03:11 -0700

Oops, I missed "in the process of"

Are you comfortable with Google docs and hangouts?  There is an author's
notes doc that I will give you write. (best for staging alternate text and
diagrams).  I would also be happy to give you comment/suggest permission on
the source doc.  (But I need a google address for you).

Thanks,
--MM--

On Tue, Apr 28, 2015 at 10:42 AM, Mirja Kühlewind <
[email-masked]> wrote:

> Hi Matt,
>
> okay, how should we continue on this? Should I finish my review and
> provide more detailed feedback or you do want to make another revision
> first? If this is going to be a larger revision, I'd definitely prefer the
> second option! :-)
>
> subsection 4.1) to the beginning of the doc because it a nice read and
> motivates the problem well.
>
> Re-Re: not using TCP: Jup, the message is not use a full TCP. And also if
> you don't use TCP, bandwidth is not the only metric you need to look at!
> These points should also be stated at the very beginning of the doc!
>
> Mirja
>
>
>> fixes as
>> behaviors,
>> services
>>     I'm in the process of reviewing
>> draft-ietf-ippm-model-based-metrics-04...
>> deriving
>>     To make my mind up what I'm actually reading I came up with the
>> following
>>     drawing:
>>