Subject: [iccrg] Fwd: draft-irtf-iccrg-tcpeval-01.txt - reviewers needed!

From: Michael Welzl <[email-masked]>
Date: Tue, 21 Oct 2014 14:21:08 +0200

Forwarding to the list with permission...

We need more comments.

Begin forwarded message:

> From: David Hayes <[email-masked]>
> Subject: Re: draft-irtf-iccrg-tcpeval-01.txt - reviewers needed!
> Date: 20. oktober 2014 16:09:49 GMT+02:00
> To: Mirja Kühlewind <[email-masked]>
> Cc: Michael Welzl <[email-masked]>, David Ros <[email-masked]>, <[email-masked]>
> 
> Hi Mirja,
> 
> Thank you for your reply. I have replied in text and hope it is helpful.
> 
> Kind regards,
> 
> David
> 
> 
> On Mon, 20 Oct 2014 13:01:33 +0200
> Michael Welzl <[email-masked]> wrote:
> 
>> Hi,
>> 
>> Thanks a lot for your answer. I'm not a co-author of this document,
>> so actually the Davids should comment on the technical bits.
>> 
>> As ICCRG chair, it's in my interest to get this document done and
>> shipped, and not change it much. There was much preceding debate
>> leading to this document, see:
>> http://www.caia.swin.edu.au/cv/landrew/pubs/TCP-suite-PFLDnet.pdf
>> for how it all started... so I guess the point of this document is
>> more to have a common ground and to have a stable, published version
>> of a document that people have already cited quite a bit.
>> 
>> Having a larger / updated / different test suite is interesting work
>> and would be nice to do in ICCRG, by whoever is interested in it. I'm
>> not personally so much interested in test suites, but I'm interested
>> in seeing activity in ICCRG, and I think we could be a nice forum for
>> such work. But that should be a follow-up when the test suite is out
>> there and published in my opinion.
>> 
>> But I'll let the Davids speak!
>> Michael
>> 
>> 
>> 
>> On 17. okt. 2014, at 15:34, Mirja Kühlewind wrote:
>> 
>>> Hi Michael,
>>> 
>>> I've read the earlier document and most of the new stuff in this
>>> document but I have to admit that, at least for my own (thesis)
>>> evaluation with our own VM-based simulation tool, I ended up not
>>> using these scenarios. In general I'm very supportive for this
>>> work, however, my main concern is about the traffic generation. For
>>> our tool it was really hard to generate a high enough traffic load
>>> (for a long enough time). In fact we were limited by the number of
>>> active sockets. Even though there have never been too much sockets
>>> open at the same time, they still seems to be blocked. I didn't dig
>>> into that any further because I in the end preferred different
>>> evaluation scenarios where I was able to easier draw conclusions.
>>> And that is also my second point about the traffic model. As this
>>> is mostly short flows, the main influence on performance is based
>>> on the Slow Start behavior. However, at least for my evaluation I
>>> didn't change the Slow Start behavior (but only Congestion
>>> Avoidance).
>>> 
> 
> The purpose of the common TCP Evaluation Suite is not provide a
> complete set of scenarios for an exhaustive evaluation of a new TCP
> extension. Many new extensions solve particular problems, and need to
> be tested under scenarios that illustrate the problem and the
> solutions. The evaluation suite does not preclude this, but rather
> complements it.
> 
> Its objective is to provide a series of standard tests, many with
> realistic Internet traffic, to paint a broader comparable picture of
> how the new extension behaves - without making the tests too onerous for
> the researcher. There was meant to be companion implementations to
> facilitate this. A testbed (at one time it was being developed at
> CalTech) and a ns2 implementation (available on bitbucket
> implementing most of the scenarios).
> 
> In some sense it is meant to help answer the following questions: Yes,
> the particular extension helps solve this particular problem; but will
> it cause problems if deployed in the Internet? How might this extension
> be broadly compared with another proposal?" 
> 
> Most tests use the Tmix traces as background traffic, some as the main
> traffic, but not all. There are tests with long lived flows.
> 
> I think that the abstract and introduction could be tightened to better
> explain this.
> 
> 
>>> I would actually prefer to define a set of more simple scenarios
>>> with a small number of long-living flows (but e.g. transient effect
>>> of starting and stopping flows) and potentially some scenarios with
>>> an artificial traffic model inducing short flows (as background
>>> traffic) with a given flow size and IAT distribution. Something
>>> similar is true for e.g. the wireless scenario. Instead of leaving
>>> the model for such a link open, I would recommend scenarios with
>>> additional delay/rate variation at the bottleneck or even
>>> additional artificial losses based on a simple loss model (with
>>> either a certain loss probability or maybe some kind of burst
>>> losses). Again this is more far away from reality but might provide
>>> even more extreme cases and allows an easier comparison and
>>> interpretation of results.
>>> 
> 
> A *realistic* model based traffic generator that captured the dynamics
> and proportions of flows in the Internet (and maybe the future
> Internet) could be good. It is my understanding (I was not at the "round
> table") that this was vigorously debated and that it was decided to use
> traffic based on real traces. I understand that this decision was
> related to how realistic models could be, and that models tended to
> miss some elements of real traffic that could be important.
> 
> This said, traffic dynamics change and I think there should be a way to
> revise the traffic to reflect this (the ns2 implementation goes some
> way to allowing this). However, I am not sure that this should be
> within the scope of the current document.
> 
>>> I'm not sure what your plan is for this document and if my current
>>> input is helpful therefore.
> 
> This is an old document that ICCRG inherited from the TMRG. Various
> forms of it (though incomplete and sketchy) have been used by
> researchers. The plan was to quickly complete the document with its
> original aims (no one really has time to work on major changes in
> direction or scenarios). This will provide something more concrete for
> people to cite, draw upon and build upon. In its current never ending
> transient state it is less useful.
> 
>>> If you would like to get some more
>>> input e.g. which scenarios I finally used for evaluation, we should
>>> maybe sit together and discuss (maybe also with other people how
>>> recently performed some CC/AQM evaluation studies (e.g. 1-2 people
>>> from rmcat or aqm working group).
> 
> Actually the original document and ns2 implementation had some AQM
> tests. They were more directed at how TCP behaves with standard (at the
> time RED) AQMs than testing AQMs. This was removed from the document
> because of the ongoing work with new AQMs and there being no clear
> candidate to have a standard test with.
> 
> 
>>> 
>>> If you want I could post some of my points (including the one I
>>> raised earlier on scenarios with overload...?) on the ICCRG
>>> mailing. But maybe it is more useful to sit together first...?
>>> Otherwise if I should just review the document as it is, I would
>>> need to know your intended plan about this document.
>>> 
>>> Mirja
>>> 
>>> 
>>> On 17.10.2014 12:30, Michael Welzl wrote:
>>>> Hi Mirja,
>>>> 
>>>> If I remember correctly, you were interested in using the doc.
>>>> Maybe you've even already read it? => May I please sign you up as
>>>> a reviewer?
>>>> 
>>>> Cheers,
>>>> Michael
>>>> 
>>>> 
>>>> Begin forwarded message:
>>>> 
>>>>> *From: *Michael Welzl <[email-masked]
>>>>> <mailto:[email-masked]>> *Subject:
>>>>> **draft-irtf-iccrg-tcpeval-01.txt - reviewers needed!* *Date:
>>>>> *17. oktober 2014 12:29:21 GMT+02:00 *To: [email-masked]
>>>>> <mailto:[email-masked]>
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> We haven't received any comments in response to this old message
>>>>> from July. There's now a page with additional information:
>>>>> http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG_tcpeval
>>>>> 
>>>>> and we need volunteers to review and comment on the document.
>>>>> Please, people, do raise your hands!
>>>>> 
>>>>> Michael
>>>>> 
>>>>> 
>>>>> 
>>>>> On 4. juli 2014, at 14:11, David Ros wrote:
>>>>> 
>>>>>> Dear all,
>>>>>> 
>>>>>> After a long hiatus, we have finally posted an update to the TCP
>>>>>> evaluation suite. Comments are welcome.
>>>>>> 
>>>>>> Besides mostly editorial fixes, the main changes wrt version -00
>>>>>> that David Hayes presented in Berlin (*) concern parameter
>>>>>> values for several scenarios (some have been fixed, some have
>>>>>> been added).
>>>>>> 
>>>>>> (*) We just realised version -00 never got posted to the IETF
>>>>>> datatracker, so there was only a "privately-hosted" version
>>>>>> online; we've just fixed this omission. Our apologies.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> David (as individual)
>>>>>> 
>>>>>> 
>>>>>> Begin forwarded message:
>>>>>> 
>>>>>>> From: [email-masked] <mailto:[email-masked]>
>>>>>>> Subject: New Version Notification for
>>>>>>> draft-irtf-iccrg-tcpeval-01.txt Date: 4 Jul 2014 13:50:53 GMT+2
>>>>>>> To: Lachlan L.H. Andrew <[email-masked]
>>>>>>> <mailto:[email-masked]>>, "Lachlan L.H. Andrew"
>>>>>>> <[email-masked] <mailto:[email-masked]>>,
>>>>>>> David Hayes <[email-masked] <mailto:[email-masked]>>,
>>>>>>> Sally Floyd <[email-masked] <mailto:[email-masked]>>, David Ros
>>>>>>> <[email-masked] <mailto:[email-masked]>>, "David Ros"
>>>>>>> <[email-masked] <mailto:[email-masked]>>, "Sally Floyd"
>>>>>>> <[email-masked] <mailto:[email-masked]>>, "David Hayes"
>>>>>>> <[email-masked] <mailto:[email-masked]>>
>>>>>>> 
>>>>>>> 
>>>>>>> A new version of I-D, draft-irtf-iccrg-tcpeval-01.txt
>>>>>>> has been successfully submitted by David Ros and posted to the
>>>>>>> IETF repository.
>>>>>>> 
>>>>>>> Name:draft-irtf-iccrg-tcpeval
>>>>>>> Revision:01
>>>>>>> Title:Common TCP Evaluation Suite
>>>>>>> Document date:2014-07-04
>>>>>>> Group:iccrg
>>>>>>> Pages:34
>>>>>>> URL:
>>>>>>> http://www.ietf.org/internet-drafts/draft-irtf-iccrg-tcpeval-01.txt
>>>>>>> Status:
>>>>>>> https://datatracker.ietf.org/doc/draft-irtf-iccrg-tcpeval/
>>>>>>> Htmlized:
>>>>>>> http://tools.ietf.org/html/draft-irtf-iccrg-tcpeval-01 Diff:
>>>>>>> http://www.ietf.org/rfcdiff?url2=draft-irtf-iccrg-tcpeval-01
>>>>>>> 
>>>>>>> Abstract:
>>>>>>> This document presents an evaluation test suite for the initial
>>>>>>> assessment of proposed TCP modifications.  The goal of the test
>>>>>>> suite is to allow researchers to quickly and easily evaluate
>>>>>>> their proposed TCP extensions in simulators and testbeds using
>>>>>>> a common set of well- defined, standard test cases, in order to
>>>>>>> compare and contrast proposals against standard TCP as well as
>>>>>>> other proposed modifications.  This test suite is not intended
>>>>>>> to result in an exhaustive evaluation of a proposed TCP
>>>>>>> modification or new congestion control mechanism.  Instead, the
>>>>>>> focus is on quickly and easily generating an initial evaluation
>>>>>>> report that allows the networking community to understand and
>>>>>>> discuss the behavioral aspects of a new proposal, in order to
>>>>>>> guide further experimentation that will be needed to fully
>>>>>>> investigate the specific aspects of such proposal.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Please note that it may take a couple of minutes from the time
>>>>>>> of submission until the htmlized version and diff are available
>>>>>>> at tools.ietf.org <http://tools.ietf.org/>.
>>>>>>> 
>>>>>>> The IETF Secretariat
>>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> iccrg mailing list
>>>>>> [email-masked] <mailto:[email-masked]>
>>>>>> https://www.irtf.org/mailman/listinfo/iccrg
>>>>> 
>>>> 
>>> 
>>> -- 
>>> ------------------------------------------
>>> Dipl.-Ing. Mirja Kühlewind
>>> Communication Systems Group
>>> Institute TIK, ETH Zürich
>>> Gloriastrasse 35, 8092 Zürich, Switzerland
>>> 
>>> Room ETZ G93
>>> phone: +41 44 63 26932
>>> email: [email-masked]
>>> ------------------------------------------
>> 
> 
> 
> 
> -- 
> 
> -------------------------
> David Hayes
> [email-masked]
> Department of Informatics
> University of Oslo