Subject: Re: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

From: Barry Constantine <[email-masked]>
Date: Fri, 3 Sep 2010 11:07:33 -0700

Hello Rudi,

I think your suggestion is reasonable.

Never hurts to be explicit, so we will add Framework to the title as you mentioned.

We'll title it:  "Framework for TCP Throughput Testing" and add some reference to framework in the intro.

Thanks again,

Barry

Principal Member of Technical Staff

JDSU Communication Test (formerly Acterna)

Emerging Markets and Technology Research         

One Milestone Center Court                              

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, August 31, 2010 5:51 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Henk,

If time allows, I'll send more comments.

If not, here's my view:

Change the document title to

"Framework for a TCP Throughput Testing Methodology"

Arguments:

The document as is does not contain any Type P Specification. See definitions in

"2.2 Metrics for TCP Throughput Tests"

The document only gives guidance on how to measure RTT, bandwidth and some auiliary 
parameters as RTT jitter (it does not specify how to measure these or refer to 
specifications of related measurements). I don't recall having seen references to 
prior related work of IPPM (last read version was draft -05). 

All in all, I doubt that two implementors are able to measure compatible metrics 
based on the documents definitions as they are by now. Even if it is only 
"Informational" I'd prefer classification as a "framework". Many or most other 
informational documents produced by IPPM contain Type P specifications as soon as 
metrics are to be established.

Regards,

Ruediger

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henk Uijterwaal
Sent: Tuesday, August 31, 2010 10:23 AM
To: IETF IPPM WG
Subject: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

IPPM Group,

This is a Working Group Last Call for the draft:

    TCP Throughput Testing Methodology
    draft-ietf-ippm-tcp-throughput-tm-06.txt

This version of the draft addresses all issues brought up on the list and
at the Maastricht meeting.  It also fixes a number of ID-nits.  The authors
believe that no further work on the draft is needed.  Note that there are
a few spelling mistakes in the draft, these will be fixed when including
any other LC comments.

Please review the draft and raise any issues by Monday, September 20,
9:00 UTC.  An URL for the draft is:

    http://www.ietf.org/id/draft-ietf-ippm-tcp-throughput-tm-06.txt

Matt & Henk

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.xs4all.nl/~henku
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

I confirm today what I denied yesterday.            Anonymous Politician.
_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm
_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm

---

From: <[email-masked]>
Date: Mon, 6 Sep 2010 12:51:11 +0200

Hello Barry,

that's great. 

Are you still interested in more detailed comments (like on Type-P proposals - I'm admittedly not an expert in specifying them), or do you want to finish the document largely as is? If my time allows, I can send some more detailed comments. But I don't want to annoy you (and would myself be happy if you tell me, I can save the effort).

Regards, Ruediger

Deutsche Telekom Netzproduktion GmbH
Zentrum Technik Einführung
Rüdiger Geib
Heinrich-Hertz-Straße 3-7, 64295 Darmstadt
+49 6151 628-2747 (Tel.)
+49 251 7985 209 (Fax)
http://www.telekom.com

Erleben, was verbindet.

Deutsche Telekom Netzproduktion GmbH
Aufsichtsrat: Dr. Steffen Roehn (Vorsitzender)
Geschäftsführung: Dr. Bruno Jacobfeuerborn (Vorsitzender), Albert Matheis, Klaus Peren
Handelsregister: Amtsgericht Bonn HRB 14190
Sitz der Gesellschaft: Bonn
USt-IdNr. DE 814645262
-----Original Message-----
From: Barry Constantine [mailto:[email-masked]] 
Sent: Friday, September 03, 2010 8:08 PM
To: Geib, Rüdiger; [email-masked]
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Hello Rudi,

Thanks again,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, August 31, 2010 5:51 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Henk,

If not, here's my view:

Arguments:

Regards,

Ruediger

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henk Uijterwaal
Sent: Tuesday, August 31, 2010 10:23 AM
To: IETF IPPM WG
Subject: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

IPPM Group,

Matt & Henk

---

From: Barry Constantine <[email-masked]>
Date: Wed, 8 Sep 2010 17:12:54 -0700

Hello Rudi,

Sorry for the delayed response.

We appreciate all the comments from the working group along the way to draft-06; the test method has improved tremendously due to all of this interaction.

We would prefer to wrap this up at this point and appreciate your directness in your question.

Thanks,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, September 06, 2010 6:51 AM
To: Barry Constantine
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Hello Barry,

that's great. 

Regards, Ruediger

Erleben, was verbindet.

-----Original Message-----
From: Barry Constantine [mailto:[email-masked]] 
Sent: Friday, September 03, 2010 8:08 PM
To: Geib, Rüdiger; [email-masked]
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Hello Rudi,

Thanks again,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, August 31, 2010 5:51 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Henk,

If not, here's my view:

Arguments:

Regards,

Ruediger

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henk Uijterwaal
Sent: Tuesday, August 31, 2010 10:23 AM
To: IETF IPPM WG
Subject: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

IPPM Group,

Matt & Henk

---

From: Henk Uijterwaal <[email-masked]>
Date: Thu, 09 Sep 2010 15:17:26 +0200

Hi Barry and others,

>     draft-ietf-ippm-tcp-throughput-tm-06.txt

Details, as a participant, not WG chair.

1. Introduction: I think the first paragraph has to come after the 1st
   sentence of the 2nd one.  TCP testing is useful in itself, turn-up
   testing is an use case.

2. Section 2: "Some specific goals".  Remove some, either you list all
   goals or you explicitely state which goals are omitted.

3. Section 2.2.  With this definition, tcp efficiency can become <0%,
   which is hard to interpret.  (Suppose all bytes are resent twice,
   admittedly a strange case).  Work-around: if <0, make it 0.  (The
   connection is not useful anyway).

4. Section 3, item #2.  For Round Trip, put in a reference to the RFC
   on 2-way delay.  If this is clear, then 2 results from different
   vendors will be more comparable.

5. Section 3.2.2.  I think we should either select one or al least
   make it a MUST that this is reported.  Otherwise it will be hard to
   compare results.

6. Section 6: Drop the first paragraph. "The author" is undefined with
   4 authors.

I already sent you a couple of spelling errors.

Henk

---

From: Barry Constantine <[email-masked]>
Date: Thu, 9 Sep 2010 07:04:06 -0700

Very good comments, Henk I will them all to the next draft.

Thanks again,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henk Uijterwaal
Sent: Thursday, September 09, 2010 9:17 AM
To: [email-masked]
Subject: Re: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Hi Barry and others,

Henk

---

From: <[email-masked]>
Date: Fri, 10 Sep 2010 15:26:26 +0200

Hi Barry,

Part 1 of the comments. Edirorial text changes express my taste. I don't insist on them. Comments Part 2 follows, if time allows.

Regards, Ruediger

-----

1. Introduction - Editorial

Last section may be reworded to

   This framework proposes a test which should be performed in addisiont 
   to traditional Layer 2/3 type tests, which are conducted to verify the 
   integrity of the network before conducting TCP tests. Examples include 
   RFC 2544 [RFC2544], iperf (UDP mode), or manual packet layer test 
   techniques where packet throughput, loss, and delay measurements 
   are conducted.
----

2. Goals of this Methodology

...are not intended to be measured or analyzed by such a methodology.

Please add
   - a method to operate permanently with high measurement loads. TCP 
     performance and optimisation data of operational networks may be 
     captured and evaluated by using data of the "TCP Extended 
     Statistics MIB" [RFC4898]. 

<-- Please add a reference to RFC 4898 or explain why it can't be used -->

----

2.2 Metrics for TCP Throughput

Henk had a comment here which may have been caused by the lack of a 
definition of "transmitted bytes". Please add a definition of the terms 
"Transmitted bytes" and "retransmitted Bytes". If the former is the sum of 
"Succesfully transmitted bytes" and "retransmitted bytes", that would solve 
Henk's problem, as then Transmitted Bytes > Retransmitted Bytes. 

Effectively your formula then is: received bytes/transmitted bytes.

You introduce some type P like singletons in the following:

- type-P-TCP Transmitted Bytes
- type-P-TCP Retransmitted Bytes
- type-P-TCP Efficiency (you've defined a formula)

These depend on a number of parameters which may vary:
RTT, Packet Loss, MSS, Path Capacity, MTU size and the time when 
you've measured.

You then introduce another two type P like singletons, 

- type-P-TCP Actual Transfer Time
- type-P-TCP Transfer Index

In adition to the above parameters, both depend on a data block size 
and the transfer index in addition depends on an "ideal TCP transfer 
time".

There's no definition of the "ideal TCP transfer time" - at least I didn't 
see any before or immediately after the Transfer Index is introduced.  
This should be added.

I've tried to scetch some Type-P language here and based it on your 
input. I leave it to you to pick it up. I'm not an expert on type p 
definitions, this definitely requires more work.

----------Finish for today--------

---

From: Barry Constantine <[email-masked]>
Date: Fri, 10 Sep 2010 06:44:21 -0700

Rudi,

Thank you for your additional comments.

I thought we agreed that if this were to be retitled a framework (and informational versus a standard), then the Type P metrics could remain unaddressed.

Please let me know if I am misunderstanding, will definitely incorporate your comments in this email (minus the Type P metric work).

Thanks,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, September 10, 2010 9:26 AM
To: Barry Constantine
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Hi Barry,

Regards, Ruediger

-----

Please add

----

----------Finish for today--------

---

From: Al Morton <[email-masked]>
Date: Fri, 10 Sep 2010 10:13:53 -0400

Barry, Rudi,

My recollection from the March IPPM session is that
  - we agreed to take this up as an Informational RFC
  - we aren't defining metrics here
   (if we were, it would be standards track)

So, I agree with the "Framework" title change.
I would clarify that "Type-P" metric specifications
are un-attempted with this scope (what the draft does now).

I will hopefully add some more comments later today (sigh).

regards,
Al

At 09:44 AM 9/10/2010, Barry Constantine wrote:
>...
>I thought we agreed that if this were to be retitled a framework 
>(and informational versus a standard), then the Type P metrics could 
>remain unaddressed.
>
>Please let me know if I am misunderstanding, will definitely 
>incorporate your comments in this email (minus the Type P metric work).
>
>
>Thanks,
>
>Barry
>
>
>
>Part 1 of the comments. Edirorial text changes express my taste. I 
>don't insist on them. Comments Part 2 follows, if time allows.
>
>Regards, Ruediger
>
>-----
>

---

From: <[email-masked]>
Date: Mon, 13 Sep 2010 10:51:04 +0200

Hi Barry,

I don't insist on the Type-P stuff, the comment was meant as an offer, should you want to go there.

Part 2 of my comments will follow this mail.

Regards,

Ruediger

-----Original Message-----
From: Barry Constantine [mailto:[email-masked]] 
Sent: Friday, September 10, 2010 3:44 PM
To: Geib, Rüdiger
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Rudi,

Thanks,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, September 10, 2010 9:26 AM
To: Barry Constantine
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Hi Barry,

Regards, Ruediger

-----

Please add

----

----------Finish for today--------

---

From: <[email-masked]>
Date: Mon, 13 Sep 2010 10:55:12 +0200

Hi Barry,

Part 2 of the comments. Edirorial text changes express my taste. I don't insist on them. Comments Part 3 follows, if time allows.

Regards, Ruediger

-----

A general comment: the document does not offer any section on 
terminology or test set up architecture. This causes a fair 
share of my questions.

-----

3. TCP Throughput Testing Methodology

....
4. Traffic Management Tests.  ....
   Multiple connection testing can verify that the network is configured
   properly for traffic shaping versus policing, various queueing 
   implementations, and RED.

<Please clarify which network element configurations are supposed to be 
characterised that way, eg. CE routers, PE routers, P routers or all 
network elements along a path. If PE or P routers are to be tested, 
please add a statement whether you expect such tests to be carried 
out by the operational staff of the provider or in cooperation with 
these.>

----

Same section, later

   Whether the TCP test host is a standard computer or dedicated test
   instrument, the following areas should be considered when selecting
   a test host:

   - TCP implementation used by the test host OS, i.e. Linux OS kernel 
   using TCP Reno, TCP options supported, etc.  This will obviously be 
   more important when using custom test equipment where the TCP 
   implementation may be customized or tuned to run in higher 
   performance hardware

<If there is any requirement to use a standard conformant TCP 
implementation, the text should express this using standard language. 
Is it clear to any reader why exactly the TCP implementation 
is an issue by just reading the above text? Or do you expect a certain 
know how level to understand this issue?  MUST or SHOULD the 
implementation used by the test equipment be recorded when performing 
a test?>

----

Same section, later

   - Most importantly, the TCP test host must be capable of generating 
   and receiving stateful TCP test traffic at the full link speed of the 
   network under test. As a general rule of thumb, testing TCP 
   throughput at rates greater than 100 Mbit/sec generally requires high
   performance server hardware or dedicated hardware based test tools.

<Is the first "must" meant to be a "MUST"?>
<Please clarify whether the "network under test" is the Layer 2/3 VPN with 
the access bandwidth agreed for this network or the complete provider 
network accross which the Layer 2/3 VPN is produced.>

----

Same section, later

   - To measure RTT and TCP Efficiency per connection, this will 
   generally require dedicated hardware based test tools. In the absence
   of dedicated hardware based test tools, these measurements may need 
   to be conducted with packet capture tools (conduct TCP throughput 
   tests and analyze RTT and retransmission results with packet 
   captures).

<Please add a reference for the "TCP Extended Statistics MIB" [RFC4898] 
and adapt text, as this might be a way get around dedicated test 
equipment and packet capture tools.> 

<Please add an additional section expressing something like:>

  - The test equipment and its acces to the network under test MUST NOT 
introduce a performance bottleneck of any kind.

----------Finish for today--------

---

From: Barry Constantine <[email-masked]>
Date: Mon, 13 Sep 2010 06:19:00 -0700

Hi Rudi,

Just to properly close these comments; we agree to the Section 1. and 2. comments and will incorporate this into the final draft.

Looking at your comments from today and will get back to you this week.

Thanks,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, September 10, 2010 9:26 AM
To: Barry Constantine
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Hi Barry,

Regards, Ruediger

-----

Please add

----

----------Finish for today--------

---

From: <[email-masked]>
Date: Thu, 16 Sep 2010 16:26:40 +0200

Hi Barry,

Part 3 of the comments. 

Regards, Ruediger

-----

Sections 3.1 and 3.2 (MTU, Round Trip and Bandwidth Measurement)

The text does not consider the effect of ECMP (IP load sharing). 
As soon as Flow Aware Transport (FAT) is deployed, the delays will 
vary depending on the path taken through the network. If the 
document doesn't consider the application of ECMP with FAT, 
please state so explicitely.

I've missed references to RFC 2681 (RTD) and RFC 5136 (network 
capacity). On the latter, I'd be interested which of the metrics 
proposed there would determine "bandwidth" in the sense of your 
draft.

With regard to RTT determination, a reference to RFC 4898 should 
be included.

Section 3.2

..."Since this 
   methodology requires that RTT be measured during the entire 
   throughput test, the extent by which the RTT varied during the 
   throughput test can be quantified."

I don't recall any prior statement that the RTT must/MUST be 
measured during the entire test and the metrics suggested don't 
require RTT measurement. If you want to define an additional 
metric here, please provide more text and a metric definition. 
The same applies to section 3.2.2:

  ..."Ideally, the bandwidth test should produce a log output of the 
   bandwidth achieved across the test interval AND the round trip delay.  

   And during the actual TCP level performance measurements (Sections
   3.3 - 3.5), the test tool must be able to track round trip time
   of the TCP connection(s) during the test.  Measuring round trip time
   variation (aka "jitter") provides insight into effects of congestive
   delay on the sustained throughput achieved for the TCP layer test."

In addition to an RTT based metric, here you seem to suggest another 
metric linking RTT and bandwidth measurement.

-----

Section 3.3.1

The diagrams at the end of the section are certainly interesting 
illustrations, but they may be moved to an appendix. 

-----

Section 3.3.2 after

..."Dedicated test equipment will generally be required, especially for 
   line rates of GigE and 10 GigE."

Should you create a measurement load of more than 10% of any provider 
backbone link bandwidth on a packet based network, you SHOULD agree 
your testing activity with the network operating center prior to 
performing these tests.

The same sentence may be added to section 3.4 Traffic management tests, 
after the same sentence as above.

-----
Section 3.3.2

   "The TCP throughput test should be run over a a long enough duration 
   to properly exercise network buffers and also characterize
   performance during different time periods of the day.  The results 
   must be logged at the desired interval and the test must record RTT 
   and TCP retransmissions at each interval. 

   This correlation of retransmissions and RTT over the course of the 
   test will clearly identify which portions of the transfer reached
   TCP Equilbrium state and to what effect increased RTT (congestive
   effects) may have been the cause of reduced equilibrium performance.

   Additionally, the TCP Efficiency and TCP Transfer time metrics should
   be logged in order to further characterize the window size tests."

Please clarify and define in your text, which metrics are of true 
relevance and which are nice to have. You apply strong requirement 
language here to capture a metric which isn't defined in your document 
(retransmission to RTT correlation). The metrics you've defined so far 
"should be performed additionally". This raises the impression that the
truly required metrics of this document aren't explicitely defined, 
I think.

-----

Finally, I wonder whether any default settings may be mentioned or 
approximately validated (you can't always reach them):

- max MTU (should be part of a contract - respect layering aspects)
- access bandwidth (may well be the bottleneck).
- a maximum delay (which may be part of an SLA)
- a default TCP window setting (you mention 64KB)

----------That's it--------

---

From: Barry Constantine <[email-masked]>
Date: Thu, 16 Sep 2010 18:02:04 -0700

Hi Rudi,

We really appreciate you taking the time to provide really good comments.

All have been very insightful, especially this last round.

I'll contact you off-list and I would like to schedule time to review all comments.  I think we can more efficiently work through these collaboratively (real-time).

Thanks again,

Barry

Germantown, MD 20876                                         

(W) 240-404-2227                                                

(C) 301-325-7069

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, September 16, 2010 10:27 AM
To: Barry Constantine
Cc: [email-masked]
Subject: RE: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt - part 3

Hi Barry,

Part 3 of the comments. 

Regards, Ruediger

-----

Section 3.2

..."Since this 

-----

Section 3.3.1

-----

Section 3.3.2 after

-----
Section 3.3.2

-----

----------That's it--------

---

From: Henk Uijterwaal <[email-masked]>
Date: Mon, 20 Sep 2010 09:34:37 +0200

IPPM group,

>     http://www.ietf.org/id/draft-ietf-ippm-tcp-throughput-tm-06.txt

A number of issues have been raised, which makes it hard to declare
consensus on this document.  The authors are invited to submit
a new draft.  This will be followed by a short second WGLC to make
sure that all issues have been addressed.

Matt & Henk.

---

From: Barry Constantine <[email-masked]>
Date: Mon, 20 Sep 2010 05:03:50 -0700

That makes sense Henk.

We're going to work on this directly with Rudi Geib (majority of comments) and hope to have new draft available sometime this week.

Thanks,
Barry

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Henk Uijterwaal [[email-masked]]
Sent: Monday, September 20, 2010 3:34 AM
To: [email-masked]
Subject: Re: [ippm] End of WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

IPPM group,

Matt & Henk.

---

From: Al Morton <[email-masked]>
Date: Mon, 20 Sep 2010 08:07:18 -0400

Henk, IPPM,

 >     http://www.ietf.org/id/draft-ietf-ippm-tcp-throughput-tm-06.txt

I didn't remember the 9:00 UTC aspect, just that the 20th was the
deadline...

I have the following comments on  *-ippm-tcp-throughput-tm-06.txt
(written on the plane yesterday, so at least they were prepared on time).
There may be some overlap with earlier comments, I didn't have access
to all of them while reading.

Al

S1 - Intro

p1 current text:
>    Testing an operational network prior to customer activation is
>    referred to as "turn-up" testing and the SLA (Service Level
>    Agreement) is generally based upon Layer 2/3 packet throughput,
>    delay, loss and jitter.

I don't think this methodology can be the basis for SLA on throughput,
even TCP throughput, because of the many reasons for variability in
the results and the fact that IPPM is not writing standards track metrics
in this draft. I have further comments along these lines, but it would
suffice to replace throughput with information rate for now.
Other suggested edits below:

NEW:
    Testing an operational network prior to customer activation is
    referred to as "turn-up" testing and the SLA (Service Level
    Agreement) is generally based upon Layer 2/3 information rate,
    packet delay, loss and delay variation.      ^^^^^^^^^^^^^^^^^
    ^^^^^^                 ^^^^^^^^^^^^^^^^

-------------------------------------------------------------------------
p7 current text:

>    One other important note, it is highly recommended that traditional
>    Layer 2/3 type tests are conducted to verify the integrity of the
>    network before conducting TCP tests.  Examples include RFC 2544
>    [RFC2544], iperf (UDP mode), or manual packet layer test techniques
>    where packet throughput, loss, and delay measurements are conducted.

There is definitely a need for this phase of testing, and I support
including this paragraph in the draft.

However, IPPM needs to be very careful with the wording here, because:

- RFC 2544 has been applied well-beyond its intended scope for
   operational service turn-up testing.  RFC 2544 is lab-use only.

It's a problem when there's a need, but no standard.
RFC 2544-like methods were applied to fill the gap.
And we've got that situation now with many manufacturers
pointing to RFC 2544 and doing something 2544-like, but not
really embracing the scope of 2544 procedures (they only work
reliably in the lab - the live network is different/wilder animal).
There is an effort intended to address the Service Activation
Test methodology in the Ethernet Services space, ITU-T Y.156sam,
a work in progress in Question 17 of Study Group 12.

Also, IETF working groups and the IESG will certainly respect the
scope of other RFCs and BMWG's scope in particular, because
if BMWG methods were let loose on networks, congestion and
other bad scenarios will result (especially without the built-in
safety net that BMWG provided - a testing-only address space that
routers can always drop if they see it).

So I suggest the following wording:
NEW:
    One other important note, it is highly recommended that traditional
    Layer 2/3 type tests are conducted to verify the integrity of the
    network before conducting TCP tests.  Examples include iperf (UDP mode)
    and manual packet layer test techniques
    where packet throughput, loss, and delay measurements are conducted.
    When available, standardized testing similar to RFC 2544 [RFC2544] but
    adapted for use on operational networks may be used (because RFC 2544
    methods are not intended for use outside the lab environment).

-----------------------------------------------------------------------

S2
OLD
2. Goals of this Methodology
suggest NEW
2. Scope and Goals of this Methodology

OLD
>    Before defining the goals of this methodology, it is important to
>    clearly define the areas that are not intended to be measured or
>    analyzed by such a methodology.
suggest NEW
    Before defining the goals of this methodology, it is important to
    clearly define the areas that are out-of-scope for this
    methodology.

suggest additional out-of-scope item:

The methodology is not intended to measure TCP throughput as part of a
SLA, or to compare the TCP performance between service providers or
to compare between implementations of this methodology (test equipment).

------------------------------------------------------------------------

2.2 Metrics for TCP Throughput Tests
...
>    The TCP Transfer time can also be used to provide a normalized ratio
>    of the actual TCP Transfer Time versus ideal Transfer Time.  This
>    ratio is called the TCP Transfer Index and is defined as:
>
>                      Actual TCP Transfer Time
>                     -------------------------
>                      Ideal TCP Transfer Time

There' no definition of "Ideal TCP Transfer Time", but I conclude from
the paragraph below that it is the theoretical raw capacity combined
with a block of information that needs to be transferred.  Since there's
no flow control/ARQ at all, I would be more comfortable calling the
denominator "Ideal Transfer Time", or "Theoretical Transfer Time".

It would also seem appropriate to explain to the user that no TCP
will ever achieve the same performance as Theoretical, and perhaps
this means we are comparing apples and oranges in this metric...

>    An example would be the bulk transfer of 100 MB upon 5 simultaneous
>    TCP connections over a 500 Mbit/s Ethernet service

Is this the Committed Information Rate (CIR) of the Ethernet Service?

>                                                      (each connection
>    uploading 100 MB).  Each connection may achieve different throughputs
>    during a test and the overall throughput rate is not always easy to
>    determine (especially as the number of connections increases).

----------------------------------------------------------------------------

>    2. Baseline

(Average?)

>Round-trip Delay and

Raw capacity (without TCP flow control).

>      . These measurements
>    provide estimates of the ideal TCP window size, which will be used in
>    subsequent test steps.

-----------------------------------------------------------------------------
(later, S3)

>    - TCP implementation used by the test host

and the OS version number,

>  i.e. Linux OS kernel

2.6.......

>    performance server hardware or dedicated hardware based test tools.

(add)
     Thus, other devices cannot realize higher TCP throughput, and user
     expectations SHOULD be set accordingly with user manual or notes on
     the results report.

At 10:13 AM 9/10/2010, Al Morton wrote:
>>(and informational versus a standard), then the Type P metrics 
>>could remain unaddressed.
>>

---

From: Henk Uijterwaal <[email-masked]>
Date: Mon, 20 Sep 2010 15:33:03 +0200

> I didn't remember the 9:00 UTC aspect, just that the 20th was the

You're only a few hours late, so I suggest that we include these
comments as well.

Henk

---

From: Barry Constantine <[email-masked]>
Date: Mon, 20 Sep 2010 06:39:40 -0700

Absolutely fine.

Thanks,

Barry

-----Original Message-----
From: Henk Uijterwaal [mailto:[email-masked]] 
Sent: Monday, September 20, 2010 9:33 AM
To: Al Morton
Cc: [email-masked]; Barry Constantine
Subject: Re: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

Henk

---

From: Barry Constantine <[email-masked]>
Date: Fri, 24 Sep 2010 05:06:19 -0700

Hello,

The new draft-07 version has been posted which incorporated all comments received during the WGLC.

Thanks to Rudi, Henk, and Al for providing excellent comments.

Barry

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henk Uijterwaal
Sent: Monday, September 20, 2010 3:35 AM
To: [email-masked]
Subject: Re: [ippm] End of WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

IPPM group,

Matt & Henk.

---

From: Henk Uijterwaal <[email-masked]>
Date: Fri, 24 Sep 2010 14:41:02 +0200

Barry,

> Thanks to Rudi, Henk, and Al for providing excellent comments.

Thanks, we'll start a second WGLC soon.

Henk