Subject: RE: AD review of: draft-ietf-tewg-measure-05.txt

From: "Ash, Gerald R (Jerry), ALABS" <[email-masked]>
Date: Fri, 25 Apr 2003 16:43:23 -0500

Bert,

Let's not throw the baby out with the bath-water.  The I-D serves a critical need and should go forward.

Regarding your comment:
'I would expect a CRISP set of "requirements for additional measurements, configurable/negotiable parameters/controls" ... but not the extensive exploration and text that I now see'.
The requirements *are* there for additional measurements, configurable/negotiable parameters/controls, but they need to be made more prominent and crisp (as you say).

I'd suggest to focus the main body of the I-D on these requirements/recommendations, and move the descriptive material to Annex(es).  The text has evolved over a long time with many comments and responses, as such it has gathered some moss along the way...  But the critical essence is there, and the requirements *need* to move forward to provide essential TE measurements.

Regarding your comment:
'if we were not yet doing any of it... then I wonder if this would be helpful at all.'
There are many important requirements/recommendations in the I-D.  As one example, the ability to collect traffic data to generate a traffic matrix (e.g., 'node-pair data', see Table in Section 10.1) is woefully lacking.  To get a flavor of the problem/need, see these papers on the topic (links to papers at http://www.research.att.com/~jrex/papers.html#tmeas):
A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, G. True, "Deriving traffic demands for operational IP networks: Methodology and experience," IEEE/ACM Transactions on Networking, June 2001, pp. 265-279. (An earlier version appeared in Proc. ACM SIGCOMM, August/September 2000.)

This work shows the difficulty in deriving a traffic matrix with today's measurements.  A traffic matrix is basic to any network engineering or planning task.

Other critical traffic/performance measurement requirements identified in the I-D are also lacking and need to go forward.

A few more comments below.

Jerry
 
> - a lot of text... but not very focused and to the point (or at least
>   I have trouble seeing the main points)
> - not a "framework", rather an exploration of TE measurement related topics
>   (more like  a summary-introduction)

As above, the I-D serves a critical need.  The text can be focused on the requirements/recommendations, as suggested, and the descriptive stuff moved to an Annex.

> - I am not an operator, but I think if I were one, then
>   - if we were already doing TE measurement stuff (most likely) then 
>     reading it seems a waste of time
>   - if we were not yet doing any of it... then I wonder if this would
>     be helpful at all.

Operators have to get by with what is available, what else can we do?  This does not mean operators are already doing 'TE measurement stuff' in an adequate way, and that nothing further is required.  E.g., see comments above on derivation of a traffic matrix.

> - W.r.t. the TEM WG work item, it says:
>      The tewg interacts with the common control and measurement plane
>      working group to abstract and define those parameters, measurements,
>      and controls that traffic engineering needs in order to engineer
>      the network. 
>   So I would expect a CRISP set of "requirements for additional measurements,
>   configurable/negotiable parameters/controls" ... but not the extensive
>   exploration and text that I now see. Why do people (or the WG) think
>   that this document meets the WG deliverable for TEM ??

The requirements are there, and are critically needed.  They need to be brought out in a clear and crisp way, as suggested above.

> - W.r.t. review:
>   - 4 people from ATT support it. Waisum is one of them and is main editor.
>     Others (Nick Duffield, Bob Cole) have some of their material listed in
>     the doc.

So AT&T is stuffing the ballot box?  I'm mentioned in acknowledgements, others in references, are we disqualified to comment?  A certain set of people (*individuals*) who are the ones most interested/knowledgeable are the ones also most likely to comment.
 
>   - 3 EDU users commented, 2 said they found it a good doc
>     3rd one asked a few questions
>   - Raymond Zhang is positive. He is from info.net ??? is that an operator?

yes.

>   - Richard Tibbs gave a thumbs up, he is from oakcitusolutions.com.
>     What role/function does he play/have? Operator, code/tool-developer?

co-author.

>   - One hotmail user (Spyrokontigiogis) gave a thumbs up. Not that
>     he/she added any comment. Do we know him/her?
>     What role/function does he/she play/have? Operator, code/tool-developer?

Level-3 employee?

>   - Dimitri Papadimitrou (Alcatel) asked a question/suggested some text.
>     I did not see if he likes the doc or not

did not object.

>   - Blain Christian (uunet, so maybe a real operator?) withdrew as co-editor
>     That does not sound good (in my view)
> - So where are the real operators that support this?

So *individuals* from AT&T, Infonet, France Telecom, and Level-3 (?) supported this, IETF does not have companies/operators positions.  Also, are you saying that the above operators are not 'real' operators versus other operators who you think are real operators?

> My thought is that I can do two things:
> 
> - send back to WG and say that this is "not good enough" and 
> that I do not 
>   feel comfortable to present it to IESG for approval.
>   It does NOT meet (in my eyes) the WG deliverable for TEM.
> - send it to IESG with my recommendation to NOT approve for 
> same reasons.
> 
> So let me try the first option first and ask the WG what they 
> have to say to my review.

I find it rather disappointing that you waited this long to come down on this I-D so hard.  If you had this level of concern, I think you should have tipped your hand long ago, but perhaps this is first time you have read the draft.  We have been admonished many times to read the draft and comment on this list, and many folks have abided by your wish.  

If we discard this I-D, the TEWG charter item also will go unfulfilled.  That's bad. I think the I-D serves an important need, and can address your comments, as above.

---

From: "Geib, Ruediger" <[email-masked]>
Date: Mon, 28 Apr 2003 15:50:10 +0200

Bert,

| > 'I would expect a CRISP set of "requirements for additional 
| > measurements, configurable/negotiable parameters/controls" 
| > ... but not the extensive exploration and text that I now see'.

I understood the document to be a Framework Document. Crisp 
requirements may be part of a requirements doc or the like. 

The quality of the document is good as a Framework Document.
I'd agree that an architecture or requirements doc 
getting more precise might be useful.  
Different network types and design philosophies result 
in different measurement requirements. I'm not sure whether an 
acceptable "General TE measurement requirements" doc is feasible.

| I do not consider them as speaking for their company, but indeed as
| individuals. I'd like to see comments/review/support from all 
| types of operators, that is carriers, telecom operators, cors ISPs,
| Access ISPs, small ISPs, Enterpise Networks... etc.

I wasn't aware that there's something like an IESG qualification 
profile if you want to vote on last call. I'd suggest that these 
profiles are published with the last call, so that people whose opinion 
isn't asked may save the time they've wasted on reading the document.

Regards, Rüdiger

---

From: "Wijnen, Bert (Bert)" <[email-masked]>
Date: Mon, 28 Apr 2003 17:47:41 +0200

Inline

> -----Original Message-----
> From: Geib, Ruediger [mailto:[email-masked]]
> Sent: maandag 28 april 2003 15:50
> To: [email-masked]
> Cc: [email-masked]
> Subject: RE: AD review of: draft-ietf-tewg-measure-05.txt 
> 
> 
> Bert,
> 
In my view, a "framework" or "architecture" document is totally 
different from what the current ID describes. 

> The quality of the document is good as a Framework Document.

But the WG charter does not ask for a "framework doc" or at least not
the way I understand it.

> acceptable "General TE measurement requirements" doc is feasible.
Well, Jeryr claims all the requirements are in the doc. Just too bad
that it is very difficult to find them. That is why I asked for such
a crsip list of requirements.

> isn't asked may save the time they've wasted on reading the document.
The IESG does NOT have a "qualification profile". 
And EVERYONE's input and review is usefull and is welcome.
What we DO want is that if we write something for OPERATIONAL networks
that we DO get the input from the various types of OPERATORS that we
know about.

Why does everybody look at my comments as an attack on their personal input?
It is rather an attack (if you can speak of that) to those type of OPERATORS
that have NOT yet given review input (positive or negative). I am (in my
responsibility as AD) trying to get sufficient and wide enough review of
the document before I feel comfortable passing it to the IESG.

Hope this explains, Bert

---

From: "Geib, Ruediger" <[email-masked]>
Date: Tue, 29 Apr 2003 09:07:51 +0200

Inline

| > The quality of the document is good as a Framework Document.
| 
| But the WG charter does not ask for a "framework doc" or at least not
| the way I understand it.

It says "Document additional measurements needed for TE (TEM)". That's 
the most precise reference I've found regarding measurements. 
 
| > I'd agree that an architecture or requirements doc 
| > getting more precise might be useful.  
| > Different network types and design philosophies result 
| > in different measurement requirements. I'm not sure whether an 
| > acceptable "General TE measurement requirements" doc is feasible.
| > 
| Well, Jeryr claims all the requirements are in the doc. Just too bad
| that it is very difficult to find them. That is why I asked for such
| a crsip list of requirements.

Jerry is absolutely right. The document lists everything which could 
be of use to support TE measurements. That's a kind of toolbox. 
It contains everything which is required and some hints how you'd
apply it. It doesn't contain a building plan. The first steps to this 
direction were an architecture document and some specific 
requirements.

| The IESG does NOT have a "qualification profile". 
| And EVERYONE's input and review is usefull and is welcome.
| What we DO want is that if we write something for OPERATIONAL networks
| that we DO get the input from the various types of OPERATORS that we
| know about.

Like Infonet.

Regards, Rüdiger