Subject: RE: AD review of: draft-ietf-tewg-measure-05.txt

From: "Wijnen, Bert (Bert)" <[email-masked]>
Date: Sat, 26 Apr 2003 18:23:52 +0200

WG,

I'd like to hear/read more comments/feedback on my review.

Inline responses to the comments from Jerry

Thanks,
Bert 

> -----Original Message-----
> From: Ash, Gerald R (Jerry), ALABS [mailto:[email-masked]]
> Sent: vrijdag 25 april 2003 23:43
> To: Wijnen, Bert (Bert); Tewg (E-mail)
> Cc: Ash, Gerald R (Jerry), ALABS
> Subject: RE: AD review of: draft-ietf-tewg-measure-05.txt 
> 
> 
> Bert,
> 
> Let's not throw the baby out with the bath-water.  The I-D 
> serves a critical need and should go forward.
> 
Can you formulate the "critical need" that is being served?

> Regarding your comment:
> 'I would expect a CRISP set of "requirements for additional 
> measurements, configurable/negotiable parameters/controls" 
> ... but not the extensive exploration and text that I now see'.
> The requirements *are* there for additional measurements, 
> configurable/negotiable parameters/controls, but they need to 
> be made more prominent and crisp (as you say).
> 
Well cedrtainly I could not see them... they are too well hidden 
I guess :-(. So making them CRISP is certainly a good thing.

> I'd suggest to focus the main body of the I-D on these 
> requirements/recommendations, and move the descriptive 
> material to Annex(es).  The text has evolved over a long time 
> with many comments and responses, as such it has gathered 
> some moss along the way...  But the critical essence is 
> there, and the requirements *need* to move forward to provide 
> essential TE measurements.
> 
So.... it seems we agree that this title:
    A Framework for Internet Traffic Engineering Measurement
is not good and should be something aka
    Requirements for essential TE measurements, parameters and controls
and then of course the main body of the document should focus on
that and be clear as to waht is a real requirement and why.

> Regarding your comment:
> 'if we were not yet doing any of it... then I wonder if this 
> would be helpful at all.'
> There are many important requirements/recommendations in the 
> I-D.  As one example, the ability to collect traffic data to 
> generate a traffic matrix (e.g., 'node-pair data', see Table 
> in Section 10.1) is woefully lacking.  To get a flavor of the 
> problem/need, see these papers on the topic (links to papers 
> at http://www.research.att.com/~jrex/papers.html#tmeas):
> A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, 
> G. True, "Deriving traffic demands for operational IP 
> networks: Methodology and experience," IEEE/ACM Transactions 
> on Networking, June 2001, pp. 265-279. (An earlier version 
> appeared in Proc. ACM SIGCOMM, August/September 2000.)
> 
I'd like to see in the document something aka:

   REQUIREMENT: ability to collect traffic data to generate trafix matrices
   DESCRIPTION: describe the details as to what traffic data is needed
   JUSTIFICATION: describe what the problem is that needs to be solved
   PRIORITY: x

Don't need to dwell too much on that... but have a concise and crisp
writeup. Maybe RFC3216 can serve as a good example.

> This work shows the difficulty in deriving a traffic matrix 
> with today's measurements.  A traffic matrix is basic to any 
> network engineering or planning task.
> 
> Other critical traffic/performance measurement requirements 
> identified in the I-D are also lacking and need to go forward.
> 
> A few more comments below.
> 
> Jerry
>  
> > - a lot of text... but not very focused and to the point 
> (or at least I have trouble seeing the main points)
> > - not a "framework", rather an exploration of TE measurement 
> >   related topics (more like  a summary-introduction)
> 
> As above, the I-D serves a critical need.  The text can be 
> focused on the requirements/recommendations, as suggested, 
> and the descriptive stuff moved to an Annex.
> 
One could wonder how much of the descriptive stuff is really
helpful or usefull. But if it is in an ANEX, then maybe that is
OK. I would say however, that if we have the requirements
crisply formulated, with description, justification and priority,
then one would hope that a lot of additional text is not needed
(not even in an anex). Think about the audience. It would be
protocol developers and/or vendors who have to understand the
requirement and be able to make modifications to their protocol
and/or products to meet the requirement.

> > - I am not an operator, but I think if I were one, then
> >   - if we were already doing TE measurement stuff (most 
> >     likely) then reading it seems a waste of time
> >   - if we were not yet doing any of it... then I wonder if 
> >     this would be helpful at all.
> 
> Operators have to get by with what is available, what else 
> can we do?  This does not mean operators are already doing 
> 'TE measurement stuff' in an adequate way, and that nothing 
> further is required.  E.g., see comments above on derivation 
> of a traffic matrix.
> 
I would hope operators can also understand and support the 
requirements. If not, then I really doubt that they are valid
requirements. And they must understand them, so that together
the operators can say to protocol developing WGs and to vendors
that these are the set of requirements to be met.

> > - W.r.t. the TEM WG work item, it says:
> >      The tewg interacts with the common control and measurement plane
> >      working group to abstract and define those parameters, measurements,
> >      and controls that traffic engineering needs in order to engineer
> >      the network. 
> >   So I would expect a CRISP set of "requirements for additional measurements,
> >   configurable/negotiable parameters/controls" ... but not the extensive
> >   exploration and text that I now see. Why do people (or the WG) think
> >   that this document meets the WG deliverable for TEM ??
> 
> The requirements are there, and are critically needed.  They 
> need to be brought out in a clear and crisp way, as suggested above.
> 
Yea.. you keep repeating that.

> > - W.r.t. review:
> >   - 4 people from ATT support it. Waisum is one of them and is main editor.
> >     Others (Nick Duffield, Bob Cole) have some of their material listed in
> >     the doc.
> 
> So AT&T is stuffing the ballot box?

That was not what I am saying. My point was more: it is clear that those
people who have contributed support the document. Would be strange if they
did not, would it not? But I can see now that maybe I should have made
my statement a bit different.

> I'm mentioned in acknowledgements, others in references, are we
> disqualified to comment?  A certain set of people (*individuals*)
> who are  the ones most interested/knowledgeable are the ones also
> most likely to comment.
>  
> >   - 3 EDU users commented, 2 said they found it a good doc
> >     3rd one asked a few questions
> >   - Raymond Zhang is positive. He is from info.net ???
> >     is that an operator?
> 
> yes.
I thought so but was not sure. Maybe Raymond can answer himself, no?
Raymond, can you fill me in what sort of network you operate?
Feel free to do so privately if you prefer that.

> 
> >   - Richard Tibbs gave a thumbs up, he is from oakcitusolutions.com.
> >     What role/function does he play/have? Operator, code/tool-developer?
> 
> co-author.
> 
Ah... ok, so obvious that he would support it. Should have been in
my first list of people above. 
You did not answer (but better if Richard does himself) if he is
and operator or what role plays in his daily life. The reason I ask
this is because I want to udnerstand their background and the eyes
and context that they probably have used to review.

> >   - One hotmail user (Spyrokontigiogis) gave a thumbs up. Not that
> >     he/she added any comment. Do we know him/her?
> >     What role/function does he/she play/have? Operator, code/tool-developer?
> 
> Level-3 employee?
> 
Can you be more specific? Is he/she in the accounting dept? I guess not.
But again, I want to understand the context within which the review was done.

> >   - Dimitri Papadimitrou (Alcatel) asked a question/suggested some text.
> >     I did not see if he likes the doc or not
> 
> did not object.
> 
> >   - Blain Christian (uunet, so maybe a real operator?) 
> >     withdrew as co-editor
> >     That does not sound good (in my view)
> > - So where are the real operators that support this?
> 
> So *individuals* from AT&T, Infonet, France Telecom, and 
> Level-3 (?) supported this, IETF does not have 
> companies/operators positions.  Also, are you saying that the 
> above operators are not 'real' operators versus other 
> operators who you think are real operators?
> 
Sorry for the bad wording. When I use company names, I try to
see how many differnt viewpoints from people active in different 
types of networks have looked at it. From their operational experience.
I do not consider them as speaking for their company, but indeed as
individuals. I'd like to see comments/review/support from all 
types of operators, that is carriers, telecom operators, cors ISPs,
Access ISPs, small ISPs, Enterpise Networks... etc.

> > My thought is that I can do two things:
> > 
> > - send back to WG and say that this is "not good enough" and 
> >   that I do not 
> >   feel comfortable to present it to IESG for approval.
> >   It does NOT meet (in my eyes) the WG deliverable for TEM.
> > - send it to IESG with my recommendation to NOT approve for 
> >   same reasons.
> > 
> > So let me try the first option first and ask the WG what they 
> > have to say to my review.
> 
> I find it rather disappointing that you waited this long to 
> come down on this I-D so hard.  If you had this level of 
> concern, I think you should have tipped your hand long ago, 
> but perhaps this is first time you have read the draft.

That is a valid complaint. We are aware that a lot of the AD/IESG level
review takes place too late in the process. Not easy to fix, we (ADs)
all have trouble to keep up with what we have on IESG agenda, let alone
to dive into many WG docuemnts early. We try to improve... but in this
case I missed it.

Now I do understand that the WG chairs have been pushing back on this
in the beginning quite a bit too. I believe that the doc has improved 
quite a bit since the earlier versions... but when I review it in
the context of the WG deliverable, then I am clearly still very
dissappointed. If you say that all the "requirements" are hidden in
that doc, then pls try to make them clear. It seems that needs major
rework.

> We have been admonished many times to read the draft and comment 
> on this list, and many folks have abided by your wish.  
> 
I guess it depends on what you count as "many". When I was a kid,
we joked about people counting:  1,2,3,many. Mmm.. maybe that statement
is not appropriate. 

> If we discard this I-D, the TEWG charter item also will go 
> unfulfilled.  That's bad. I think the I-D serves an important 
> need, and can address your comments, as above.  
> 
But I'd rather have us confess that we were unable to deliver on a
WG item than us delivering something that is not in good shape
(or bad quality, which I think is the case).

Hope this explains my posting a bit better.
Bert

---

From: "JACQUENET Christian FTLD/IAP" <[email-masked]>
Date: Sun, 27 Apr 2003 12:41:11 +0200

Dear all,

Additional comments below.

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]]De la
part de Wijnen, Bert (Bert)
Envoyé : samedi 26 avril 2003 18:24
À : Ash, Gerald R (Jerry), ALABS; Tewg (E-mail)
Objet : RE: AD review of: draft-ietf-tewg-measure-05.txt

WG,

Thanks,
Bert

CJ: the specification and the enforcement of a traffic engineering policy
(or a set of) rely upon the activation of complex features, hence yielding
the critical need to qualify how well is such a policy enforced and how
efficient it is, from both a network resource optimization standpoint (hence
affecting the network planning policy) and from a QoS standpoint (like the
provisioning of (hard) guarantees (loss, inter-packet delay variation,
one-way transit delay, etc.) as far as the level of quality associated to a
given service is concerned). Qualification of the efficiency of such
policies obviously rely upon measurement techniques and parameters, and
IMHO, this draft depicts a pretty good set of requirements that not only
reflect the aforementioned (and defined) critical need, but which also
provides an accurate reference/cornerstone for the definition of TE
measurement framework and tools. That's the reason why I fully support this
draft.

CJ: I'd like to hear other comments about the readability of the contents of
this draft. The fact you find these requirements hard too find does not
necessarily mean that all the reading community (including IESG members)
shares this point of view.

CJ: "Requirements for TE measurement" should be sufficient.

CJ: it's basically a matter of presenting things (and I don't like the
notion of priority you seem to intriduce in your suggestion, because (1) the
so-called importance of a requirement may very well vary from one operator
to another (because of the range of service offerings actually supported,
because of the network topology, etc.), and (2) priority generally implies
classification and reference, which to me, is pointless as far as a
requirements document is concerned - see RFC 3139, for example).

CJ: your suggestion of putting examples in an appendix is fine, but I'm
still not convinced of the efficiency of your proposal (description,
justification, etc. - see above comment), simply because you personally
found this document hard to read.

CJ: I fully understand and support the requirements described in this draft.
And I'm pretty sure I'm not the only operator's representative who fully
understands and supports the contents of this draft.

CJ: being ironic and cynical doesn't help the discussion that much.

CJ: I'm not sure I understand what you're trying to get to here. Are we
supposed to *justify* our position and/or our networking conditions to have
the right to write and submit a document to the IETF? I'd like to hear  the
opinion of IESG representatives on this subject.

Can you be more specific? Is he/she in the accounting dept? I guess not.
But again, I want to understand the context within which the review was
done.

CJ: as far as the document's review is concerned, it has followed the basic
rules - submitted to the tewg WG mailing list, waited for comments, issued
the updated version, gone into a WG last call procedure and then submitted
to the IESG for review. Why are you *demanding* the background, the position
(maybe the skills, resumes while you're in this questioning process?), etc.
of the people who took some of their time to writte and/or review this
document and possibly make some comments/suggestions? What's so specific
with this document that justifies this kind of inquisition?

CJ: why don't you clearly state what you dislike about this document instead
of demanding an echo from all the Internet community (you should also
include end customers, since TE may have an impact on the level of quality
of the service they've subscribed to, hence imposing service providers that
they fulfill their commitment by providing the appropriate measurements on a
regular basis)?

CJ: this is the kind of argument we could also put forward when considering
other initiatives, such as the recent creation of the netconf WG for
example, based on an individual contribution that didn't get too much echo.
In this case, surprisingly enough, I can see that a silent majority was good
enough to launch yet another WG (hence additional contributions).

But when, say, 5 to 10 people have expressed their opinions on a
*requirements* document that is part of the tewg WG charter, and that this
document has been updated accordingly so that yet another silent majority
didn't make any specific objection so far, then the document is judged of
bad quality by a single person.

This is simply *unfair*, especially when this person is ready to consider
specifications of solutions (I'm referring to the inter-domain TE stuff -
see the IETF56 tewg WG minutes) *before* the accompanying/reference
requirements document has been accepted by the tewg WG as a WG document, and
discussed by the WG members.

Cheers,

Christian.

---

From: [email-masked]
Date: Sun, 27 Apr 2003 14:08:11 +0200

bert,

the proposed update meets the comments i raised,

now my two cents, as it stands now i see this i-d more
as an "analysis of the requirements for tem models and
techniques" since imho the authors took an analytical
approach and in order to probably to come with more
"crispy" input for the tewg (since the objective was
here to "Document additional measurements needed for
TE (TEM)") expansion starting from the conclusion
might be a way to proceed (take this as a suggestion
to help things moving forward) if this is what the
tewg consents to do.

see also in-line...

> Can you formulate the "critical need" that is being served?

well, i think here you ask a critical question but is your
question not just expanding on the reasons why this has
been included within the charter of the wg ? i think that
section 4.4 of RFC 3346 expand on this

> I guess :-(. So making them CRISP is certainly a good thing. 

this is what i'd suggest also (in case of re-spin), expand
section were such statement can be summarized in order to
reflect the achievements here for the tewg ...

> that and be clear as to waht is a real requirement and why.

well your question make sense in terms of "ease of read and
further use" and i think the reason why "content has been
spread" is also due to what kind of requirement (functional
vs protocol) where drafted wrt to the wg expectations - does
this just not translate the complexity of the subject ? -

> writeup. Maybe RFC3216 can serve as a good example.

again, this is probably the issue but i'd like to mention that
the priority and justification may depend on a per user basis

also i took a look at this RFC and compare for instance the abstract

"This document describes the objectives for a new data definition
  language, suitable for the modeling of network management constructs,
  that can be directly mapped into SNMP and COPS-PR protocol
  operations."

well let's try and see if this is achievable here..

"For interoperable compatibility, uniform definitions across vendors
  and operators must be ensured, e.g., in the distinction between
   offered load and achieved throughput

=> so covers terminology

"To aid network dimensioning, mechanisms to collect node-pair-based 
traffic data should be developed to facilitate the derivation of 
per-service-class traffic matrix statistics.

=> associated requirements to acheive node-pair-based traffic data
    and expectation in terms of derivation mechanisms in providing
    stat's

"For service assurance, there is a need for the use of higher-order 
statistics. To preserve representative traffic detail at manageable 
sample volumes, there is a need for packet-sampled measurements.

=> requirements in terms packet-sampling measurements

"To manage large volume of measured data, use of bulk transfer and 
filtering/aggregation mechanisms may be appropriate."

if a re-spin is needed this might be a way to proceed.

> and/or products to meet the requirement.

bert, here that's a critical point, i have mentioned it here
above as well => where the expected outcomes in terms of functions/
protocol aspects or both ?

now annex might be useful if using the proposed scheme as
per RFC 3216, expansion on the "why" (as you have mentioned
here above) might difficult to infer, for instance - let's
first see what it gives before expanding on whether it belongs
to an annex or should be dropped from the i-d - i am afraid
that if we use one short paragraph or sentence for the why
people will show up and ask for more content afterwards -

>>further is required.  E.g., see comments above on derivation 

jerry, as a suggestion this might be a way to introduce the
topic, "why operators think that current TEM when available
is not performed adequately and what's expected to achieve
what you need"

> Yea.. you keep repeating that.

bert, i think the content is there ... imho we speak
about the form it took rather than the content here
(it might always be improved but this is the case for
any standard track rfc) in all the comments you have
listed i do not see any major "content" issue

section 4.4 of RFC 3346 shows that this aspect must
be covered by the wg

last, i don't want to expand too much on the below, i
think it becomes a bit touchy

thanks,
- dimitri.

>>did not object

-- 
Papadimitriou Dimitri
E-mail : [email-masked]
Private: http://www.rc.bel.alcatel.be/~papadimd/index.html
E-mail : [email-masked]
Public : http://psg.com/~dpapadimitriou/
Address: Fr. Wellesplein 1, B-2018 Antwerpen, Belgium
Phone  : +32 3 240-8491