Subject: [CCAMP] 2nd Working group last call: draft-ietf-ccamp-lsp-dppm

From: Lou Berger <[email-masked]>
Date: Tue, 08 Sep 2009 16:54:45 -0400

This email begins a two week working group last call on
draft-ietf-ccamp-lsp-dppm-08.txt

http://tools.ietf.org/html/draft-ietf-ccamp-lsp-dppm-08

Please send your comments to the list or the authors before the last
call closes on September 22, 2009.

As discussed on the list, we are holding an additional last call on
this draft due to the changes made as a result of the previous Last
Call. For convenience, here's the url to see the diff:
http://tools.ietf.org/rfcdiff?difftype=--hwdiff&url1=draft-ietf-ccamp-lsp-dppm-05.txt&url2=draft-ietf-ccamp-lsp-dppm-08.txt

---

From: "Weiqiang Sun" <[email-masked]>
Date: Wed, 9 Sep 2009 10:46:53 +0800

Thanks, Lou. I did not realize we have gone so far since the 1st LC. :)

A summary of the changes -

A number of editorial changes, including
1. As per Tom's comments, we have droped the following two sentenses.
- They can also be used in operational networks for carriers to monitor
  the control plane performance in realtime.

- On the other hand, it can also be used in operational environments for
   carriers to monitor the control plane operation in real-time.  For
   example, a new object can be added to GMPLS TE STD MIB [RFC4802] so
   that the current and past control plane performance can be monitored
   through network management systems.  The extension of TE-MIB to
   support the defined metrics is outside the scope of this document.

2. The change made throughout the document: Unidirectional/Bidirectional -> 
Uni-directional/Bi-directional

3. And a lot others...

Three technical changes, including
1. As per Lou's LC comments on formal conformance language, we have added 
formal language at the points where we think applicable.
These include:
a. Section 2 is recovered.
b. In Section 3: An implementation MAY choose whether to implement metrics 
in the two documents together.
c. In Section 3: However, it is RECOMMENDED that both measurements are 
performed to complement each other.
d. In section 4.6-8.6: Simple upper bounds MAY be used.
e. In section 4.6-8.6: In practice, the upper bound should be chosen 
carefully and the value MUST be reported.
f. In section 4.6: If ingress node sends out the PATH message to set up an 
LSP, but never receive receives the corresponding RESV message, the 
uni-directional LSP setup delay MUST be set to undefined.
g. In section 4.6: If the ingress node sends out the PATH message to set up 
an LSP but receive receives a PathErr message, the uni-directional LSP setup 
delay MUST be set to undefined.
h. In section 6.6: If the ingress node sends out the PATH message to set up 
the LSP, but never receives the corresponding RESV message, single 
bi-directional LSP setup delay MUST be set to undefined.
i. In section 6.6: If the ingress node sends out the PATH message to set up 
the LSP, but receives PathErr message, single bi-directional LSP setup delay 
MUST be set to undefined.
j. In section 7.6: If the ingress node sends out the PATH messages to set up 
the LSPs, but never receives all the corresponding RESV messages, the 
multiple bi-directional LSPs setup delay MUST be set to undefined.
k. In section 7.6: If the ingress node sends out the PATH messages to set up 
the LSPs, but receives one or more responding PathErr messages, the multiple 
bi-directional LSPs setup delay is also deemed to MUST be set to undefined.
l. In section 8.6: ..., but the egress node never receives the corresponding 
RESV PathTear message, LSP graceful release delay MUST be set to undefined.
m. In section 8.6: ..., but never receives the corresponding PathTear 
message, LSP graceful release delay MUST be set to undefined;
n. In section 9.5-11.5: The selection of Th SHOULD take into account that 
the network has sufficient resource to perform subsequent tests.
o. In section 9.5-11.5: The value of Th MAY be constant during one sampling 
process for simplicity considerations.
p. In section 14.1: In computing this, undefined values SHOULD be treated as 
infinitely large.
q. In section 14.1: In addition, the metric minimum SHOULD be set to 
undefined if the sample is empty.
r. In section 14.3: Undefined values MUST NOT be counted in.

2. As per Lou's LC comments, We removed section "13. Discussion for 
unsuccessful setup/release cases" in the original text and added a 
sub-section in section 14: "14.4.  Failure statistics of Metric"
And also, in each metrics definition sections, we added a referencing 
paragraph:
   The undefined value of this metric indicates an event of Single Uni-
   directional LSP Setup Failure, and would be used to report a count or
   an percentage of Single Uni-directional LSP Setup failures.  See
   section Section 14.4 for definitions of LSP setup/release failures.

3. As per Reinhard's comments, we redefined percentile into the current 
look:
14.3.  The percentile of Metric

   The "empirical distribution function" (EDF) of a set of scalar
   measurements is a function F(x) which for any x gives the fractional
   proportion of the total measurements that were <= x.

   Given a percentage X, the X-th percentile of Metric means the
   smallest value of x for which F(x) >= X. Undefined values
   MUST NOT be counted in.

   See RFC2330 for further details.

Thank you,
Weiqiang

--------------------------------------------------
From: "Lou Berger" <[email-masked]>
Sent: Wednesday, September 09, 2009 4:54 AM
To: <[email-masked]>
Subject: [CCAMP] 2nd Working group last call: draft-ietf-ccamp-lsp-dppm

> _______________________________________________
> CCAMP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ccamp

---

From: Lou Berger <[email-masked]>
Date: Fri, 11 Sep 2009 18:08:14 -0400

Authors,

The following are LC comments.  There are a couple of followup points
from the 1st LC and then a slew of non-technical/editorial comments.

- I'm happy to see additional RFC2119 directives to match your
  desired document track.  Don't you think it would be worthwhile
  to add directives on what reporting is needed for the metrics
  actually being measured?  As I read it, only bounds are reported.
  This really doesn't seem to make sense.  Am I missing something?

- I think Section 14.4 addresses much of my earlier comment on
  "undefined" values.  I still have a couple of additional points:

   Don't you think it would be worthwhile to also report the Success
   Count as an integer?  (as percentages can yield rounding errors.)

   As part of the rfc2119 language you're adding related to the
   reporting of metrics defined in section 14, I suggest that you
   highlight the easily missed point of "we only consider the
   finite values" by including "success" as part of the metric
   name.  I'm not sure on exact phrasing as this will depend on
   how you write the language.

   Is there a reason that you included Minimum and Median in
   Section 14, but not maximum?

General comments:

- More Your choice of the word "depict" is a bit odd.  I
  suggest changing all instances of "depict" to "characterize"
  (or "quantify" where appropriate).  You'll probably need to
  slightly the rest of the sentence to align the grammar
  as well.

- RSVP message names should match rfc2205 conventions:
  OLD         NEW
 PATH         Path
 RESV         Resv
 RESV_CONF    ResvConf

- directionality should use 2205/3473 conventions:
  OLD              NEW
 uni-directional  unidirectional
 bi-directional   bidirectional

Specific comments (sorry, these are out of order):

OLD:
Dynamic provisioning ability of these physically diverse...
NEW:
The dynamic provisioning ability of these physically diverse...

OLD: (section 4.2)
   single uni-directional LSP setup delay
NEW:
   Single uni-directional LSP setup delay

OLD:
systems, Add- Drop Multiplexers (ADMs), photonic cross-connects (PXCs),
NEW: (drop space)
systems, Add-Drop...

Remove duplicate word (multiple locations):
"See section Section 14.4 for"

missing "message" after "PathErr" in multiple places:
OLD:
... response message is PathErr, the ...
NEW
... response is a PathErr message, the ...

OLD:
... a count or an percentage of ...
NEW:
... a count or a percentage of ...

OLD:
but receives PathErr message,
NEW
but receives a PathErr message,

OLD:
In practice, the upper bound should be chosen carefully ...
NEW
In practice, the upper bound SHOULD be chosen carefully ...

OLD:
The arrival rate of the Poisson process Lambda_m should be
NEW:
The arrival rate of the Poisson process Lambda_m SHOULD be

- Counted in what?
"Undefined values MUST NOT be counted in."

- Another subjective word choice comment: numerically
How about "integer" or "integer value"?

OLD:
14.3.  The percentile of Metric
NEW:
14.3.  The Percentile of Metric

OLD:
implementation,configuration
NEW:
implementation, configuration

OLD:
using three way signalling
NEW: (to match rest of document)
using three way signaling

That's it.

Lou

---

From: "Weiqiang Sun" <[email-masked]>
Date: Tue, 15 Sep 2009 13:23:49 +0800

Hi Lou,

Thank you for the comments. Please see below.

Best regards,
Weiqiang

--
Weiqiang Sun
Shanghai Jiao Tong University
http://front.sjtu.edu.cn/~sunwq/

--------------------------------------------------
From: "Lou Berger" <[email-masked]>
Sent: Saturday, September 12, 2009 6:08 AM
To: <[email-masked]>
Cc: <[email-masked]>
Subject: Re: [CCAMP] 2nd Working group last call: draft-ietf-ccamp-lsp-dppm

>  This really doesn't seem to make sense.  Am I missing something?
Maybe we can change the text into:
``In practice, the upper bound should be chosen carefully and the value MUST
be reported along with the metric itself."
Do you think it is helpful?

>   Count as an integer?  (as percentages can yield rounding errors.)
Please be noted that we already have the Failure Count(14.4.1). Success
Count can be easily deduced from: ``the number of all values in a sample -
Failure Count". Reporting the Success Count seems a bit redundant,
when the Failure count is far more important and necessary.

>   how you write the language.
Since the undefined values are properly treated  in section 14 using the
rfc2119 language, maybe we can just drop
the sentence "In the following discussion, we only consider the finite
values." How do you think?

>   Section 14, but not maximum?
Minimum and Median values are generally more stable than maximum in a large
measurement and can provide valuable sights to the DUT/SUT.
We do have a bunch of other statistics that are not yet included in the
document. What we can offer is a suggested sub-set, rather than a whole set.
The current selection is based on rfc2679 and rfc2681.

> General comments:
Many thanks for the comments below. We will make the changes accordingly.

---

From: Lou Berger <[email-masked]>
Date: Tue, 15 Sep 2009 16:21:43 -0400

Weiqiang,
	Please see below.

> Thank you for the comments. Please see below.

How about (section 4-8):
   In practice, the upper bound should be chosen carefully.

o  All collected metrics together with the selected upper bound MUST
   be reported.

Section 9, 10, 11,12 ,13 :

o  All collected metrics together with the selected parameters MUST
   be reported.

> when the Failure count is far more important and necessary.

I don't see where the total number of requests in a sample is described.
 If it is, then I agree that Success Count isn't needed.

> values." How do you think?

If total or success count are added, this is fine.

> measurement and can provide valuable sights to the DUT/SUT.

Sure, but maximums provide an important view into what can be expected
from the system.  Without the max value there is no way to understand
worst case behavior. I think it's worth including.

> The current selection is based on rfc2679 and rfc2681.

well, seems like you should add a caveate along the lines of 2679, i.e.,
"These statistics are offered mostly to be illustrative of what could be
done."

> Many thanks for the comments below. We will make the changes accordingly.

Thanks,
Lou

---

From: "Weiqiang Sun" <[email-masked]>
Date: Thu, 17 Sep 2009 15:38:36 +0800

Lou,

Please see below.

--------------------------------------------------
From: "Lou Berger" <[email-masked]>
Sent: Wednesday, September 16, 2009 4:21 AM
To: "Weiqiang Sun" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Subject: Re: [CCAMP] 2nd Working group last call: draft-ietf-ccamp-lsp-dppm

> o  All collected metrics together with the selected parameters MUST
The suggested text sounds good to me.

> If it is, then I agree that Success Count isn't needed.
Once a sample is obtained and reported, the total number is then fixed. Do 
you think it is necessary to give an explicit description of the total 
number of values in a sample? It is so obvious to me...

> If total or success count are added, this is fine.
Please see above.

> worst case behavior. I think it's worth including.
Yes it may be helpful, but it might also be dangerous to say the maximum 
reflects the worst case behavior, when all reported values are bounded (by a 
preset upper bound).
In this sense the ``maximum" by its name may be misleading...

> "These statistics are offered mostly to be illustrative of what could be
Sounds good to me.

Thanks always,
-weiqiang

---

From: Lou Berger <[email-masked]>
Date: Thu, 17 Sep 2009 07:05:39 -0400

Weiqiang,
	It seems we're left with two open topics:

1) I propose addressing the total count topic using text along the lines of:

(section 4-8):
o  All collected metrics together with the total number of values
   (both real and undefined) and the selected upper bound MUST be
   reported.

o  All collected metrics together with the total number of values
   (both real and undefined) and the selected parameters MUST be
   reported.

2)

> reflects the worst case behavior, when all reported values are bounded
> (by a preset upper bound).

Well this depends on the selected upper bound.  Also, yes statistics
*can* be misread, but if the data isn't present there's no way for the
results to be properly interpreted.

Lou

---

From: "Weiqiang Sun" <[email-masked]>
Date: Thu, 17 Sep 2009 22:14:57 +0800

Hi Lou,

I agree with you on 1.
As for 2, what I can come up with is the text below. I can not persuade 
myself to put it in the draft :(. Maybe you have better ways of putting 
it...

14.x.  The Maximum of Metric

   The maximum of metric is the maximum of all the dT values in the
   sample.  In computing this, undefined values MUST NOT be counted in.
   Note that this means that the maximum could thus be undefined if all
   the dT values are undefined.  In addition, the metric maximum SHOULD
   be set to undefined if the sample is empty.

   It also must be noted that exceptionally a measurement may result in a
   value that is greater than the upper bound and is thus deemed undefined.
   Because of this, the maximum of metric must be used with caution.

Thanks,
Weiqiang

--------------------------------------------------
From: "Lou Berger" <[email-masked]>
Sent: Thursday, September 17, 2009 7:05 PM
To: "Weiqiang Sun" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Subject: Re: [CCAMP] 2nd Working group last call: draft-ietf-ccamp-lsp-dppm

---

From: Lou Berger <[email-masked]>
Date: Fri, 18 Sep 2009 15:51:50 -0400

Weiqiang,
	How about something along the lines of :

  The maximum of metric is the maximum of all the dT values in the
  sample.  In computing this, undefined values MUST NOT be included.
  Note that this means that measurements that exceed the upper bound
  are not reported in this statistic.  This is an important
  consideration when evaluating the maximum when the number of
  undefined measurements is non-zero.

Lou

---

From: "Weiqiang Sun" <[email-masked]>
Date: Mon, 21 Sep 2009 10:18:28 +0800

Lou,

This sounds great to me.
Many thanks,

Weiqiang

Sent: Saturday, September 19, 2009 3:51 AM

---

From: Lou Berger <[email-masked]>
Date: Wed, 23 Sep 2009 12:00:43 -0400

This last call is complete.  Authors please submit any revisions you may
have, and summarize changes to the list.

Lou