Subject: Re: [mpls] ISIS TE Metric Extension Discussion (IETF-85 Followup)

From: Curtis Villamizar <[email-masked]>
Date: Tue, 19 Feb 2013 19:56:08 -0500

Alia,

How many different sets of ISIS and OSPF standards for advertising
delay and jitter should we be aiming for?

I think that the answer should be one.  Not two.  Not three or more.

Therefore it would be beneficial to look at the set of requirements
and see if using one document as the base - fine with me if
draft-atlas-mpls-te-express-path is used as the base - and then see if
adding just a little to that covers CL requirements, and maybe adding
just a little more covers anything missed in
draft-fuxh-mpls-delay-loss-* that is deemed important.

The delay related requirements in CL are spelled out in the CL
framework.  They are (from 7.1.  Brief Review of Requirements):

   delay and delay variation

       FR#7 (expose lower layer measured delay),
       FR#8 (precision of latency reporting),
       FR#9 (limit latency on per LSP basis),
       FR#15 (minimum delay path),
       FR#16 (bounded delay path),
       FR#17 (bounded jitter path)

FR#7 essentially asks for interworking with delay measured at a lower
layer.  Obviously this would *not* include queuing delay.  Maybe no
impact on draft-atlas-mpls-te-express-path since it does not specify
where to get the information.

FR#8 (in CL requirements) is:

   FR#8    The precision of latency reporting SHOULD be configurable.
	   A reasonable default SHOULD be provided.  Implementations
	   SHOULD support precision of at least 10% of the one way
	   latencies for latency of 1 ms or more.

It seems that the remaining four are covered.  So meeting CL
requirements may be a NOOP except perhaps FR#8.

Some critical questions which I don't see answered in any draft, are
1) whether queuing delay is measured, 2) if so, can measurements at
different PSC (direct mapping of TC bits to PHB for E-LSP) be handled,
3) if so, how are different PSC handled.  It would be nice if the
precision of the delay measurement was reported, for example reporting
a range for the link (ie: 10-12 msec, rounded to 1 msec).  If delay
was 9.8-11.5 for an interval and 10.2-12.2 the next, it need not be
readvertised but its nice to know through the reported precision that
it could be in the range of 9.5-12.499 and still count as 10-12.

Then there is the greater issue of measurement interval and
confidence.  Is the reported value the lowest reading over the
interval, highest, or some statistical norm?  Is it a mean value over
the interval? 98th percentile?  Is it necessary to communicate the
measurement interval?  For example, would a NPO be satisfied for delay
if the measurement was made over the previous hour and any changes
would be reflected in the next measurement an hour later?  My
suggestion was a range and precision.

I don't think you can just punt on oscillations and stability by
saying for further study.

As for the draft-fuxh- set, I think the only requirement not met is
the delay difference between working and protect paths.  For that you
need something along the lines of a min delay and max delay.  Where
the working and protect are expected to each fall within a range, such
that this requirement is met.  Note that the mpls-tp-use-cases draft
has the same exaple.  The authors of the draft-fuxh- set can correct
me if I missed anything.

If you don't mind, I think we should take this thread to MPLS.  I
copied MPLS and bcc'd ISIS.  This way ISIS will get this message and
know the conversation moved, but the conversation continues on MPLS.
On the MPLS list I'll make more detailed comments on a new thread.

Curtis

In message <[email-masked]>
Alia Atlas writes:

Curtis,

I believe that we are emphatically saying that:

  draft-fuxh-mpls-delay-loss-te-problem-statement and/or
  draft-fuxh-mpls-delay-loss-te-framework

are UNCONNECTED to the drafts that we are discussing, which are:

   draft-atlas-mpls-te-express-path
   draft-ietf-ospf-te-metric-extensions
   draft-previdi-isis-te-metric-extensions

They are also not trying to solve the composite-link requirements.

I know that all of these talk about delay and jitter concerns, but the 3 drafts we're discussing are trying to narrow down what to send to a reasonable set that doesn't provoke stability concerns and can be used to solve the types of issues that Spencer described as in the mpls draft.

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 1:57 AM
To: Alia Atlas
Cc: Lou Berger; [email-masked]; John E Drake; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

In message <[email-masked]>
Alia Atlas writes:
 
> Should we add a reference to draft-atlas-mpls-te-express-path to help 
> clarify that this is explicitly NOT delay-based IP routing?
>  
> The Fuxh drafts were trying to go closer to that (and worry about
> queue-delay) last time I looked - certainly closer than these drafts 
> are.
>  
> Alia

Alia,

We should make it clear which express-path work is going forward and whether it has any relationship to draft-ietf-rtgwg-cl-requirements.

Right now we have:

  draft-atlas-mpls-te-express-path

  draft-giacalone-ospf-te-express-path
  (presumed abandonned)

  draft-fuxh-mpls-delay-loss-te-problem-statement and/or
  draft-fuxh-mpls-delay-loss-te-framework

The latter two were presented at the last IETF.

Would the authors of the set of documents above please combine them.
Would they also please consider explicitly addressing the requirments in draft-ietf-rtgwg-cl-requirements, such as stability, keeping the amount of churn down, supporting a grouping of CL component links.  If you do this, would you also please include me in the discussion.

IMO - the ospf and isis work should be halted until there is agreement on what problem is being solved, and the mechanisms to go about solving it.  Until then, we have protocol syntax and no semantics, and no assurance that the current syntax will match the semantics that is later decided is needed (by way of a framework document).

Curtis

ps - sorry to join the discussion on this list late, but I was out for a few days last week and over the long weekend (long in the US).

---

From: John E Drake <[email-masked]>
Date: Thu, 21 Feb 2013 17:10:24 +0000

Curtis,

Comments inline.

Irrespectively Yours,

John

> Sent: Tuesday, February 19, 2013 4:56 PM
> To: Alia Atlas
> Cc: [email-masked]; Lou Berger; [email-masked];
> John E Drake; [email-masked]; [email-masked];
> [email-masked]; [email-masked];
> [email-masked]; [email-masked]; equinox-
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]

JD:  The Express Path drafts are the only drafts that propose how to advertise these parameters.

> 
> Therefore it would be beneficial to look at the set of requirements and
> see if using one document as the base - fine with me if draft-atlas-
> mpls-te-express-path is used as the base - and then see if adding just
> a little to that covers CL requirements, and maybe adding just a little
> more covers anything missed in

JD:  Before doing this it would be useful to see whether there is any current interest in composite links.

> where to get the information.

JD:  The Express Path drafts are specifically agnostic about the source of advertised parameter values.

> requirements may be a NOOP except perhaps FR#8.

JD:  Why is this not covered by the Express Path drafts?

> 1) whether queuing delay is measured,

JD:  It is specifically excluded in the Express Path drafts.

> 2) if so, can measurements at

JD:  It is specifically excluded in the Express Path Drafts.

> a range for the link (ie: 10-12 msec, rounded to 1 msec).  If delay was
> 9.8-11.5 for an interval and 10.2-12.2 the next, it need not be

JD:  Why would it be nice?  What would a node do differently if it had this information?

> 
> Then there is the greater issue of measurement interval and confidence.
> Is the reported value the lowest reading over the interval, highest, or
> some statistical norm?  Is it a mean value over the interval? 98th
> percentile?  Is it necessary to communicate the measurement interval?
> For example, would a NPO be satisfied for delay if the measurement was
> made over the previous hour and any changes would be reflected in the
> next measurement an hour later?  My suggestion was a range and
> precision.

JD:  This information is not available for the existing TE parameters so why is it necessary for the parameters in the Express Path drafts?  What would a node do differently if it had this information?

> 
> I don't think you can just punt on oscillations and stability by saying
> for further study.

JD:  The Express Path drafts address both of these points.

> 
> As for the draft-fuxh- set, I think the only requirement not met is the
> delay difference between working and protect paths.  For that you need
> something along the lines of a min delay and max delay.  Where the
> working and protect are expected to each fall within a range, such that
> this requirement is met.  Note that the mpls-tp-use-cases draft has the
> same exaple.  The authors of the draft-fuxh- set can correct me if I
> missed anything.

JD:  The Express Path drafts address this with the Anomalous flag.

> On the MPLS list I'll make more detailed comments on a new thread.

JD:  It would be courteous to see whether the WG agrees with the change in venue you are proposing.

> 
> Curtis
> 
> 
> In message
> <[email-masked]
> .outlook.com>
> Alia Atlas writes:
> 
> Curtis,
> 
> I know that all of these talk about delay and jitter concerns, but the
> 3 drafts we're discussing are trying to narrow down what to send to a
> reasonable set that doesn't provoke stability concerns and can be used
> to solve the types of issues that Spencer described as in the mpls
> draft.
> 
> Alia
> 
> 
> Cc: Lou Berger; [email-masked]; John E Drake;
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; equinox-
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> In message
> <[email-masked]
> .outlook.com>
> Alia Atlas writes:
> 
> We should make it clear which express-path work is going forward and
> whether it has any relationship to draft-ietf-rtgwg-cl-requirements.
> 
> Right now we have:
> 
> Would they also please consider explicitly addressing the requirments
> in draft-ietf-rtgwg-cl-requirements, such as stability, keeping the
> amount of churn down, supporting a grouping of CL component links.  If
> you do this, would you also please include me in the discussion.
> 
> IMO - the ospf and isis work should be halted until there is agreement
> on what problem is being solved, and the mechanisms to go about solving
> it.  Until then, we have protocol syntax and no semantics, and no
> assurance that the current syntax will match the semantics that is
> later decided is needed (by way of a framework document).
> 
> Curtis
> 
> ps - sorry to join the discussion on this list late, but I was out for
> a few days last week and over the long weekend (long in the US).
> 
> 
>

---

From: Alia Atlas <[email-masked]>
Date: Thu, 21 Feb 2013 15:56:34 -0500

Hi Curtis,

> I think that the answer should be one.  Not two.  Not three or more.

In general, we should avoid duplication.  Where there is strong
disagreement about the requirements and usage, or where there isn't
sufficient maturity on all the aspects, then it makes sense to think about
the problem.

There's also the difference between trying to make a cup of tea and boiling
the ocean.

Finally, it is good to get things done - particularly when drafts are
sitting for 1+ years without comments...

> draft-fuxh-mpls-delay-loss-* that is deemed important.

There are some  requirements in draft-fuxh-mpls-delay-loss which are
concerning, such as the desire to
include node queuing delay and use that as a criteria for paths without
addressing any of the stability concerns
that started this conversation.  I have NO interest in trying to meet that
requirement  without someone doing the necessary
convincing studies, analysis & convincing.  Someone else can define a new
sub-TLV to advertise node delays and related information.   In this case,
IMHO, it's useful to separate out the fairly static link delay from the
traffic-dependent queuing delay.
So - no duplication there.   It needs significant research and analysis to
prove it's useful; I don't see a reason to wait on the
draft-ietf-ospf-te-metric-extensions or
draft-previdi-isis-te-metric-extensions for that.

As for the RSVP-TE path related requirements, these drafts do go much much
farther than draft-atlas-mpls-te-express-path does.   This is the
ocean-boiling (perhaps pond) of handling inter-area, inter-as, and LSPs
without explicit routes completely computed by the ingress.  Those aspects
may be addressing very real use-cases - but that is much more extensive and
complicated than a simple informational draft showing how the extended TE
metrics can be useful today with minimal effort.

> where to get the information.

Right, the intent is that the information could come from many places, and
John Drake
can quote the RFC that is a good example.

> requirements may be a NOOP except perhaps FR#8.

If you look at draft-ietf-ospf-te-metric-extensions, the delay can range
from 1 micro-second to
16.777215 seconds.  Do we really need more dynamic range and associated
complexity?

Certainly, if draft-ietf-ospf-te-metric-extensions and
draft-previdi-isis-te-metric-extensions meet
the composite-link requirements, that is useful; it's always nice to solve
multiple problems at the
same time.

> it could be in the range of 9.5-12.499 and still count as 10-12.

(1),(2), and (3) It's not measuring queuing delay.  I'll work on improved
text to clarify this.

> suggestion was a range and precision.

The draft states that it should be an average.  Why would this information
WANT to be
communicated?  It seems the type of thing that would be documented - perhaps
configurable on a router, but I can't see a reason for signaling it.

 I can see that there might be an argument for using a 98th percentile vs.
an average.

> saying for further study.

Why not?  What is being measured and reported does not change as a result of
traffic load.  It's based on characteristics of the link and not queuing.
 Perhaps one should
put in some words about future study for stability for virtual links, whose
measurements may
include queuing delay in them?  Perhaps we should should say that these
values shouldn't
be advertised for such links?  Or have a way of measuring them that assumes
maximum
queuing delay at each hop on the virtual link's path?

> has the same exaple.  The authors of the draft-fuxh- set can correct

Again, the draft-fuxh-set is trying to do much more complicated stuff for
computation;
I can see those authors writing a suitable draft to explain how to compute
LSPs
between a min delay and a max delay.

> On the MPLS list I'll make more detailed comments on a new thread.

I've included ISIS back as well, because the concerns were raised there
about
draft-previdi-isis-te-metric-extensions and the application and usage
discussed
is important for ISIS to draw conclusions.

Alia

>
> Curtis
>
>
> In message <
> [email-masked]
> >
> Alia Atlas writes:
>
> Curtis,
>
> I know that all of these talk about delay and jitter concerns, but the 3
> drafts we're discussing are trying to narrow down what to send to a
> reasonable set that doesn't provoke stability concerns and can be used to
> solve the types of issues that Spencer described as in the mpls draft.
>
> Alia
>
>
> [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> In message <
> [email-masked]
> >
> Alia Atlas writes:
>
> Would they also please consider explicitly addressing the requirments in
> draft-ietf-rtgwg-cl-requirements, such as stability, keeping the amount of
> churn down, supporting a grouping of CL component links.  If you do this,
> would you also please include me in the discussion.
>
> IMO - the ospf and isis work should be halted until there is agreement on
> what problem is being solved, and the mechanisms to go about solving it.
>  Until then, we have protocol syntax and no semantics, and no assurance
> that the current syntax will match the semantics that is later decided is
> needed (by way of a framework document).
>
> Curtis
>
> ps - sorry to join the discussion on this list late, but I was out for a
> few days last week and over the long weekend (long in the US).
>
>
>
> _______________________________________________
> mpls mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mpls
>

---

From: "Mcdysan, David E" <[email-masked]>
Date: Thu, 21 Feb 2013 16:48:34 -0500

Some responses in line below related to draft-fuxh-mpls-delay-loss-te-problem-statement and -framework in line below. Cross posting to camp since the mols wg and the AD's decided in Atlanta that these drafts should be discussed in camp since they include multi-layer aspects.
Dave
From: Alia Atlas <[email-masked]<mailto:[email-masked]>>
Date: Thursday, February 21, 2013 3:56 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Mpls <[email-masked]<mailto:[email-masked]>>, Nabil Bitar <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Alia Atlas <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [mpls] ISIS TE Metric Extension Discussion (IETF-85 Followup)

Hi Curtis,

On Tue, Feb 19, 2013 at 7:56 PM, Curtis Villamizar <[email-masked]<mailto:[email-masked]>> wrote:

Alia,

In general, we should avoid duplication.  Where there is strong disagreement about the requirements and usage, or where there isn't sufficient maturity on all the aspects, then it makes sense to think about the problem.

There's also the difference between trying to make a cup of tea and boiling the ocean.

Finally, it is good to get things done - particularly when drafts are sitting for 1+ years without comments...

I think that mpls-te-express-path and the igp drafts cover case 1 of Section 5.3 of draft-fuxh-mpls-delay-loss-te-problem-statement-01

1.  The case of few nodes with large geographic separation, (e.g.,trans-oceanic), where link delay alone would be a good approximation.

Would like to hear from others if the data center and mix of link/node delay for cases 2 and 3 is of strong interest to anyone.

draft-fuxh-mpls-delay-loss-te-framework documented our understanding of what requirements the express path igp and mpls drafts met. Suggest that the IGP and MPLS draft authors review that draft to see if anything in there is covered, or could be easily added/changed.

There are some  requirements in draft-fuxh-mpls-delay-loss which are concerning, such as the desire to
include node queuing delay and use that as a criteria for paths without addressing any of the stability concerns
that started this conversation.

In the -01 version we tried to state requirements that would address these stability concerns, for example in section 5.2, draft-fuxh-mpls-delay-loss-te-problem-statement-01

The solution SHOULD provide a means to dynamically measure and/or estimate the one-way link and node performance parameters for delay, loss and delay variation.
As defined in section 2.2. , the estimation interval for the performance parameters is assumed to be on the order of minutes. The solution MUST not impact stability nor significantly increase convergence time if performance parameters change over a timeframe on the order of minutes.

 I have NO interest in trying to meet that requirement  without someone doing the necessary
convincing studies, analysis & convincing.
In my opinion, the mandatory requirement is to allow configuration of the parameters, which would enable an operator to measure parameters and configure them that could be reported on in an analysis/study as stated in 5.2.

The solution MUST provide a means to configure the one-way link and node performance parameters for delay, loss and delay variation.

 Someone else can define a new sub-TLV to advertise node delays and related information.   In this case, IMHO, it's useful to separate out the fairly static link delay from the traffic-dependent queuing delay.
So - no duplication there.   It needs significant research and analysis to prove it's useful; I don't see a reason to wait on the draft-ietf-ospf-te-metric-extensions or draft-previdi-isis-te-metric-extensions for that.

Agreed, Section 5.3 of draft-fuxh-mpls-delay-loss-te-problem-statement-01 states that node level delay is a separate requirement that is desirable (SHOULD), not mandatory.

As for the RSVP-TE path related requirements, these drafts do go much much farther than draft-atlas-mpls-te-express-path does.   This is the ocean-boiling (perhaps pond) of handling inter-area, inter-as, and LSPs without explicit routes completely computed by the ingress.  Those aspects may be addressing very real use-cases - but that is much more extensive and complicated than a simple informational draft showing how the extended TE metrics can be useful today with minimal effort.

Right, the intent is that the information could come from many places, and John Drake
can quote the RFC that is a good example.

If you look at draft-ietf-ospf-te-metric-extensions, the delay can range from 1 micro-second to
16.777215 seconds.  Do we really need more dynamic range and associated complexity?

Certainly, if draft-ietf-ospf-te-metric-extensions and draft-previdi-isis-te-metric-extensions meet
the composite-link requirements, that is useful; it's always nice to solve multiple problems at the
same time.

(1),(2), and (3) It's not measuring queuing delay.  I'll work on improved text to clarify this.

The draft states that it should be an average.  Why would this information WANT to be
communicated?  It seems the type of thing that would be documented - perhaps
configurable on a router, but I can't see a reason for signaling it.

In my opinion, the mandatory requirement from fuxh-problem-statement in section 5.2 is to allow configuration of the parameters, which would enable an operator to change these based upon static or dynamic measurements.

The solution MUST provide a means to configure the one-way link and node performance parameters for delay, loss and delay variation.
I think that the IGP drafts support this.

 I can see that there might be an argument for using a 98th percentile vs. an average.

Why not?  What is being measured and reported does not change as a result of
traffic load.  It's based on characteristics of the link and not queuing.  Perhaps one should
put in some words about future study for stability for virtual links, whose measurements may
include queuing delay in them?  Perhaps we should should say that these values shouldn't
be advertised for such links?  Or have a way of measuring them that assumes maximum
queuing delay at each hop on the virtual link's path?

This is a fair summary. Note that the requirement is desirable (SHOULD).

Again, the draft-fuxh-set is trying to do much more complicated stuff for computation;
I can see those authors writing a suitable draft to explain how to compute LSPs
between a min delay and a max delay.

If you don't mind, I think we should take this thread to MPLS.

At the last IETF, the mpls wg and the AD's decided that the fuxh drafts should go to camp. I missed the deadline to submit a -00 draft to camp and they won't get discussed there. Would really like to get some feedback on what aspects are more important than others. I made a number of the requirements SHOULD/MAY since IMO they are of lower priority. There are still a number of Open MUST requirements. Cross posting this thread to camp for this reason.

 I

I've included ISIS back as well, because the concerns were raised there about
draft-previdi-isis-te-metric-extensions and the application and usage discussed
is important for ISIS to draw conclusions.

Alia

Curtis

In message <[email-masked]<mailto:[email-masked]>>
Alia Atlas writes:

Curtis,

are UNCONNECTED

I would say not completely unconnected, the drafts mentioned below meet some of the requirements in   draft-fuxh-mpls-delay-loss-te-problem-statement-01 as documented in  draft-fuxh-mpls-delay-loss-te-framework draft.

to the drafts that we are discussing, which are:

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, February 19, 2013 1:57 AM
To: Alia Atlas
Cc: Lou Berger; [email-masked]<mailto:[email-masked]>; John E Drake; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Alia,

Right now we have:

Would the authors of the set of documents above please combine them.
Would they also please consider explicitly addressing the requirments in draft-ietf-rtgwg-cl-requirements, such as stability, keeping the amount of churn down, supporting a grouping of CL component links.  If you do this, would you also please include me in the discussion.
 draft-fuxh-mpls-delay-loss-te-problem-statement-01 tried to explicitly state the CL requirements which overlap with it.

If I missed something, please identify and I can update -- if the community believes collecting these requirements (near and longer term, straightforward and controversial in one place) is of value, I would like to receive some affirmation before investing more time updating draft-fuxh-mpls-delay-loss-te-problem-statement and -framework.

Curtis

_______________________________________________
mpls mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/mpls

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 21 Feb 2013 18:30:40 -0500

> John

Hi John,

Thanks for the reply.  Discussion is good.  See inline.

Curtis

> advertise these parameters.

There are three Express Path drafts (see paragraph below), one written
by Alia.  I think you mean the te-metric-extension drafts.

The point is that if there are three sets of requirements and they
don't perfectly overlap, then you end up with more than one TLV to
express delay and jitter and related parameters.

> current interest in composite links.

CL requirements went through WGLC a long time ago and was awaiting
progress on CL Framework.  Alia just suggested we repeat WGLC and
advance CL requirements alone.

One of the things holding back CL Framework is the uncertain state of
some of the referenced work, including work on delay and jitter.  I'd
be happy to change the references to draft-atlas-mpls-te-express-path
as you will note two paragraphs below.  This would consolidate related
efforts within IETF which is generally considered a good thing.

> of advertised parameter values.

And that is an issue.  It is in conflict with what Alia wrote
previously on this thread, claiming we need not worry about stability
because te-express-path was concerned only with the relatively static
propogation delay (and maybe serialization in geographically small
networks) and specifically not queuing delay.  Your assertion agrees
with my read of the document.  Therefore we do have to worry about
stability.

The problem though is that the document doesn't say what to do.  As
you said, it is agnostic.  Being agnostic in that way is bad for
interoperability because some implementations may assume no queuing
delay and set timers short for quick reaction to an underlying layer
of some link switching to a protection path, and the sender may
include queuing delay.  There may be a mix of interpretations, with
some LSR reporting queuing delay, and others not doing so.

Agnostic is fine as long as it doesn't impact interoperability.  If
te-express-path wants to be agnostic, then the TLVs in the two
te-metric-extension should (or IMO MUST) report whether they are
including queuing delay.

If queuing delay is reported, the PHB over which the measurement is
sent (for example, using TP DM OAM) should be reported.

This is one reason why the MPLS WG needs to settle on what is needed
in te-express-path before te-metric-extension can advance.

> JD: Why is this not covered by the Express Path drafts?

Not explicitly.  Since it is configurable and can affect the path
decision, it should also be reported in the TLVs.

> JD: It is specifically excluded in the Express Path drafts.

You just said it was agnostic.  If queuing delay is prohibited, please
point out where in the draft it is prohibited.  I read:

   When considering performance-based data, it is obvious that there
   are additional contributors beyond just the links.  Clearly
   end-to-end latency is a combination of router latency, queuing
   latency, physical link latency and other factors.  However, if
   application traffic requires paths to be selected based upon
   latency constraints, the same traffic might be in an Expedited
   Forwarding Per-Hop- Behavior[RFC3246] with minimal queuing delay or
   another PHB with known maximal per-hop queuing delay.  While
   traversing a router can cause delay, that can be included in the
   advertised link delay.

   This document does not specify how a router determines what values
   to advertise by the IGP.  [...]

The last two sentences indicate that the queuing delay can be included
in the reported delay.  If the first paragraph is trying to say don't
include queuing delay in the reported delay, it certainly seems to me
to read the other way around and is at best unclear.  The second
paragraph clearly says it doesn't care what values are advertised.

> JD: It is specifically excluded in the Express Path Drafts.

Again.  Where is it excluded.  Note the paragraph above says;

   However, if application traffic requires paths to be selected based
   upon latency constraints, the same traffic might be in an Expedited
   Forwarding Per-Hop- Behavior[RFC3246] with minimal queuing delay or
   another PHB with known maximal per-hop queuing delay.

It doesn't (clearly) say which to measure, the delay without any
queuing delay, the delay at EF PHB, or the delay at BE PHB.

You just said it was agnostic, then said queuing delay is excluded,
but the draft doesn't clearly give any restrictions.

It is also worth noting that jitter is extremely small without queuing
delay (but non-zero), so it would be interesting to hear why jitter is
needed if measurements exclude queuing delay.

Would you or Alia please reword this paragraph and resend it.

> JD: Why would it be nice?  What would a node do differently if it had

If a LSP ingress gets the number 10, it would be nice to know if that
means 9.9 to 10.1 or 9.5 to 12.5 (or something else, like <=10.000).
If the budget had only 11 msec before violating NPO, then it would
make a different path decision depending on which it was.  To reduce
readvertisement and to improve stability using a range and rounding
would be useful, particularly if queuing delay is included.

The te-metric-extension drafts indicates one value in microseconds
with no qualifier.  A SONET frame is 64KB and a ODU superframe is
larger.  In either case, a packet coming in a 10G POS/OC192 or
GFP/ODU3 can experience a different delay depending on whether a low
utilization put a late frame in the front of the layer2 frame or high
utilization put it later in the frame.  This jitter is well over a
microsecond (52 usec for POS and larger for GFP/ODU3).  This number is
even higher for lower speed POS interfaces (like OC3,12,48).  So even
without queuing delay, in reality there is a range that is much
greater than the reported 1 usec.

> JD: This information is not available for the existing TE parameters
> so why is it necessary for the parameters in the Express Path drafts?
> What would a node do differently if it had this information?

There is no measurement interval and confidence applicable to
administatively assigned metric or administratively assigned
attributes (colors) or reservable bandwidth at any given priority or
any of the other existing TLVs.

Measurement interval and confidence is applicable to delay and jitter.
I gave an example of why this can matter so please reread the example.

> JD: The Express Path drafts address both of these points.

Not that I see.  If you think the points are addressed, please point
to where the points are addressed.

> JD: The Express Path drafts address this with the Anomalous flag.

The point was that the delay difference between the working and
protect paths should be bounded.  To do so, assuming the working path
has a shorter delay, you need the minimum delay of the working path
and the maximum delay of the protect path.

If jitter excludes queuing delay then the measurement interval does
not matter for relatively long measurement intervals (ie: intervals
are certainly going to be much longer than one msec).  For queuing
delay the measurement interval matters.  Jitter over a 100 msec might
be small but over 10s of seconds could be much larger due to the
sawtooth offerred load of one or more extremely large TCP flows.

> in venue you are proposing.

Since this discussion is mostly about draft-atlas-mpls-te-express-path
and that is intended for the MPLS WG, IMO this discussion should
remain on MPLS.  The discussion of whether
draft-atlas-mpls-te-express-path needs to preceed the two
te-express-path drafts can be a matter for OSPF and ISIS.

Acee wrote:

  Hi Spence, 

  Note that the bar for WG last call and request for publication as a
  standards track RFC is quite a bit higher than for WG adoption (OSPF
  WG copied). We need to have this discussion now and either address
  the questions surrounding the delay metrics or agree that they are
  out of scope.

  Having said that, speaking as a WG member of both WGs, I have
  re-read the draft and don't have problems with the guidance given
  for delay metrics when these metrics are used for TE.

  Thanks,
  Acee 

That seems to me to indicate that discussion is needed.  I'm quite
sure Acee is also on the MPLS WG mailing list.

---

From: John E Drake <[email-masked]>
Date: Fri, 22 Feb 2013 00:00:48 +0000

Curtis,

I am busy on other things and it will be a while before I respond in detail.  I have a couple of quick points though.

When I was using the term 'agnostic' it was wrt how the measurement was made, not what was being measured.  For example, we want to advertise link propagation delay, but whether the measurement was made via RFC 6374, received from a lower layer network, or performed using optical interferometry, is immaterial.

It has also been a while since I read Alia's draft and I was surprised with your quote from it.  I think it might benefit from another serious review. 

Irrespectively Yours,

John

> Sent: Thursday, February 21, 2013 3:31 PM
> To: John E Drake
> Cc: [email-masked]; Alia Atlas; Lou Berger;
> [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> <[email-masked]
> .outlook.com>
> John E Drake writes:
> 
> > Curtis,
> >
> > Comments inline.
> >
> > Irrespectively Yours,
> >
> > John
> 
> 
> Hi John,
> 
> equinox-
> > > draft-atlas- mpls-te-express-path is used as the base - and then
> see
> > > if adding just a little to that covers CL requirements, and maybe
> > > adding just a little more covers anything missed in
> > > FR#7 essentially asks for interworking with delay measured at a
> > > lower layer.  Obviously this would *not* include queuing delay.
> > > Maybe no impact on draft-atlas-mpls-te-express-path since it does
> > > not specify where to get the information.
> >
> > JD: The Express Path drafts are specifically agnostic about the
> source
> > of advertised parameter values.
> 
> The problem though is that the document doesn't say what to do.  As you
> said, it is agnostic.  Being agnostic in that way is bad for
> delay and set timers short for quick reaction to an underlying layer of
> some link switching to a protection path, and the sender may include
> queuing delay.  There may be a mix of interpretations, with some LSR
> reporting queuing delay, and others not doing so.
> 
> Agnostic is fine as long as it doesn't impact interoperability.  If te-
> express-path wants to be agnostic, then the TLVs in the two te-metric-
> extension should (or IMO MUST) report whether they are including
> queuing delay.
> 
> This is one reason why the MPLS WG needs to settle on what is needed in
> te-express-path before te-metric-extension can advance.
> 
> You just said it was agnostic, then said queuing delay is excluded, but
> the draft doesn't clearly give any restrictions.
> 
> > > precision of the delay measurement was reported, for example
> > > reporting a range for the link (ie: 10-12 msec, rounded to 1 msec).
> > > If delay was
> > > readvertised but its nice to know through the reported precision
> > > that it could be in the range of 9.5-12.499 and still count as 10-
> 12.
> >
> If the budget had only 11 msec before violating NPO, then it would make
> a different path decision depending on which it was.  To reduce
> The te-metric-extension drafts indicates one value in microseconds with
> no qualifier.  A SONET frame is 64KB and a ODU superframe is larger.
> In either case, a packet coming in a 10G POS/OC192 or
> without queuing delay, in reality there is a range that is much greater
> than the reported 1 usec.
> 
> confidence.
> > > Is the reported value the lowest reading over the interval,
> highest,
> > > or some statistical norm?  Is it a mean value over the interval?
> > > 98th percentile?  Is it necessary to communicate the measurement
> interval?
> > > For example, would a NPO be satisfied for delay if the measurement
> > > was made over the previous hour and any changes would be reflected
> > > in the next measurement an hour later?  My suggestion was a range
> > > and precision.
> >
> administatively assigned metric or administratively assigned attributes
> (colors) or reservable bandwidth at any given priority or any of the
> other existing TLVs.
> 
> Not that I see.  If you think the points are addressed, please point to
> where the points are addressed.
> 
> > > the delay difference between working and protect paths.  For that
> > > you need something along the lines of a min delay and max delay.
> > > Where the working and protect are expected to each fall within a
> > > range, such that this requirement is met.  Note that the
> > > mpls-tp-use-cases draft has the same exaple.  The authors of the
> > > draft-fuxh- set can correct me if I missed anything.
> >
> The point was that the delay difference between the working and protect
> paths should be bounded.  To do so, assuming the working path has a
> shorter delay, you need the minimum delay of the working path and the
> maximum delay of the protect path.
> 
> If jitter excludes queuing delay then the measurement interval does not
> matter for relatively long measurement intervals (ie: intervals are
> certainly going to be much longer than one msec).  For queuing delay
> the measurement interval matters.  Jitter over a 100 msec might be
> small but over 10s of seconds could be much larger due to the sawtooth
> offerred load of one or more extremely large TCP flows.
> 
> > JD: It would be courteous to see whether the WG agrees with the
> change
> > in venue you are proposing.
> 
> and that is intended for the MPLS WG, IMO this discussion should remain
> on MPLS.  The discussion of whether draft-atlas-mpls-te-express-path
> needs to preceed the two te-express-path drafts can be a matter for
> OSPF and ISIS.
> 
> Acee wrote:
> 
>   Hi Spence,
> 
> That seems to me to indicate that discussion is needed.  I'm quite sure
> Acee is also on the MPLS WG mailing list.
> 
> > > Curtis
> > >
> > >
> > > In message
> > >
> > > 3 drafts we're discussing are trying to narrow down what to send to
> > > a reasonable set that doesn't provoke stability concerns and can be
> > > used to solve the types of issues that Spencer described as in the
> > > mpls draft.
> > >
> > > Alia
> > >
> > >
> > > [email-masked]; [email-masked];
> > > [email-masked]; [email-masked]; [email-masked];
> > > [email-masked]; [email-masked];
> > > [email-masked]; equinox- [email-masked];
> > > [email-masked]; [email-masked]; [email-masked];
> > > [email-masked]; [email-masked]; [email-masked];
> > > [email-masked]
> > > > Should we add a reference to draft-atlas-mpls-te-express-path to
> > > > help clarify that this is explicitly NOT delay-based IP routing?
> > > >
> > > > queue-delay) last time I looked - certainly closer than these
> > > > drafts are.
> > > >
> > > > Alia
> > >
> > >
> > > Alia,
> > >
> > > Would they also please consider explicitly addressing the
> > > requirments in draft-ietf-rtgwg-cl-requirements, such as stability,
> > > keeping the amount of churn down, supporting a grouping of CL
> > > component links.  If you do this, would you also please include me
> in the discussion.
> > >
> > > IMO - the ospf and isis work should be halted until there is
> > > agreement on what problem is being solved, and the mechanisms to go
> > > about solving it.  Until then, we have protocol syntax and no
> > > semantics, and no assurance that the current syntax will match the
> > > semantics that is later decided is needed (by way of a framework
> document).
> > >
> > > Curtis
> > >
> > > ps - sorry to join the discussion on this list late, but I was out
> > > for a few days last week and over the long weekend (long in the
> US).

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 22 Feb 2013 00:08:59 +0000

Dear Alia, et al.,
"What is being measured and reported does not change as a result of
traffic load.  It's based on characteristics of the link and not queuing.  Perhaps one should
put in some words about future study for stability for virtual links, whose measurements may
include queuing delay in them?  Perhaps we should should say that these values shouldn't
be advertised for such links?  Or have a way of measuring them that assumes maximum
queuing delay at each hop on the virtual link's path?"
I think that uncertainty of whether it is possible to measure delay without queing requires careful consideration  since:

  *    if in fact such measurements are not plausible then delay effectively becomes propagation delay and is, for practical purposes, static metric.
  *   but if delay metric is dynamically measureable, then, I believe, performance measurement methodology must be consistent throughout the IGP domain (test packet TC setting, packet length, inter-paket interval etc.)

And, piggyabcking on, IGP-TE question:

  *   how Unreserved Bandwidth is different from Residiual BW proposed in IGP TE documents as new sub-TLV.

    Regards,
        Greg

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Alia Atlas
Sent: Thursday, February 21, 2013 12:57 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Samita Chakrabarti; Acee Lindem; Alia Atlas; [email-masked]; [email-masked]
Subject: Re: [mpls] ISIS TE Metric Extension Discussion (IETF-85 Followup)

Hi Curtis,

Alia,

There are some  requirements in draft-fuxh-mpls-delay-loss which are concerning, such as the desire to
include node queuing delay and use that as a criteria for paths without addressing any of the stability concerns
that started this conversation.  I have NO interest in trying to meet that requirement  without someone doing the necessary
convincing studies, analysis & convincing.  Someone else can define a new sub-TLV to advertise node delays and related information.   In this case, IMHO, it's useful to separate out the fairly static link delay from the traffic-dependent queuing delay.
So - no duplication there.   It needs significant research and analysis to prove it's useful; I don't see a reason to wait on the draft-ietf-ospf-te-metric-extensions or draft-previdi-isis-te-metric-extensions for that.

Alia

Curtis

Curtis,

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, February 19, 2013 1:57 AM
To: Alia Atlas
Cc: Lou Berger; [email-masked]<mailto:[email-masked]>; John E Drake; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Alia,

Right now we have:

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 21 Feb 2013 22:51:52 -0500

> Hi Curtis,

Hi Alia,

> sufficient maturity on all the aspects, then it makes sense to think
> about the problem.

OK.  Consolidation is good.

> There's also the difference between trying to make a cup of tea and

I agree here too.  If you look at the thread with John I am asking for
relatively little changes.  I have yet to send a full review of the
document.  We are discussion issues at the moment and where things in
the document are underspecified or unspecified, the discussion can be
more open ended.

> sitting for 1+ years without comments...

No comment can mean lack of interest.  Now you have some comments from
me which is good because there is at least some interest.  I don't
think dismissing substantive comments and moving on is useful.

So if you don't mind, lets set aside philosophical or procedural
discussion and continue the technical discussion and decide whether
anything in te-express-path and the protocol documents needs changing.

> concerning, such as the desire to include node queuing delay and use
> that as a criteria for paths without addressing any of the stability
> concerns that started this conversation.  I have NO interest in trying
> to meet that requirement without someone doing the necessary
> convincing studies, analysis & convincing.  Someone else can define a
> new sub-TLV to advertise node delays and related information.  In this
> case, IMHO, it's useful to separate out the fairly static link delay
> from the traffic-dependent queuing delay.  So - no duplication there.
> It needs significant research and analysis to prove it's useful; I
> don't see a reason to wait on the draft-ietf-ospf-te-metric-extensions
> or draft-previdi-isis-te-metric-extensions for that.

BTW- I and most other people consider queuing delay to be a component
of link delay since the queues are per link.  Node delay is usually
considered to be the delay involved in crossing the switching fabric,
which is relatively constant per interface.

If you want to exclude queuing delay, then that should be very clearly
stated in te-express-path and in both te-metric-extension drafts.
That would end a lot of discussion.  It would also mean changing this
text:

   This document does not specify how a router determines what values
   to advertise by the IGP.  However, the end-to-end performance that
   is computed for an LSP path SHOULD be built from the individual
   link data.  Any end-to-end characterization used to determine an
   LSP's performance compliance should be fully reflected in the
   Traffic Engineering Database so that a CSPF calculation can also
   determine whether a path under consideration would be in
   compliance.

A reasonable replacement would be:

   When considering performance-based data, it is obvious that there
   are additional contributors beyond just the links.  Clearly
   end-to-end latency is a combination of router latency, queuing
   latency, physical link latency and other factors.  Only physical
   link latency is considered.  While traversing a router can small
   delay, such as the delay associated with traversing a switch
   fabric, that can be included in the advertised link delay.

   Queuing latency is specifically excluded to insure freedom from
   oscillations and stability issues that have plauged prior attempts
   to use delay as a routing metric.  If application traffic which
   follows path based upon latency constraints, the same traffic might
   be in an Expedited Forwarding Per-Hop-Behavior [RFC3246] with
   minimal queuing delay or another PHB with potentially very
   substantial per-hop queuing delay.  Only traffic which experiences
   relatively low congestion, such as Expedited Forwarding traffic,
   will experience delays very close to the sum of the reported link
   delays.

   This document does not specify how a router determines the link
   delays that are advertised by the IGP.  However, the end-to-end
   performance estimate that is computed for an LSP path SHOULD be
   built from the individual link data.  Any end-to-end
   characterization used to determine an LSP's performance compliance
   should be fully reflected in the Traffic Engineering Database so
   that a CSPF calculation can also determine whether a path under
   consideration would be in compliance.

Note that the last sentence, which remains unchanged indicates that if
TP DM OAM or other measure of path delay is different, then that must
be indicated in the TE-LSDB.  That would only be possible if a FA was
created for the LSP.  If the LSP was rerouted, then the measurement no
longer exists and the FA should also disappear.  That would create a
chance for oscillation.

A better approach would be to create a zero bandwidth LSP on a
potential path specifically for the purpose of measuring end-to-end
delay at high priority (high TC value is using diffserv CS service -
IP Prec) and run DM OAM or other measurement of delay on the LSP.
Since the LSP never goes away, there is no chance of oscillation.
These LSP would only exist on path that were either in use, or were
feasible based on the sum of the delays, but were infeasible based on
the path measurements.

> As for the RSVP-TE path related requirements, these drafts do go much
> much farther than draft-atlas-mpls-te-express-path does.  This is the
> ocean-boiling (perhaps pond) of handling inter-area, inter-as, and
> LSPs without explicit routes completely computed by the ingress.
> Those aspects may be addressing very real use-cases - but that is much
> more extensive and complicated than a simple informational draft
> showing how the extended TE metrics can be useful today with minimal
> effort.

Yes.  I absolutely agree that tackling intra-AS first and any other
inter-domain later (if ever) is a better approach.  I have made
comments of this sort to the authors of draft-fuxh-mpls-delay-loss-*
but have not received a reply.  If that situation continues (no reply
from the authors) then perhaps we should consider
draft-fuxh-mpls-delay-loss-* to be abandonned.

> Right, the intent is that the information could come from many places,
> and John Drake can quote the RFC that is a good example.

OK John.  You are on stage next.  I don't know which RFC Alia is
referring to.

I think we are in agreement that the measurement should be link delay,
specifically exclude queuing delay, and optionally include any
relatively fixed (and generally extremely small) router delay due to
such things as the fabric in with the link delay.  Note that fabric
delays are generally a fairly small number of microseconds, as long as
any VOQ related delays are excluded.

> If you look at draft-ietf-ospf-te-metric-extensions, the delay can
> range from 1 micro-second to 16.777215 seconds.  Do we really need
> more dynamic range and associated complexity?

The hysteresis should be configurable.  If the hysteresis is such that
we are sending an updated TLV for any given link only when there is a
substantial change, then the range will still be in the tens or
hundreds of microseconds.  What most people don't realize is that for
packet data, even the number of FEC corrected bit-errors affects the
number of FEC passes and the latency.  This can be a few multiples of
the TDM (ie: SONET or OTN) superframe time.  With very few or no
correctable bit errors (ie: very clean and one FEC pass), for packet
data there is still a jitter of about a superframe time, as I
explained in my last email to John.

I'm fine with excluding queuing delay.  If so, then providing a range
of delay is far less important.  This would be fine for high priority
services that have small queuing delay (no more than a few packet
times).

If anything the resolution is too high.  In a datacenter where there
is negligible geographic delay, then microseconds may matter.  In a
microsecond light travels 200m in fiber (300m in vacume), so for most
datacenters at most only a few microseconds are added when going
between buildings.  Serialization of 1500B at 1Gb/s is 12 usec (which
is why cut-through is popular in datacenter switches and routers).  It
seems to be only the datacenter and campus networks where microsecond
resolution is useful.

> draft-previdi-isis-te-metric-extensions meet the composite-link
> requirements, that is useful; it's always nice to solve multiple
> problems at the same time.

Yes.  And I think there is very little that needs to be changed to
accomplish that.

> (1),(2), and (3) It's not measuring queuing delay.  I'll work on improved

Though it was not clear from a read of your document, it is clear from
your email and John's email that you intended to exclude queuing
delay.

What about a MPLS server layer which is introducing delay?  That would
be a variable delay coming from the server layer.

Also for CL where heterogeneous component links are desired, there is
a need to advertise a range for a group, so that we don't end up with
one group per component link (destroying any benefit of grouping).
Therefore I think a range would be good.

Long term drift in the measured delay could be reported by way of the
range.  Any relatively short term variation in the delay would be
reported as jitter.  For real time applications the jitter would
indicate the size of jitter buffer needed, with drift compensation
through very slight speedup or slow down of the playback (dropping or
interpolating data points typically rather than clock adjustment).

> The draft states that it should be an average.  Why would this
> information WANT to be communicated?  It seems the type of thing that
> would be documented - perhaps configurable on a router, but I can't
> see a reason for signaling it.

The word average does not appear in te-express-path.

> I can see that there might be an argument for using a 98th percentile

I kinda like using a range.  It also has a use for CL if we are
reporting values in microseconds and there is a slight variation in
the length of parallel fibers.

> Why not?  What is being measured and reported does not change as a
> result of traffic load.  It's based on characteristics of the link and
> not queuing.  Perhaps one should put in some words about future study
> for stability for virtual links, whose measurements may include
> queuing delay in them?  Perhaps we should should say that these values
> shouldn't be advertised for such links?  Or have a way of measuring
> them that assumes maximum queuing delay at each hop on the virtual
> link's path?

If you are excluding queuing delay, then you can cite that as a reason
that oscillations should not occur.

You should also point out that if the underlying server layer, such as
an MPLS server layer, has an inherent queuing delay, then there is a
chance from oscillation and even instability, if offerred load is high
enough to impact congestion at the measured PHB.

> Again, the draft-fuxh-set is trying to do much more complicated stuff
> for computation; I can see those authors writing a suitable draft to
> explain how to compute LSPs between a min delay and a max delay.

The working path is computed with a max path delay.  Once a working
path is determined, the protect path is computed with a max path
delay.  If both paths are within range, then they are set up and
rerouting (preferably of the protect path) may be required if the
working path is still within its max path delay but the delay
difference between working and protect is too great as a result of a
change.

> I've included ISIS back as well, because the concerns were raised
> there about draft-previdi-isis-te-metric-extensions and the
> application and usage discussed is important for ISIS to draw
> conclusions.
>  
> Alia

If you agree that both te-metric-extension drafts need to await
resolving issues in te-express-path, then we don't need to cross
post.  Cross posting is a bad practice.

I don't think it will take long to resolve issues with
te-express-path.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 22 Feb 2013 00:50:46 -0500

Dave,

It is extremely obnoxious to those who prefer to read IETF email as
plain text when someone indicates their comments in HTML attributes
such as color and font.  Your mailer
(Microsoft-MacOutlook/14.2.5.121010 according to the headers) did a
terrible job of providing a text alternative.

It is also somewhat obnoxious to cross post.  We now have three IETF
WGs on the Cc.  We need to consolidate to one.  Therefore my followup
will be on the MPLS mailing list with CCAMP and ISIS in the Bcc only,
so they get dropped from the thread but know why it happenned.

Please do not add more WGs to the Cc.

Also please note that I sent mail to the authors of
draft-fuxh-mpls-delay-loss-te-problem-statement and got no reply.

I tend to agree with Alia and John Drake that
draft-fuxh-mpls-delay-loss-* are not related and I do think those
drafts should not go forward at all at least in their current state.

In my prior email I suggested that the authors of the set of drafts
draft-atlas-mpls-te-express-path and draft-fuxh-mpls-delay-loss-* get
together and resolve differences.  If there is anything reasonable in
draft-fuxh-mpls-delay-loss-*, then it can be put into
draft-atlas-mpls-te-express-path.  If there is anything that needs to
be done in followup work, then any draft-fuxh-mpls-delay-loss-*
residuals can become a new followup draft covering topics such as
interlayer interworking or inter-domain extensions.

See inline.

Curtis

> Some responses in line below related to
> draft-fuxh-mpls-delay-loss-te-problem-statement and -framework in line
> below. Cross posting to camp since the mols wg and the AD's decided in
> Atlanta that these drafts should be discussed in camp since they
> include multi-layer aspects.
>  
> Dave
>  
> > From: Alia Atlas <[email-masked]>
> > To: "[email-masked]" <[email-masked]>
> > Cc: Mpls <[email-masked]>, Nabil Bitar
> >     <[email-masked]>, "[email-masked]"
> >     <[email-masked]>, "[email-masked]" <[email-masked]>,
> >     "[email-masked]" <[email-masked]>, "[email-masked]"
> >     <[email-masked]>, "[email-masked]" <[email-masked]>,
> >     "[email-masked]" <[email-masked]>,
> >     "[email-masked]" <[email-masked]>, Alia Atlas
> >     <[email-masked]>, "[email-masked]" <[email-masked]>,
> >     "[email-masked]"
> >     <[email-masked]>
> > Finally, it is good to get things done - particularly when drafts
> > are sitting for 1+ years without comments...
> >
> >>
> I think that mpls-te-express-path and the igp drafts cover case 1 of
> Section 5.3 of draft-fuxh-mpls-delay-loss-te-problem-statement-01
>  
> 1.  The case of few nodes with large geographic separation,
> (e.g.,trans-oceanic), where link delay alone would be a good
> approximation.

Node delay is a small contributor in any network where links are
greater than about 20 km and therefore link delays are greater than
100 usec.

> Would like to hear from others if the data center and mix of link/node
> delay for cases 2 and 3 is of strong interest to anyone.

It is mentioned in te-express-path that node delay (ie: fabric
contribution to delay) can be added to the link delay.  That would
cover the small metro (links < 20km) and data centers.

> draft-fuxh-mpls-delay-loss-te-framework documented our understanding
> of what requirements the express path igp and mpls drafts met. Suggest
> that the IGP and MPLS draft authors review that draft to see if
> anything in there is covered, or could be easily added/changed.

There is apparently some question among te-express-path authors
whether draft-fuxh-mpls-delay-loss-* has any relevance (and I tend to
agree).  Perhaps the authors of this set of draft can work this out
offline, but I'd like to be on the Cc.

> > include node queuing delay and use that as a criteria for paths
> > without addressing any of the stability concerns 
> In the -01 version we tried to state requirements that would address
> these stability concerns, for example in section 5.2,
> draft-fuxh-mpls-delay-loss-te-problem-statement-01
>  
> The solution SHOULD provide a means to dynamically measure and/or
> estimate the one-way link and node performance parameters for delay,
> loss and delay variation.
>  
> As defined in section 2.2. , the estimation interval for the
> performance parameters is assumed to be on the order of minutes. The
> solution MUST not impact stability nor significantly increase
> convergence time if performance parameters change over a timeframe on
> the order of minutes.

That is the slow slosh option we discussed on ISIS.  It is not a good
solution.  Excluding queuing delay is a better solution.  Providing a
range allows the delay TLV to remain unchanged as long as it moves
around within the advertised range (and if not, then the range can be
widenned).

> In my opinion, the mandatory requirement is to allow configuration of
> the parameters, which would enable an operator to measure parameters
> and configure them that could be reported on in an analysis/study as
> stated in 5.2.
>  
> The solution MUST provide a means to configure the one-way link and
> node performance parameters for delay, loss and delay variation.

You and Alia seem to be talking past each other.  Alia is defining
node delay to include queuing delay which is just plain wrong.  You
are referring to the fabric delay that is only applicable to
geographically small networks.  The problem Alia doesn't want to
tackle without research is the problem of queuing delay.

> >  Someone else can define a new sub-TLV to advertise node delays and
> > related information.   In this case, IMHO, it's useful to separate
> > out the fairly static link delay from the traffic-dependent queuing
> > delay.
> > So - no duplication there.   It needs significant research and
> > analysis to prove it's useful; I don't see a reason to wait on the
> Agreed, Section 5.3 of
> draft-fuxh-mpls-delay-loss-te-problem-statement-01 states that node
> level delay is a separate requirement that is desirable (SHOULD), not
> mandatory.

Its nice that you agree but if you are defining node delay differently
than Alia it is hard to understand what you are agreeing to.

> > As for the RSVP-TE path related requirements, these drafts do go
> > much much farther than draft-atlas-mpls-te-express-path does.   This
> > is the ocean-boiling (perhaps pond) of handling inter-area, inter-as,
> > and LSPs without explicit routes completely computed by the ingress.
> > Those aspects may be addressing very real use-cases - but that is
> > much more extensive and complicated than a simple informational draft
> > range from 1 micro-second to 
> > information WANT to be
> In my opinion, the mandatory requirement from fuxh-problem-statement
> in section 5.2 is to allow configuration of the parameters, which
> would enable an operator to change these based upon static or dynamic
> measurements.
>  
> The solution MUST provide a means to configure the one-way link and
> node performance parameters for delay, loss and delay variation.
>  

I think the "Anomalous State" mechanism is a bad idea.  We can discuss
that later on the MPLS WG mailing list only.

> > result of

If you mean supporting the delay difference is a SHOULD, does that
mean that it is an option to not support it in the protocol.  If so,
then maybe te-express-path doesn't need to support it.

> At the last IETF, the mpls wg and the AD's decided that the fuxh
> drafts should go to camp. I missed the deadline to submit a -00 draft
> to camp and they won't get discussed there. Would really like to get
> some feedback on what aspects are more important than others. I made a
> number of the requirements SHOULD/MAY since IMO they are of lower
> priority. There are still a number of Open MUST requirements. Cross
> posting this thread to camp for this reason.

If it is decided that draft-fuxh-mpls-delay-loss-* are not related to
te-express-path, then draft-fuxh-mpls-delay-loss-* serves no purpose.

It might be better to combine and remove anything related to
interlayer interworking and inter-domain and put that in separate
drafts which extend draft-atlas-te-express-path.

> > there about
> I would say not completely unconnected, the drafts mentioned below
> meet some of the requirements in
> draft-fuxh-mpls-delay-loss-te-problem-statement-01 as documented in
> draft-fuxh-mpls-delay-loss-te-framework draft.

I think what Alia and John are trying to say is (sic)
draft-fuxh-mpls-delay-loss-* are trying to boil the ocean and
therefore are making themselves irrelevant in doing so.

> >> I know that all of these talk about delay and jitter concerns, but
> >> the 3 drafts we're discussing are trying to narrow down what to send
> >> to a reasonable set that doesn't provoke stability concerns and can
> >> be used to solve the types of issues that Spencer described as in
> >> the mpls draft.
> >>
> >> Alia
> >>
> >>

---

From: Alia Atlas <[email-masked]>
Date: Sat, 23 Feb 2013 14:34:05 -0500

Curtis,

I am severely trimming this thread and addressing only the parts related to
draft-atlas-mpls-te-express-path here.

>    consideration would be in compliance.

I've updated those 2 paragraphs to be the following 3 paragraphs (keeping
your new middle paragraph intact & just modifying a bit about the other
two):

"When considering performance-based data, it is obvious that there
are additional contributors beyond just the links. Clearly end-to-end
latency is a combination of router latency, queuing latency, physical
link latency and other factors.  While traversing a router can cause
delay, that can be included in the advertised link delay.  As
described in [I-D.ietf-ospf-te-metric-extensions] and
[I-D.previdi-isis-te-metric-extensions], queuing delay
should not be included in the measurements advertised by OSPF or ISIS.

Queuing latency is specifically excluded to insure freedom from
oscillations and stability issues that have plagued prior attempts to
use delay as a routing metric.  If application traffic which follows
path based upon latency constraints, the same traffic might be in an
Expedited Forwarding Per-Hop-Behavior [RFC3246] with
minimal queuing delay or another PHB with potentially very substantial
per-hop queuing delay.  Only traffic which experiences relatively low
congestion, such as Expedited Forwarding traffic, will experience
delays very close to the sum of the reported link delays.

This document does not specify how a router determines what values
to advertise by the IGP; it does assume that the constraints specified
in [I-D.ietf-ospf-te-metric-extensions] and
[I-D.previdi-isis-te-metric-extensions] are followed.
Additionally, the end-to-end performance that is computed for an LSP
path SHOULD be built from the individual link data. Any end-to-end
characterization used to determine an LSP's performance compliance
should be fully reflected in the Traffic Engineering Database so that
a CSPF calculation can also determine whether a path under
consideration would be in compliance.
"

> feasible based on the sum of the delays, but were infeasible based on

[Alia] I don't see that an FA appearing and disappearing would add more
chance of
oscillation than having a physical link appear and disappear.

I'm not opposed to adding some thoughts and details for FAs - but I'm not
sure how much
we want to specify.  The (new) guidance in the IGP drafts is pretty clear
that queuing delay
should be excluded.

> draft-fuxh-mpls-delay-loss-* to be abandonned.

[Alia] We did hear from Dave :-)  Before we even go inter-area/level, a
key question is whether there is serious interest in computing these LSPs
someplace other than
at the ingress or a PCE?  draft-atlas-mpls-te-express-path doesn't define
that and I'd prefer not to
unless there is a clear requirement and need for it.

> difference between working and protect is too great as a result of a

[Alia] I understand that a path might require rerouting if the delay
difference is too great,
but I'm not clear on what changes one would make to the CSPF (mods to the
topology
really) to avoid having the same issue again.  Certainly, one can exclude
links in the old
path, but that could still cause either no solution or a new solution that
suffers from the same
delay based problem.   I could brainstorm about algorithms (maybe based on
Surballe's?)
but (concerns about standardizing CSPF algorithms aside) I don't see a
clear and
simple method off-hand.  IF you have a good idea and want to write it up,
then I'm happy to
discuss.  I know you've been poking into these types of algorithms for
years. :-)

> te-express-path.

[Alia] Before (hopefully) I republish it, do you have concerns for
draft-atlas-mpls-te-express-path other than
those addressed by the changed paragraphs in the introduction?

Thanks,
Alia

---

From: "Mcdysan, David E" <[email-masked]>
Date: Mon, 25 Feb 2013 09:59:59 -0500

Curtis,

Responses in line.

Dave

>terrible job of providing a text alternative.

I responded to the message in the format which I received it, which was
formatted rather oddly. Replying to this in plain text, so hopefully the
formatting is better.
>draft-fuxh-mpls-delay-loss-te-problem-statement and got no reply.
We are working on a reply.
>draft-fuxh-mpls-delay-loss-* are not related

IMO, the express-path solve some of the problems and meet some of the
requirements of the draft-fuxh-mpls-delay-loss-problem-statement draft,
AND provide a potential foundation for solving some of the the other more
complex problems and meeting future requirements stated in that draft. In
those sense, I believe they are related.

IMO, the express-path drafts provide a reasonable foundation and I am not
advocating delaying advancement of the express-path drafts to wait for the
draft-fuxh-mpls-delay-loss-problem-statement draft to advance or be
updated.

>and I do think those
>draft-atlas-mpls-te-express-path.

I think that we are all in agreement here.
> If there is anything that needs to

This is getting to be a rather long, convoluted email thread, which is
somewhat obnoxious to some people
;-)

In summary, my position on the -express drafts are:
- the express-path drafts provide a reasonable foundation to meet the
requirements draft-fuxh-mpls-delay-loss-problem-statement draft
- draft-fuxh-mpls-delay-loss-problem-statement draft DOES NOT state that
dynamic measurements are used to update advertised performance parameters
(e.g., queuing delay)
- What it does state is a means to configure the IGP performance
parameters. We could add a type parameter to indicate the source/semantics
of these parameters.
- The anamolous links mechanism in mols-express-path should be further
discussed (suggest on a separate thread).

>cover the small metro (links < 20km) and data centers.
Probably need to add half the delay per link.
>offline, but I'd like to be on the Cc.
OK
>node delay to include queuing delay which is just plain wrong.
I agree that node delay != queuing delay.

>You

draft-fuxh-mpls-delay-loss-problemstatement DOES NOT require that queuing
delay be used. IT DOES require that the performance parameter be settable
(see above).

>> > showing how the extended TE metrics can be useful today with minimal
But the IGP drafts become a foundation, or pre-requisite, which I think is
reasonable.
>>reporting

As editor of draft-fuxh-mpls-delay-loss-problemstatement, I tried to write
up the prior text for this "Anomalous State" mechanism, and I too am not
clear as to what value it adds. I can foresee some problems that it could
create.
>> > percentile vs. an average.
Within a single IGP domain is there a need to define the semantics of the
advertised performance parameters? However, when multiple domains and/or
layers are involved, consistent semantics are necessary to compose an
estimate of the end-end performance. Possibly there should be a type value
for the performance parameter. For example, type ={L1 measurement,
configured value, measurement xyz, estimate abc, unknown, etc.}?

>><[email-masked]
>>utlook.com>
>> >> Alia Atlas writes:
>> >>
>> >> Curtis,
>> >>