Subject: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

From: Christian Hopps <[email-masked]>
Date: Thu, 14 Feb 2013 07:35:55 -0500

There was much discussion @ IETF-85 regarding the ISIS TE Metric Extension draft. The plan was to continue the discussion on the list; however, it appears that this did not transpire. In an attempt to kickstart the tabled discussion prior to Orlando, I've CC'd everyone that came to the mic during that session as well as the draft authors. Can everyone please have a look at the minutes, and if you're question still stands can you please re-ask it here so that we can get the authors of the draft involved in answering them?

The Minutes: http://www.ietf.org/proceedings/85/minutes/minutes-85-isis
The Draft: http://tools.ietf.org/html/draft-previdi-isis-te-metric-extensions-02

A companion draft to this one has been advancing in OSPF and is a WG document at this point.

Thanks,
Chris.

---

From: John E Drake <[email-masked]>
Date: Thu, 14 Feb 2013 17:58:04 +0000

Hi,

I read the minutes and I think the subject draft addresses the question that were asked at IETF-85.  So, before asking the their questions again, would people actually read the draft?

Irrespectively Yours,

John

> -----Original Message-----
> From: Christian Hopps [mailto:[email-masked]]
> Sent: Thursday, February 14, 2013 4:36 AM
> To: [email-masked] list
> Cc: Christian Hopps; Himanshu Shah; Acee Lindem; Donald Fedyk; Curtis
> Villamizar; [email-masked]; Bitar, Nabil N; David
> Lamparter; [email-masked]; David D D Ward; George Swallow;
> raszuk; Stefano Previdi; [email-masked]; Lou Berger; John E
> Drake; Alia Atlas; cfilsfil (cfilsfil);
> [email-masked]
> Subject: ISIS TE Metric Extension Discussion (IETF-85 Followup)
> 
> There was much discussion @ IETF-85 regarding the ISIS TE Metric
> Extension draft. The plan was to continue the discussion on the list;
> however, it appears that this did not transpire. In an attempt to
> kickstart the tabled discussion prior to Orlando, I've CC'd everyone
> that came to the mic during that session as well as the draft authors.
> Can everyone please have a look at the minutes, and if you're question
> still stands can you please re-ask it here so that we can get the
> authors of the draft involved in answering them?
>

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Thu, 14 Feb 2013 22:28:51 +0000

Hi John

Yes I had time to read the draft.  No it does not answer all of my questions :-)
  
To me this seems like putting statistics in routing (yes there is a section on rolling delays and throttling but there is measuring and distribution of delay values.) 
When I think about how this could be used at granularity collected (measured on some unspecified class of traffic),  I don't understand the usefulness of this versus just a static propagation delay or delay metric.  The delay variation seems to be aimed at some form of utilization/performance criteria. We have the past reject adaptive routing features.

Summary

I'm a strong advocate of delay metrics focused on minimizing propagation delay.  

I don't believe in putting statistics in the routing database. I think that local storage and some form of off-line collection is better than putting it in the routing system.  And I think the delay statistics should be per traffic class to be useful. 

It seems to me this draft is trying to walk between a delay metric and per class delay statistics.

So I think more clarity around these points would be useful,

Thanks
Don  

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Thursday, February 14, 2013 12:58 PM
To: Christian Hopps; [email-masked] list
Cc: Himanshu Shah; Acee Lindem; Fedyk, Donald (Don); Curtis Villamizar; [email-masked]; Bitar, Nabil N; David Lamparter; [email-masked]; David D D Ward; George Swallow; raszuk; Stefano Previdi; [email-masked]; Lou Berger; Alia Atlas; cfilsfil (cfilsfil); [email-masked]
Subject: RE: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Hi,

Irrespectively Yours,

John

> raszuk; Stefano Previdi; [email-masked]; Lou Berger; John 
> E Drake; Alia Atlas; cfilsfil (cfilsfil); 
> The Minutes: 
> http://www.ietf.org/proceedings/85/minutes/minutes-85-isis

---

From: John E Drake <[email-masked]>
Date: Thu, 14 Feb 2013 23:02:05 +0000

Don,

What is being advertised is performance related information associated with the link itself and not with the queuing system feeding it, so I don't think traffic class applies.  If this isn't clear enough, we can update both drafts (IS-IS & OSPF) to make this clearer.

According to Spencer (and please chime in), for the networks which he operates the link performance characteristics can change dramatically and suddenly, so static values are not particularly useful.  

Irrespectively Yours,

John

> From: Fedyk, Donald (Don) [mailto:[email-masked]]
> Sent: Thursday, February 14, 2013 2:29 PM
> To: John E Drake; Christian Hopps; [email-masked] list
> Cc: Himanshu Shah; Acee Lindem; Curtis Villamizar;
> [email-masked]; Bitar, Nabil N; David Lamparter;
> [email-masked]; David D D Ward; George Swallow; raszuk; Stefano
> Previdi; [email-masked]; Lou Berger; Alia Atlas; cfilsfil
> (cfilsfil); [email-masked]
> To me this seems like putting statistics in routing (yes there is a
> section on rolling delays and throttling but there is measuring and
> distribution of delay values.) When I think about how this could be
> used at granularity collected (measured on some unspecified class of
> traffic),  I don't understand the usefulness of this versus just a
> static propagation delay or delay metric.  The delay variation seems to
> be aimed at some form of utilization/performance criteria. We have the
> past reject adaptive routing features.
> 
> Summary
> 
> I don't believe in putting statistics in the routing database. I think
> that local storage and some form of off-line collection is better than
> putting it in the routing system.  And I think the delay statistics
> should be per traffic class to be useful.
> 
> Cc: Himanshu Shah; Acee Lindem; Fedyk, Donald (Don); Curtis Villamizar;
> [email-masked]; Bitar, Nabil N; David Lamparter;
> [email-masked]; David D D Ward; George Swallow; raszuk; Stefano
> Previdi; [email-masked]; Lou Berger; Alia Atlas; cfilsfil
> (cfilsfil); [email-masked]
> I read the minutes and I think the subject draft addresses the question
> that were asked at IETF-85.  So, before asking the their questions
> again, would people actually read the draft?
> 
> Irrespectively Yours,
> 
> John
>

---

From: Lou Berger <[email-masked]>
Date: Fri, 15 Feb 2013 09:05:36 -0500

Spencer,

On 2/14/2013 7:19 PM, you wrote:
> The use case is my network, where I want to setup TE tunnels based on
> performance, not cost, and I want to re-groom when changes occur,
> such as when optical path protect kicks in underneath me.
>

...

> To your comment about adaptive routing, if you read the draft, it's
> clear we are aiming at TE tunnel setup/comp, not IGP manipulation
> 

I think you're providing critical context that some may have missed
during the presentation.  A slide or two on use case (rather then just
the generic nature of the extension) probably would have helped...

WRT the draft, I think referencing the related drafts
(draft-fuxh-mpls-delay-loss-te-problem-statement and/or
draft-fuxh-mpls-delay-loss-te-framework) would provide the needed
pointer for those not familiar with the context.

Lou

---

From: Lou Berger <[email-masked]>
Date: Fri, 15 Feb 2013 10:29:47 -0500

Spencer,

On 2/15/2013 9:45 AM, [email-masked] wrote:
> Hi Lou, 
> 
> Here is a link to the preso in case anyone missed it. I can always add more of my personal context; At the time, I didn't think it was appropriate to include, but happy to change that. 
> 
> http://www.ietf.org/proceedings/80/slides/rtgwg-1.pdf

The ISIS presentation I was referring to was
http://tools.ietf.org/agenda/85/slides/slides-85-isis-1.pdf

> 
> We are really trying to look at the Fuxh drafts separately from the Metric Extension work, but happy to discuss the linkages and better wording if people have ideas. 
> 

Fair enough, but without context, it sounds like this aim of the
protocol extension is delay based (IP) routing which is currently
considered heresy by most given the ARPANET experience*.

(*) google "The revised ARPANET routing metric" (from SIGCOMM '89) for a
good description of this bit of history...

Lou

> -S
> From: Lou Berger [mailto:[email-masked]] 
> Sent: Friday, February 15, 2013 9:06 AM
> To: Giacalone, Spencer (Financial&Risk)
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
> Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)
> 
> Spencer,
>

---

From: Alia Atlas <[email-masked]>
Date: Fri, 15 Feb 2013 12:53:53 -0500

Should we add a reference to draft-atlas-mpls-te-express-path to help
clarify that this is explicitly NOT delay-based IP routing?

The Fuxh drafts were trying to go closer to that (and worry about
queue-delay) last time I looked - certainly closer than these drafts are.

One can picture other things potentially to do with the information beyond
TE - but any such ideas are out of scope and not defined...

Alia

On Fri, Feb 15, 2013 at 10:29 AM, Lou Berger <[email-masked]> wrote:

> > Here is a link to the preso in case anyone missed it. I can always add
> more of my personal context; At the time, I didn't think it was appropriate
> to include, but happy to change that.
> >
> > We are really trying to look at the Fuxh drafts separately from the
> Metric Extension work, but happy to discuss the linkages and better wording
> if people have ideas.
> >
>
> > Cc: [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked];
> [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> _______________________________________________
> Isis-wg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/isis-wg
>

---

From: Edward Crabbe <[email-masked]>
Date: Fri, 15 Feb 2013 10:33:18 -0800

Alia;

I hate to disagree with you, but this does clearly provide a method for
making path decisions based on delay metrics (which I believe is what Lou
was alluding to, whether the decision occurs at L3 on a hop by hop basis or
not.)

---

From: Alia Atlas <[email-masked]>
Date: Fri, 15 Feb 2013 13:40:43 -0500

Ed,

Sure - the information is provided - but not considering queue length and
in a slow averaged fashion.
The intended usage is for TE, as described in the
draft-atlas-mpls-te-express-path.
There is no intended suggestion to use the delay information for basic IP
routing - and the draft doesn't
specify how one would - at the least capabilities and so on would be needed
for a reasonable distributed computation!

The concern with delay-based is the "oxygen rushing to one corner of the
room" issue - if all the routers simultaneously
compute and start shipping traffic on the lowest-delay path.   Using TE,
there is not the same simultaneous computation
and with bandwidth reservation, suggested changes can be punted.  Also, the
ability to optimally compute a path based on
multiple constraints is NP-hard; the draft just suggests using the delay
information to remove link while doing a standard CSPF -
just like a link without a needed administrative color would be pruned.

Alia

On Fri, Feb 15, 2013 at 1:33 PM, Edward Crabbe <[email-masked]> wrote:

>> One can picture other things potentially to do with the information
>> beyond TE - but any such ideas are out of scope and not defined...
>>
>> Alia
>>

---

From: Lou Berger <[email-masked]>
Date: Fri, 15 Feb 2013 13:47:33 -0500

Seems reasonable to me.  The key is provide the (uninformed) reader with
a pointer to the context.  Perhaps also include an applicability note
that narrows the scope too.

Lou

On 2/15/2013 12:50 PM, Alia Atlas wrote:
> Should we add a reference to draft-atlas-mpls-te-express-path to help clarify that this is explicitly NOT delay-based IP routing?
> 
> The Fuxh drafts were trying to go closer to that (and worry about queue-delay) last time I looked - certainly closer than these drafts are.
> 
> Alia
> 
> Sent: Friday, February 15, 2013 10:30 AM
> To: [email-masked]
> Cc: John E Drake; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Alia Atlas; [email-masked]
> The ISIS presentation I was referring to was http://tools.ietf.org/agenda/85/slides/slides-85-isis-1.pdf
> 
>>
> Fair enough, but without context, it sounds like this aim of the protocol extension is delay based (IP) routing which is currently considered heresy by most given the ARPANET experience*.
> 
> (*) google "The revised ARPANET routing metric" (from SIGCOMM '89) for a good description of this bit of history...
> 
> Lou
> 
>> -S
>>
>>
>>
>>
>>

---

From: Alia Atlas <[email-masked]>
Date: Fri, 15 Feb 2013 13:49:00 -0500

That seems reasonable to me.  Do you have any suggestions on wording or
shall I wing it?

Alia

On Fri, Feb 15, 2013 at 1:47 PM, Lou Berger <[email-masked]> wrote:

> > Cc: John E Drake; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]; [email-masked];
> [email-masked]; Alia Atlas; [email-masked]

---

From: Lou Berger <[email-masked]>
Date: Fri, 15 Feb 2013 13:54:53 -0500

Ed,
	I really was referring to (avoiding) delay based SPF algorithms in hop
by hop routing.  I'm open to (at least, to discussing) the use of delay
in TE path selection, or even as a TE reroute trigger.

Lou

On 2/15/2013 1:33 PM, Edward Crabbe wrote:
> making path decisions based on delay metrics (which I believe is what
> Lou was alluding to, whether the decision occurs at L3 on a hop by hop
> basis or not.)  
> 
> 
> 
> On Fri, Feb 15, 2013 at 9:53 AM, Alia Atlas <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Should we add a reference to draft-atlas-mpls-te-express-path to
>     help clarify that this is explicitly NOT delay-based IP routing?
> 
>     On Fri, Feb 15, 2013 at 10:29 AM, Lou Berger <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>         Spencer,
> 
>         On 2/15/2013 9:45 AM, [email-masked]
>         <mailto:[email-masked]> wrote:
>         > Hi Lou,
>         >
>         > Here is a link to the preso in case anyone missed it. I can
>         always add more of my personal context; At the time, I didn't
>         think it was appropriate to include, but happy to change that.
>         >
>         > We are really trying to look at the Fuxh drafts separately
>         from the Metric Extension work, but happy to discuss the
>         linkages and better wording if people have ideas.
>         >
> 
>         > From: Lou Berger [mailto:[email-masked]
>         <mailto:[email-masked]>]
>         > Cc: [email-masked] <mailto:[email-masked]>;
>         [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>;
>         [email-masked] <mailto:[email-masked]>;
>         [email-masked] <mailto:[email-masked]>;
>         [email-masked] <mailto:[email-masked]>;
>         [email-masked] <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>; [email-masked]
>         <mailto:[email-masked]>
>         [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Edward Crabbe <[email-masked]>
Date: Fri, 15 Feb 2013 11:09:24 -0800

I understand and am well aware of the concern.

My point is simply that Lou's comment re: arpanet is applicable to path
selection based on performance metrics in general, not just to hop by hop
routing.  There is no guarantee w/ cspf that you will not end up with the
same situation, it simply provides you with some safeguards in terms of
both the rate at which traffic is moved (the length of the asynchronous TE
cycle across participating LERs) and the degree of protection you have on a
per link basis (in terms of reservable bw, modulo transport protocol
elasticity.)  RFC3209's bandwidth increase procedure creates a number of
situations in which either the signalled bw or the measurement of demand
may be inaccurate under contention, especially when using autobw (although
 the proposed Unidirectional Residual Bandwidth Sub-TLV takes care of a
subset of these, which I am a big fan of).

To be clear, I'm not against delay based computations, or at least, less so
then many in the community.  I've been in a few environments now where they
have been used (with adequate safeguards) to good effect.  I *do* think
that this needs to be addressed directly in the draft and  am a bit leary
of the statistics that will now be floating around in the IGP in terms of
both flux and total state, at least for larger environments.

---

As an aside on path computation using multiple constraints;  a number of
environments I'm aware of use multiple constraints in their path
computations.  These are fairly large deployments involve significant
numbers of logical tunnels and endpoints.  While it's certainly true that
path splitting constraints force the use of MIPs and ILPs that are formally
intractable, there are many ways of working around this.

On Fri, Feb 15, 2013 at 10:40 AM, Alia Atlas <[email-masked]> wrote:

> specify how one would - at the least capabilities and so on would be
> needed for a reasonable distributed computation!
>
> and with bandwidth reservation, suggested changes can be punted.  Also,
> the ability to optimally compute a path based on
>>>> > Here is a link to the preso in case anyone missed it. I can always
>>>> add more of my personal context; At the time, I didn't think it was
>>>> appropriate to include, but happy to change that.
>>>> >

---

From: Lou Berger <[email-masked]>
Date: Fri, 15 Feb 2013 14:36:13 -0500

Your call...

Thanks,
Lou

On 2/15/2013 1:49 PM, Alia Atlas wrote:
> On Fri, Feb 15, 2013 at 1:47 PM, Lou Berger <[email-masked]
>     > From: Lou Berger [mailto:[email-masked] <mailto:[email-masked]>]
>     <mailto:[email-masked]>
>     > Cc: John E Drake; [email-masked]
>     <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>; Alia Atlas; [email-masked]
>     always add more of my personal context; At the time, I didn't think
>     it was appropriate to include, but happy to change that.
>     >>
>     >> We are really trying to look at the Fuxh drafts separately from
>     the Metric Extension work, but happy to discuss the linkages and
>     better wording if people have ideas.
>     >>
>     >
>     > (*) google "The revised ARPANET routing metric" (from SIGCOMM '89)
>     for a good description of this bit of history...
>     >
>     > Lou
>     >
>     >> -S
>     >>
>     >>
>     >>
>     >>
>     >>
>     >> From: Lou Berger [mailto:[email-masked] <mailto:[email-masked]>]
>     <mailto:[email-masked]>;
>     >> [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>;
>     >> [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     >> [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>;
>     >> [email-masked] <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>;
>     >> [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>; [email-masked]

---

From: Lou Berger <[email-masked]>
Date: Fri, 15 Feb 2013 14:50:01 -0500

Ed,
	I think you're making a slightly different comment than mine, but it's
a good one, and I'm also in complete agreement with its gist.

Lou

On 2/15/2013 2:09 PM, Edward Crabbe wrote:
> selection based on performance metrics in general, not just to hop by
> hop routing.  There is no guarantee w/ cspf that you will not end up
> with the same situation, it simply provides you with some safeguards in
> terms of both the rate at which traffic is moved (the length of the
> asynchronous TE cycle across participating LERs) and the degree of
> protection you have on a per link basis (in terms of reservable bw,
> modulo transport protocol elasticity.)  RFC3209's bandwidth increase
> procedure creates a number of situations in which either the signalled
> bw or the measurement of demand may be inaccurate under contention,
> especially when using autobw (although  the proposed Unidirectional
> Residual Bandwidth Sub-TLV takes care of a subset of these, which I am a
> big fan of). 
> 
> To be clear, I'm not against delay based computations, or at least, less
> so then many in the community.  I've been in a few environments now
> where they have been used (with adequate safeguards) to good effect.  I
> *do* think that this needs to be addressed directly in the draft and  am
> a bit leary of the statistics that will now be floating around in the
> IGP in terms of both flux and total state, at least for larger
> environments.  
> 
> ---
> 
> numbers of logical tunnels and endpoints.  While it's certainly true
> that path splitting constraints force the use of MIPs and ILPs that are
> formally intractable, there are many ways of working around this.    
> 
> 
> On Fri, Feb 15, 2013 at 10:40 AM, Alia Atlas <[email-masked]
>     Sure - the information is provided - but not considering queue
>     length and in a slow averaged fashion.
>     There is no intended suggestion to use the delay information for
>     basic IP routing - and the draft doesn't
>     The concern with delay-based is the "oxygen rushing to one corner of
>     the room" issue - if all the routers simultaneously
>     compute and start shipping traffic on the lowest-delay path.   Using
>     TE, there is not the same simultaneous computation
>     and with bandwidth reservation, suggested changes can be punted.
>      Also, the ability to optimally compute a path based on
>     multiple constraints is NP-hard; the draft just suggests using the
>     delay information to remove link while doing a standard CSPF -
>     On Fri, Feb 15, 2013 at 1:33 PM, Edward Crabbe <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>         Alia;  
> 
>         I hate to disagree with you, but this does clearly provide a
>         method for making path decisions based on delay metrics (which I
>         believe is what Lou was alluding to, whether the decision occurs
>         at L3 on a hop by hop basis or not.)  
> 
> 
> 
>             Should we add a reference to
>             draft-atlas-mpls-te-express-path to help clarify that this
>             is explicitly NOT delay-based IP routing?
> 
>             The Fuxh drafts were trying to go closer to that (and worry
>             about queue-delay) last time I looked - certainly closer
>             than these drafts are.
> 
>             One can picture other things potentially to do with the
>             information beyond TE - but any such ideas are out of scope
>             and not defined...
> 
>             Alia
> 
> 
>             On Fri, Feb 15, 2013 at 10:29 AM, Lou Berger
>             <[email-masked] <mailto:[email-masked]>> wrote:
> 
>                 Spencer,
> 
>                 On 2/15/2013 9:45 AM,
>                 > Here is a link to the preso in case anyone missed it.
>                 I can always add more of my personal context; At the
>                 time, I didn't think it was appropriate to include, but
>                 happy to change that.
>                 >
>                 > We are really trying to look at the Fuxh drafts
>                 separately from the Metric Extension work, but happy to
>                 discuss the linkages and better wording if people have
>                 ideas.
>                 >
> 
>                 [email-masked] <mailto:[email-masked]>;
>                 [email-masked] <mailto:[email-masked]>;
>                 [email-masked]
>                 [email-masked]
>                 <mailto:[email-masked]>; [email-masked]
>                 <mailto:[email-masked]>; [email-masked]

---

From: <[email-masked]>
Date: Fri, 15 Feb 2013 00:19:11 +0000

>"I don't understand the usefulness of this versus just a static propagation delay or delay metric."

The use case is my network, where I want to setup TE tunnels based on performance, not cost, and I want to re-groom when changes occur, such as when optical path protect kicks in underneath me. Yes, I could theoretically manually set cost based on actual performance statically, but it's a huge pain, gets outdated, and biases my other traffic. We get bogged down building, pre-proving and monitoring a lot of TE tunnels for performance. Unlike most SPs, my SLAs are based on load delivered within certain strict latencies. It's important to note that (and I think the draft says or said this) I have a big network where I don't own the underlying transport. This technology would also free me up to use different types of underlying transport (We won't for example use packet based transport) as well as not worrying about things like path protect events, which we are often not notified of. Re-grooming a bunch of market data TE tunnels after an undersea link failure sends my traffic an extra 3000 miles is not fun. Yes, we could (and do) use TE management tools, but how do you think they learn the best performance paths?  

To your comment about adaptive routing, if you read the draft, it's clear we are aiming at TE tunnel setup/comp, not IGP manipulation (though we don't explicitly deny an implementation from doing that). 

I believe most of this is mentioned in the draft, as John says, and it was also generally covered in the presentation I gave when we intro'd OSPF Express Path, which you can find this online. 

I think we should be past use-case at this point. 

On the other hand, queue delay is too transient IMHO. 

Does this help?

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Thursday, February 14, 2013 6:02 PM
To: Fedyk, Donald (Don); Christian Hopps; [email-masked] list
Cc: Himanshu Shah; Acee Lindem; Curtis Villamizar; [email-masked]; Bitar, Nabil N; David Lamparter; [email-masked]; David D D Ward; George Swallow; raszuk; Stefano Previdi; [email-masked]; Lou Berger; Alia Atlas; cfilsfil (cfilsfil); Giacalone, Spencer (Financial&Risk)
Subject: RE: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Don,

Irrespectively Yours,

John

---

From: <[email-masked]>
Date: Fri, 15 Feb 2013 14:45:43 +0000

Hi Lou, 

-S

-----Original Message-----
From: Lou Berger [mailto:[email-masked]] 
Sent: Friday, February 15, 2013 9:06 AM
To: Giacalone, Spencer (Financial&Risk)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Spencer,

...

Lou

---

From: <[email-masked]>
Date: Fri, 15 Feb 2013 18:22:32 +0000

If that's the concern, yes, it should really be cleared up in the "application" drafts that make use of the data we send in the Extensions draft. As has been said many times, nothing in the Extensions draft specifies how the data is used, though we do infer. Personally, I thought your draft was reasonably clear, but then again I may be too close to it to see the opposite.

From: Alia Atlas [mailto:[email-masked]]
Sent: Friday, February 15, 2013 12:54 PM
To: Lou Berger
Cc: Giacalone, Spencer (Financial&Risk); [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

One can picture other things potentially to do with the information beyond TE - but any such ideas are out of scope and not defined...

Alia

On Fri, Feb 15, 2013 at 10:29 AM, Lou Berger <[email-masked]<mailto:[email-masked]>> wrote:
Spencer,

Lou

> From: Lou Berger [mailto:[email-masked]<mailto:[email-masked]>]
> Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
_______________________________________________
Isis-wg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/isis-wg

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Tue, 19 Feb 2013 00:35:33 +0000

Hi Spencer

Thanks for all the pointers (and reading material). I agree there is a problem to solve with respect to delay ( Several of us tried to put something in many moons ago http://tools.ietf.org/html/draft-fedyk-isis-ospf-te-metrics-01). 

The logic behind our draft was based on experience with class of service routing from the 90s. 
A typical minimization criteria for an MPLS path is minimum hops subject to bandwidth X. But that assumes the delay of the links or hops is roughly equal.  So we wanted to allow minimize delay (or other) subject to bandwidth X. 

When we analyzed delay we had to come up with a minimization criteria for delay.
The problem is delay is transmission delay + propagation delay primarily. 

People have long since figured out how to measure delay. (however really accurate delay is not trivial.)  
But let's Look at draft-previdi just the delay part:  
One way delay in microseconds is a number. 1usec to 16.777215 sec.

Propagation delay of a 10Mb/s and a 10Gb/s link is roughly the same for the same distance.
But Transmission delay is 4 orders of magnitude different. 80us to less than a micro second for 100 bytes.

Therefore what we proposed was a normalized number has to be put in for the delay metric (Note I differentiate a metric as a quantity versus a measured value or a statistic (measured at an interval) although some people call characteristics such delay, delay variation etc, metrics). We actually never specified the metric values but proposed that there should be a formula for a given network.  The idea was to use a  delay metric based on given packet size. That way as speeds increase you could get some differentiation between the higher speeds if you wanted. 

I would say you could measure delay point to point at a resolution but what you advertise as a metric is a value that can be scaled and possible adjusted in the future when the dynamic range of the value need to be adjusted. (scaling is reported once per node.) 
Also you could configure values for links that don't have the capability. Depending on the goal you should think about how you could scale the metric otherwise it may be useless. (Even if you are just going to decide to change it to 100s of nanoseconds from microseconds.) 
This is a way I can see addressing the very specialized "low delay" arbitrage networks. But perhaps we should not encourage that behavior anyway ;-)   

When reporting a single value for TE links there are issues with various link models when the link is not singular. Reserved bandwidth can be reported whether there is 1 or more links in a link group between points just by reporting the maximum/priority. The delay is per link and if the links are all the same (or nearly) that is ideal but if not then the delay metric should be per link or it should be based on the link that is reporting the free bandwidth in the TE BW profile but you cannot just report a minimum. 

Another point is based on the problem description in your presentation what needs to be monitored is the performance characteristics of the MPLS path end to end. That is based on two unidirectional and not necessarily congruent paths. Paths that exceed their profile should request to be rerouted.  This can happen even if no link changes its delay.

Therefore you have at least two monitoring points that need to be correlated sometimes: paths to move and links to be avoided.   

Let's take your example of a resource underneath you changed its delay, assuming the link stays up. How do you correlate the two events that path performance changed and link delay changed?  The best way to do that IMHO was with another mechanism we proposed at the time,  feedback.  Many of the concepts got pulled into RFC4920 where crank back procedures could be used.  You have the issue that when performance degrades how do you figure out what links have changed? The Brute force way is to have links update the delay metric as proposed in draft-previdi (etc),  distribute that and then have some correlation functions that can match established paths with resources that have changed. But you either have the paths hunting for changes on routing or routing scanning significant changes and looking of paths.  

The other way to do it is to have paths register for significant events on the transit nodes and have notification messages propagate along the paths to the head-end. The paths know which resource to exclude without having to wait for routing. They could also know locally if the performance change is significant to that path. That reduces the routing burden significantly. And if a change is noted on the end to end path monitoring but there is no signal?  Then you can just move the path in question after some delay by excluding the existing path. 

What's the bottom line? 
Measure what you want point to point. Other offline systems can be used to tune this.
Put only normalized and significant values in routing metrics. An agreed formula for delay is a good idea IMHO.  
Allow for configured values to ease this into systems that might not have the measurement. 
Target the significant changes at the paths that require the change reducing race conditions.  

Don 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, February 14, 2013 7:19 PM
To: [email-masked]; Fedyk, Donald (Don); [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: RE: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Does this help?

-----Original Message-----
From: John E Drake [mailto:[email-masked]]
Sent: Thursday, February 14, 2013 6:02 PM
To: Fedyk, Donald (Don); Christian Hopps; [email-masked] list
Cc: Himanshu Shah; Acee Lindem; Curtis Villamizar; [email-masked]; Bitar, Nabil N; David Lamparter; [email-masked]; David D D Ward; George Swallow; raszuk; Stefano Previdi; [email-masked]; Lou Berger; Alia Atlas; cfilsfil (cfilsfil); Giacalone, Spencer (Financial&Risk)
Subject: RE: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Don,

Irrespectively Yours,

John

> [email-masked]; David D D Ward; George Swallow; raszuk; 
> Stefano Previdi; [email-masked]; Lou Berger; Alia Atlas; 
> cfilsfil (cfilsfil); [email-masked]
> static propagation delay or delay metric.  The delay variation seems 
> to be aimed at some form of utilization/performance criteria. We have 
> the past reject adaptive routing features.
> 
> Summary
> 
> Cc: Himanshu Shah; Acee Lindem; Fedyk, Donald (Don); Curtis 
> raszuk; Stefano Previdi; [email-masked]; Lou Berger; Alia 
> Atlas; cfilsfil (cfilsfil); [email-masked]
> I read the minutes and I think the subject draft addresses the 
> question that were asked at IETF-85.  So, before asking the their 
> questions again, would people actually read the draft?
> 
> Irrespectively Yours,
> 
> John
> 
> > Cc: Christian Hopps; Himanshu Shah; Acee Lindem; Donald Fedyk; 
> > Curtis Villamizar; [email-masked]; Bitar, Nabil N; 
> > David Lamparter; [email-masked]; David D D Ward; George 
> > Swallow; raszuk; Stefano Previdi; [email-masked]; Lou 
> > Berger; John E Drake; Alia Atlas; cfilsfil (cfilsfil); 
> > Extension draft. The plan was to continue the discussion on the 
> > list; however, it appears that this did not transpire. In an attempt 
> > to kickstart the tabled discussion prior to Orlando, I've CC'd 
> > everyone that came to the mic during that session as well as the 
> > draft
> authors.

---

From: <[email-masked]>
Date: Tue, 19 Feb 2013 12:51:16 +0000

The express path draft was adopted by the opsf wg.

Of course I disagree that we should stop, and as I have elaborated I feel the requirements are clear. 

Further, the relationship between the metric distribution and "application" drafts has been made clear and is pointed out in my preso

----- Original Message -----
From: Curtis Villamizar [mailto:[email-masked]]
Sent: Tuesday, February 19, 2013 12:56 AM
To: Alia Atlas <[email-masked]>
Cc: Lou Berger <[email-masked]>; Giacalone, Spencer (Financial&Risk); John E Drake <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

> queue-delay) last time I looked - certainly closer than these drafts

Alia,

We should make it clear which express-path work is going forward and
whether it has any relationship to draft-ietf-rtgwg-cl-requirements.

Right now we have:

  draft-atlas-mpls-te-express-path

  draft-giacalone-ospf-te-express-path
  (presumed abandonned)

  draft-fuxh-mpls-delay-loss-te-problem-statement and/or
  draft-fuxh-mpls-delay-loss-te-framework

The latter two were presented at the last IETF.

Would the authors of the set of documents above please combine them.
Would they also please consider explicitly addressing the requirments
in draft-ietf-rtgwg-cl-requirements, such as stability, keeping the
amount of churn down, supporting a grouping of CL component links.  If
you do this, would you also please include me in the discussion.

IMO - the ospf and isis work should be halted until there is agreement
on what problem is being solved, and the mechanisms to go about
solving it.  Until then, we have protocol syntax and no semantics, and
no assurance that the current syntax will match the semantics that is
later decided is needed (by way of a framework document).

Curtis

ps - sorry to join the discussion on this list late, but I was out for
a few days last week and over the long weekend (long in the US).

---

From: <[email-masked]>
Date: Tue, 19 Feb 2013 12:58:36 +0000

Pls note that the ospf wg has already adopted express path as ospf metric extensions.

We really need to stop talking about the fuxh drafts. A) we (john, dave, alia, myself) are not primary authors, and b) those drafts are "application" drafts. Pls read my slide deck for more info on that

Even assuming you don't agree with fuxh or Alias draft (which I support) then we do NOT need to hold up metric extensions. The metric extensions drafts support the others, but are not the same thing

----- Original Message -----
From: Curtis Villamizar [mailto:[email-masked]]
Sent: Tuesday, February 19, 2013 12:42 AM
To: Lou Berger <[email-masked]>
Cc: Giacalone, Spencer (Financial&Risk); [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

> pointer for those not familiar with the context.

My point at IETF-85 which still stands are:

  1.  There needs to be a problem statement, requirements, and
      framework providing that context.

  2.  draft-fuxh-mpls-delay-loss-te-problem-statement and/or
      draft-fuxh-mpls-delay-loss-te-framework are candidates but in
      poor shape.  [btw- I emailed the set of authors of
      problem-statement about a week or more ago and got no response.]

  3.  As is the requirements of draft-fuxh-mpls-delay-loss-te-* are
      not met by the isis or ospf te extensions (for example, a
      working and protect relative delay is discussed, which would
      require a min delay and max delay, not just a delay which is
      implied to be min).

  4.  There are Composite Link requirements that should be met.  The
      draft-fuxh-mpls-delay-loss-te-* drafts cite this work but
      address a subset of the issues related to delay and jitter.  For
      example, the topic of stability is completely ignored.

So my points still stand.

It was premature to move the ospf work to wg.  It is premature to move
this to wg.  The same issues need to be addressed in both since they
are functionally identical.

Curtis

---

From: Alia Atlas <[email-masked]>
Date: Tue, 19 Feb 2013 09:11:51 -0500

Hi Curtis & Ed,

First, I agree that the details of using the extended TE metrics for TE
belong in the associated MPLS draft.
I'm happy to update/modify that - and would welcome suggestions, text, etc.

Second, to Ed's point on bw needed by LSP's being different from what's
signaled, the draft, the draft does
have the "available bw" signaled, which is what's left over after TE LSPs
are reserved and measured LDP/IP
traffic is accounted for.  It's not perfect, in that it isn't measuring the
amount of bw used by TE LSPs - but it does
give some idea of how much actual bandwidth is available for new traffic
(whether TE or IP/LDP).

Third, to Curtis' thought on how to handle delay, I agree that it is a
path-based constraint rather than a per-link
constraint, and that is what I specified in Sec 2.1
of draft-atlas-mpls-te-express-path-02.

If you have time to read through that draft and make suggestions, I'd be
quite appreciative.
In the meantime, we need to add a reference to it for the ISIS draft and
some language about the added information
being intended for use by TE and any other application would be "for future
study that is informed by
previous experiences with issues is delay-based hop-by-hop routing".

Regards,
Alia

On Tue, Feb 19, 2013 at 2:16 AM, Curtis Villamizar <[email-masked]> wrote:

>
> In message <CACKN6JEXRSDinWC4FV3=1W4ytZ_Jj0Tm0S65R=
> [email-masked]>
> Edward Crabbe writes:
> >
> so
> I agree that this needs to be addressed.
>
> I just think that in *this* draft is the wrong place.  In one of the
> te-express drafts would be the right place, preferably getting the set
> of authors of the three strains of draft to agree.
>
> > ---
> >
> Optimizing multiple metrics is NP-hard.  Using multiple constraints is
> easy.
>
> The te-express work suggests a per-link constraint like administrative
> attributes which IMO is the wrong way to handle delay.  The right way
> would be to treat delay and jitter as cummulative constraints.  Note
> that jitter expressed to within a less than 100% confidence level may
> not combine linearly.  The CSPF would have to truncate a search down a
> link only after the cummulative constraint was exceeded.  The
> constraint is a delay constraint on the path, not constraints on
> specific links.
>
> ISIS is not the WG to be discussing this on.  It would be RTGWG or
> MPLS.
>
> Curtis
>
> the
> TE,
> delay
> the
> for a

---

From: Acee Lindem <[email-masked]>
Date: Tue, 19 Feb 2013 14:24:25 +0000

Hi Spence, 
Note that the bar for WG last call and request for publication as a standards track RFC is quite a bit higher than for WG adoption (OSPF WG copied). We need to have this discussion now and either address the questions surrounding the delay metrics or agree that they are out of scope. 
Having said that, speaking as a WG member of both WGs, I have re-read the draft and don't have problems with the guidance given for delay metrics when these metrics are used for TE. 
Thanks,
Acee

---

From: John E Drake <[email-masked]>
Date: Tue, 19 Feb 2013 15:34:43 +0000

Curtis,

Comments inline.

Irrespectively Yours,

John

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Monday, February 18, 2013 10:43 PM
To: Lou Berger
Cc: [email-masked]; John E Drake; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Alia Atlas; [email-masked]
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

In message <[email-masked]>
Lou Berger writes:
 
> > The use case is my network, where I want to setup TE tunnels based 
> > on performance, not cost, and I want to re-groom when changes occur, 

JD:  If the WGs (OSPF & IS-IS) want additional text we would be happy to add it.

JD:  Those drafts are addressing a different problem.

JD:  Since those drafts are addressing a different problem, this does not seem to follow.  The anomalous bit addresses your comment about working & protect delay.

JD:  These drafts have nothing to do with Composite Link.  Please feel free to write another draft that describes how these TLVs would be used for Composite Links. 

JD:  Not really.

It was premature to move the ospf work to wg.  It is premature to move this to wg.  The same issues need to be addressed in both since they are functionally identical.

JD:  Thanks for your input as a WG participant.

Curtis

---

From: <[email-masked]>
Date: Tue, 19 Feb 2013 16:47:42 +0000

Hi Acee, 

I know :-)

-----Original Message-----
From: Acee Lindem [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 9:24 AM
To: Giacalone, Spencer (Financial&Risk)
Cc: Curtis Villamizar; Lou Berger; John E Drake; [email-masked]; Chris Hopps; [email-masked]; Himanshu Shah; Samita Chakrabarti; [email-masked]; [email-masked]; Huaimo Chen; [email-masked]; George Swallow; Robert Raszuk; Stefano Previdi (sprevidi); [email-masked]; Alia Atlas; Clarence Filsfils; OSPF List
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Hi Spence,

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 19 Feb 2013 16:49:02 +0000

> Lou

  Makes for spectacular oscillations under congestion .. ;)

  Peter

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 19 Feb 2013 16:54:27 +0000

>in TE path selection, or even as a TE reroute trigger.
                                    -------------------
>Lou

Better make sure its heavily damped .. or you are gonna get/cause oscillations with TE too.

Peter

---

From: <[email-masked]>
Date: Tue, 19 Feb 2013 16:56:54 +0000

Does that not depend on how you configure your switchover thresholds?

-----Original Message-----
From: AshwoodsmithPeter [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 11:54 AM
To: Lou Berger; Edward Crabbe
Cc: Bitar, Nabil N; [email-masked]; [email-masked]; Clarence Filsfils (cfilsfil); [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Alia Atlas; stefano previdi; Giacalone, Spencer (Financial&Risk); [email-masked]; [email-masked]
Subject: RE: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

                                    -------------------

Peter

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 19 Feb 2013 17:44:08 +0000

>  Does that not depend on how you configure your switchover thresholds?

Hey Spencer,

As long as the delay does not change as a result of the paths/traffic you are moving then I suppose it would be stable but if the traffic on the paths you are moving affect the delay that you use for subsequent path placement then I think the best you can do is a slow oscillation? 

Peter

-----Original Message-----
From: AshwoodsmithPeter [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 11:54 AM
To: Lou Berger; Edward Crabbe
Cc: Bitar, Nabil N; [email-masked]; [email-masked]; Clarence Filsfils (cfilsfil); [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Alia Atlas; stefano previdi; Giacalone, Spencer (Financial&Risk); [email-masked]; [email-masked]
Subject: RE: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

                                    -------------------

Peter

---

From: Edward Crabbe <[email-masked]>
Date: Tue, 19 Feb 2013 09:46:37 -0800

Curtis, thanks for the response;  comments inline:

> [email-masked]>

[snip]

> of authors of the three strains of draft to agree.

Yes, apologies for the confusion.  I was responding specifically to Alia's
message and did not differentiate between it and the global scope of the
conversation; my bad.  To be clear  I *do not* think that this exposition
belongs in draft-previdi-isis-te-metric-extensions.  I *do* believe that
this should be addressed in draft-atlas-mpls-te-express-path.

> Optimizing multiple metrics is NP-hard.  Using multiple constraints is

Depends on how you represent your constraints. ;P As you know, there are a
number of algorithms for optimizing multiple metrics in /polynomial time/
(eg, if dealing with linear constraints, Karmarkar's algorithm).
 Unfortunately, we are often forced into using integer methods in TE
optimization.

Much of this is academic in the context of in-band,
forwarding-plane-congruent decisions, and is better left for routing
applications with some degree of centralization.  I'd love to talk more
about it with you, but we should probably move it off-thread.

> ISIS is not the WG to be discussing this on.  It would be RTGWG or

Agree.  Thanks for calling this out.

best,

  -ed

---

From: <[email-masked]>
Date: Tue, 19 Feb 2013 18:09:49 +0000

Of course that's a point, which is also why we measure loss. As background, bandwidth is sort of secondary in many cases. On the other hand, I can't completely control delay and pathing (I can but I use most/all paths depending on what/where we are talking). In other words, bandwidth is an easier fix for me than delay.

-----Original Message-----
From: AshwoodsmithPeter [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 12:44 PM
To: Giacalone, Spencer (Financial&Risk); [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

Hey Spencer,

Peter

-----Original Message-----
From: AshwoodsmithPeter [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 11:54 AM
To: Lou Berger; Edward Crabbe
Cc: Bitar, Nabil N; [email-masked]; [email-masked]; Clarence Filsfils (cfilsfil); [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Alia Atlas; stefano previdi; Giacalone, Spencer (Financial&Risk); [email-masked]; [email-masked]
Subject: RE: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

                                    -------------------

Peter

---

From: John E Drake <[email-masked]>
Date: Tue, 19 Feb 2013 18:58:01 +0000

Peter,

As we have explained before on the list and in the draft, we are advertising a link's propagation delay and not the delay associated with the queuing system that is feeding it.  Given that, I don't understand your comment.

Irrespectively Yours,

John

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of AshwoodsmithPeter
> Sent: Tuesday, February 19, 2013 9:44 AM
> To: [email-masked]; [email-masked];
> [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> Alia Atlas; [email-masked]; [email-masked]
> As long as the delay does not change as a result of the paths/traffic
> you are moving then I suppose it would be stable but if the traffic on
> the paths you are moving affect the delay that you use for subsequent
> path placement then I think the best you can do is a slow oscillation?
> 
> Peter
> 
> Cc: Bitar, Nabil N; [email-masked]; [email-masked]; Clarence
> Filsfils (cfilsfil); [email-masked]; [email-masked];
> [email-masked]; [email-masked];
> [email-masked]; Alia Atlas; stefano previdi; Giacalone,
> Spencer (Financial&Risk); [email-masked]; [email-masked]
> >     queue-delay) last time I looked - certainly closer than these
> drafts
> >     are.
> >
> information
> >         >> The use case is my network, where I want to setup TE
> tunnels
> >         based on
> >         > I think you're providing critical context that some may
> have
> >         missed
> >         > during the presentation.  A slide or two on use case
> (rather
> >         then just

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 19 Feb 2013 20:13:39 +0000

Ok, fair enough, as long as that delay value is stable there is no issue.

Peter

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 1:58 PM
To: AshwoodsmithPeter; [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Alia Atlas; [email-masked]; [email-masked]
Subject: RE: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

Peter,

Irrespectively Yours,

John

---

From: John E Drake <[email-masked]>
Date: Tue, 19 Feb 2013 20:26:30 +0000

Peter,

It's stable until there is a discontinuity, which is why we have the anomalous bit.

Irrespectively Yours,

John

> Sent: Tuesday, February 19, 2013 12:14 PM
> To: John E Drake; [email-masked];
> [email-masked]; [email-masked]
> As we have explained before on the list and in the draft, we are
> advertising a link's propagation delay and not the delay associated
> with the queuing system that is feeding it.  Given that, I don't
> understand your comment.
> 
> Irrespectively Yours,
> 
> John
> 
> 
> > > making path decisions based on delay metrics (which I believe is
> > > what Lou was alluding to, whether the decision occurs at L3 on a
> hop
> > > by
> > hop
> > > basis or not.)
> > >
> > >
> > >
> about
> separately
> > >         Fair enough, but without context, it sounds like this aim
> of
> > the
> > >         (*) google "The revised ARPANET routing metric" (from
> SIGCOMM
> > >         '89) for a
> [email-masked]
> > >         >> performance, not cost, and I want to re-groom when
> > > changes
> > occur,
> > >         > draft-fuxh-mpls-delay-loss-te-framework) would provide
> the
> > needed

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 19 Feb 2013 01:42:53 -0500

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 19 Feb 2013 01:56:57 -0500

Alia,

Right now we have:

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 19 Feb 2013 02:16:13 -0500

In message <CACKN6JEXRSDinWC4FV3=1W4ytZ_Jj0Tm0S65R=[email-masked]>

Curtis

---

From: Alia Atlas <[email-masked]>
Date: Tue, 19 Feb 2013 14:53:31 +0000

Curtis,

I believe that we are emphatically saying that:

are UNCONNECTED to the drafts that we are discussing, which are:

   draft-atlas-mpls-te-express-path
   draft-ietf-ospf-te-metric-extensions
   draft-previdi-isis-te-metric-extensions

They are also not trying to solve the composite-link requirements.

I know that all of these talk about delay and jitter concerns, but the 3 drafts we're discussing are trying to narrow down what to send to a reasonable set that doesn't provoke stability concerns and can be used to solve the types of issues that Spencer described as in the mpls draft.

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 1:57 AM
To: Alia Atlas
Cc: Lou Berger; [email-masked]; John E Drake; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Alia,

We should make it clear which express-path work is going forward and whether it has any relationship to draft-ietf-rtgwg-cl-requirements.

Right now we have:

Would the authors of the set of documents above please combine them.
Would they also please consider explicitly addressing the requirments in draft-ietf-rtgwg-cl-requirements, such as stability, keeping the amount of churn down, supporting a grouping of CL component links.  If you do this, would you also please include me in the discussion.

IMO - the ospf and isis work should be halted until there is agreement on what problem is being solved, and the mechanisms to go about solving it.  Until then, we have protocol syntax and no semantics, and no assurance that the current syntax will match the semantics that is later decided is needed (by way of a framework document).

Curtis

ps - sorry to join the discussion on this list late, but I was out for a few days last week and over the long weekend (long in the US).

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 19 Feb 2013 19:07:28 -0500

In http://datatracker.ietf.org/ a search for express-path yields only
one non-expired draft and that is draft-atlas-mpls-te-express-path.

What draft are you referring to as having been accepted by OSPF.  The
only thing accepted is draft-ietf-ospf-te-metric-extensions.

I did say in another email on this thread that it was a mistake for
the OSPF WG to accept the te-metric-extensions draft without a clear
statement of what the extensions would be used for and repeated that
comment below.

You've stated that you feel the requirements are clear, but the only
requirements are documented in the express-path drafts, none of which
are WG documents.  That is why below, I have suggested that the
authors of the three drafts combine efforts.  I also feel that it
should be a prerequisite that the express-path drafts advance before
the te-metric-extensions drafts in OSPF and ISIS.  Requirements and
framework before protocol extensions.

Curtis

In message <[email-masked]>
[email-masked] writes:

----- Original Message -----
From: Curtis Villamizar [mailto:[email-masked]]
Sent: Tuesday, February 19, 2013 12:56 AM
To: Alia Atlas <[email-masked]>
Cc: Lou Berger <[email-masked]>; Giacalone, Spencer (Financial&Risk); John E Drake <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Alia,

Right now we have:

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 19 Feb 2013 19:14:12 -0500

btw- Please stop top posting.

The metric extensions do not adequately support the requirements being
discussed and there is insufficient consensus on the requirements.
That is the whole point of holding up the metric extensions until
there is agreement on the requirements.

Curtis

In message <[email-masked]>
[email-masked] writes:

----- Original Message -----
From: Curtis Villamizar [mailto:[email-masked]]
Sent: Tuesday, February 19, 2013 12:42 AM
To: Lou Berger <[email-masked]>
Cc: Giacalone, Spencer (Financial&Risk); [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 19 Feb 2013 20:06:02 -0500

You can configure for slow slosh or rapid oscillation.  You can't
entirely damp a set of binary decisions (ie: path A vs path B).

If you set for slow slosh, then reaction to a delay change is also
slow.  One possibility is to detect and adapt to oscillations
(increase the trigger interval if switching back and forth).

Would be nice if we settled on this before protocol specs that say
nothing at all about oscillations.  This should be in express-path,
therefore MPLS WG with a hold on OSPF and ISIS work.

Curtis

In message <[email-masked]>
[email-masked] writes:

-----Original Message-----
From: AshwoodsmithPeter [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2013 11:54 AM
To: Lou Berger; Edward Crabbe
Cc: Bitar, Nabil N; [email-masked]; [email-masked]; Clarence Filsfils (cfilsfil); [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Alia Atlas; stefano previdi; Giacalone, Spencer (Financial&Risk); [email-masked]; [email-masked]
Subject: RE: [Isis-wg] ISIS TE Metric Extension Discussion (IETF-85 Followup)

                                    -------------------

Peter

---

From: Alia Atlas <[email-masked]>
Date: Thu, 21 Feb 2013 15:56:34 -0500

Hi Curtis,

On Tue, Feb 19, 2013 at 7:56 PM, Curtis Villamizar <[email-masked]> wrote:

>
> Alia,
>
> How many different sets of ISIS and OSPF standards for advertising
> delay and jitter should we be aiming for?
>
> I think that the answer should be one.  Not two.  Not three or more.
>

In general, we should avoid duplication.  Where there is strong
disagreement about the requirements and usage, or where there isn't
sufficient maturity on all the aspects, then it makes sense to think about
the problem.

There's also the difference between trying to make a cup of tea and boiling
the ocean.

Finally, it is good to get things done - particularly when drafts are
sitting for 1+ years without comments...

> Therefore it would be beneficial to look at the set of requirements
> and see if using one document as the base - fine with me if
> draft-atlas-mpls-te-express-path is used as the base - and then see if
> adding just a little to that covers CL requirements, and maybe adding
> just a little more covers anything missed in
> draft-fuxh-mpls-delay-loss-* that is deemed important.
>

There are some  requirements in draft-fuxh-mpls-delay-loss which are
concerning, such as the desire to
include node queuing delay and use that as a criteria for paths without
addressing any of the stability concerns
that started this conversation.  I have NO interest in trying to meet that
requirement  without someone doing the necessary
convincing studies, analysis & convincing.  Someone else can define a new
sub-TLV to advertise node delays and related information.   In this case,
IMHO, it's useful to separate out the fairly static link delay from the
traffic-dependent queuing delay.
So - no duplication there.   It needs significant research and analysis to
prove it's useful; I don't see a reason to wait on the
draft-ietf-ospf-te-metric-extensions or
draft-previdi-isis-te-metric-extensions for that.

As for the RSVP-TE path related requirements, these drafts do go much much
farther than draft-atlas-mpls-te-express-path does.   This is the
ocean-boiling (perhaps pond) of handling inter-area, inter-as, and LSPs
without explicit routes completely computed by the ingress.  Those aspects
may be addressing very real use-cases - but that is much more extensive and
complicated than a simple informational draft showing how the extended TE
metrics can be useful today with minimal effort.

> The delay related requirements in CL are spelled out in the CL
> framework.  They are (from 7.1.  Brief Review of Requirements):
>
>    delay and delay variation
>
>        FR#7 (expose lower layer measured delay),
>        FR#8 (precision of latency reporting),
>        FR#9 (limit latency on per LSP basis),
>        FR#15 (minimum delay path),
>        FR#16 (bounded delay path),
>        FR#17 (bounded jitter path)
>
> FR#7 essentially asks for interworking with delay measured at a lower
> layer.  Obviously this would *not* include queuing delay.  Maybe no
> impact on draft-atlas-mpls-te-express-path since it does not specify
> where to get the information.
>

Right, the intent is that the information could come from many places, and
John Drake
can quote the RFC that is a good example.

> FR#8 (in CL requirements) is:
>
>    FR#8    The precision of latency reporting SHOULD be configurable.
>            A reasonable default SHOULD be provided.  Implementations
>            SHOULD support precision of at least 10% of the one way
>            latencies for latency of 1 ms or more.
>
> It seems that the remaining four are covered.  So meeting CL
> requirements may be a NOOP except perhaps FR#8.

If you look at draft-ietf-ospf-te-metric-extensions, the delay can range
from 1 micro-second to
16.777215 seconds.  Do we really need more dynamic range and associated
complexity?

Certainly, if draft-ietf-ospf-te-metric-extensions and
draft-previdi-isis-te-metric-extensions meet
the composite-link requirements, that is useful; it's always nice to solve
multiple problems at the
same time.

> Some critical questions which I don't see answered in any draft, are
> 1) whether queuing delay is measured, 2) if so, can measurements at
> different PSC (direct mapping of TC bits to PHB for E-LSP) be handled,
> 3) if so, how are different PSC handled.  It would be nice if the
> precision of the delay measurement was reported, for example reporting
> a range for the link (ie: 10-12 msec, rounded to 1 msec).  If delay
> was 9.8-11.5 for an interval and 10.2-12.2 the next, it need not be
> readvertised but its nice to know through the reported precision that
> it could be in the range of 9.5-12.499 and still count as 10-12.
>

(1),(2), and (3) It's not measuring queuing delay.  I'll work on improved
text to clarify this.

> Then there is the greater issue of measurement interval and
> confidence.  Is the reported value the lowest reading over the
> interval, highest, or some statistical norm?  Is it a mean value over
> the interval? 98th percentile?  Is it necessary to communicate the
> measurement interval?  For example, would a NPO be satisfied for delay
> if the measurement was made over the previous hour and any changes
> would be reflected in the next measurement an hour later?  My
> suggestion was a range and precision.
>

The draft states that it should be an average.  Why would this information
WANT to be
communicated?  It seems the type of thing that would be documented - perhaps
configurable on a router, but I can't see a reason for signaling it.

 I can see that there might be an argument for using a 98th percentile vs.
an average.

> I don't think you can just punt on oscillations and stability by
> saying for further study.
>

Why not?  What is being measured and reported does not change as a result of
traffic load.  It's based on characteristics of the link and not queuing.
 Perhaps one should
put in some words about future study for stability for virtual links, whose
measurements may
include queuing delay in them?  Perhaps we should should say that these
values shouldn't
be advertised for such links?  Or have a way of measuring them that assumes
maximum
queuing delay at each hop on the virtual link's path?

> As for the draft-fuxh- set, I think the only requirement not met is
> the delay difference between working and protect paths.  For that you
> need something along the lines of a min delay and max delay.  Where
> the working and protect are expected to each fall within a range, such
> that this requirement is met.  Note that the mpls-tp-use-cases draft
> has the same exaple.  The authors of the draft-fuxh- set can correct
> me if I missed anything.
>

Again, the draft-fuxh-set is trying to do much more complicated stuff for
computation;
I can see those authors writing a suitable draft to explain how to compute
LSPs
between a min delay and a max delay.

>
> If you don't mind, I think we should take this thread to MPLS.  I
> copied MPLS and bcc'd ISIS.  This way ISIS will get this message and
> know the conversation moved, but the conversation continues on MPLS.
> On the MPLS list I'll make more detailed comments on a new thread.
>

I've included ISIS back as well, because the concerns were raised there
about
draft-previdi-isis-te-metric-extensions and the application and usage
discussed
is important for ISIS to draw conclusions.

Alia

>
> Curtis
>
>
> In message <
> [email-masked]
> >
> Alia Atlas writes:
>
> Curtis,
>
> I know that all of these talk about delay and jitter concerns, but the 3
> drafts we're discussing are trying to narrow down what to send to a
> reasonable set that doesn't provoke stability concerns and can be used to
> solve the types of issues that Spencer described as in the mpls draft.
>
> Alia
>
>
> Cc: Lou Berger; [email-masked]; John E Drake;
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> In message <
> [email-masked]
> >
> Alia Atlas writes:
>
> Would they also please consider explicitly addressing the requirments in
> draft-ietf-rtgwg-cl-requirements, such as stability, keeping the amount of
> churn down, supporting a grouping of CL component links.  If you do this,
> would you also please include me in the discussion.
>
> IMO - the ospf and isis work should be halted until there is agreement on
> what problem is being solved, and the mechanisms to go about solving it.
>  Until then, we have protocol syntax and no semantics, and no assurance
> that the current syntax will match the semantics that is later decided is
> needed (by way of a framework document).
>
> Curtis
>
> ps - sorry to join the discussion on this list late, but I was out for a
> few days last week and over the long weekend (long in the US).
>
>
>
> mpls mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mpls
>

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 21 Feb 2013 22:51:52 -0500

> Hi Curtis,

Hi Alia,

> sufficient maturity on all the aspects, then it makes sense to think
> about the problem.

OK.  Consolidation is good.

> There's also the difference between trying to make a cup of tea and
> boiling the ocean.

I agree here too.  If you look at the thread with John I am asking for
relatively little changes.  I have yet to send a full review of the
document.  We are discussion issues at the moment and where things in
the document are underspecified or unspecified, the discussion can be
more open ended.

> sitting for 1+ years without comments...

No comment can mean lack of interest.  Now you have some comments from
me which is good because there is at least some interest.  I don't
think dismissing substantive comments and moving on is useful.

So if you don't mind, lets set aside philosophical or procedural
discussion and continue the technical discussion and decide whether
anything in te-express-path and the protocol documents needs changing.

> concerning, such as the desire to include node queuing delay and use
> that as a criteria for paths without addressing any of the stability
> concerns that started this conversation.  I have NO interest in trying
> to meet that requirement without someone doing the necessary
> convincing studies, analysis & convincing.  Someone else can define a
> new sub-TLV to advertise node delays and related information.  In this
> case, IMHO, it's useful to separate out the fairly static link delay
> from the traffic-dependent queuing delay.  So - no duplication there.
> It needs significant research and analysis to prove it's useful; I
> don't see a reason to wait on the draft-ietf-ospf-te-metric-extensions
> or draft-previdi-isis-te-metric-extensions for that.

BTW- I and most other people consider queuing delay to be a component
of link delay since the queues are per link.  Node delay is usually
considered to be the delay involved in crossing the switching fabric,
which is relatively constant per interface.

If you want to exclude queuing delay, then that should be very clearly
stated in te-express-path and in both te-metric-extension drafts.
That would end a lot of discussion.  It would also mean changing this
text:

   When considering performance-based data, it is obvious that there
   are additional contributors beyond just the links.  Clearly
   end-to-end latency is a combination of router latency, queuing
   latency, physical link latency and other factors.  However, if
   application traffic requires paths to be selected based upon
   latency constraints, the same traffic might be in an Expedited
   Forwarding Per-Hop- Behavior[RFC3246] with minimal queuing delay or
   another PHB with known maximal per-hop queuing delay.  While
   traversing a router can cause delay, that can be included in the
   advertised link delay.

   This document does not specify how a router determines what values
   to advertise by the IGP.  However, the end-to-end performance that
   is computed for an LSP path SHOULD be built from the individual
   link data.  Any end-to-end characterization used to determine an
   LSP's performance compliance should be fully reflected in the
   Traffic Engineering Database so that a CSPF calculation can also
   determine whether a path under consideration would be in
   compliance.

A reasonable replacement would be:

   When considering performance-based data, it is obvious that there
   are additional contributors beyond just the links.  Clearly
   end-to-end latency is a combination of router latency, queuing
   latency, physical link latency and other factors.  Only physical
   link latency is considered.  While traversing a router can small
   delay, such as the delay associated with traversing a switch
   fabric, that can be included in the advertised link delay.

   Queuing latency is specifically excluded to insure freedom from
   oscillations and stability issues that have plauged prior attempts
   to use delay as a routing metric.  If application traffic which
   follows path based upon latency constraints, the same traffic might
   be in an Expedited Forwarding Per-Hop-Behavior [RFC3246] with
   minimal queuing delay or another PHB with potentially very
   substantial per-hop queuing delay.  Only traffic which experiences
   relatively low congestion, such as Expedited Forwarding traffic,
   will experience delays very close to the sum of the reported link
   delays.

   This document does not specify how a router determines the link
   delays that are advertised by the IGP.  However, the end-to-end
   performance estimate that is computed for an LSP path SHOULD be
   built from the individual link data.  Any end-to-end
   characterization used to determine an LSP's performance compliance
   should be fully reflected in the Traffic Engineering Database so
   that a CSPF calculation can also determine whether a path under
   consideration would be in compliance.

Note that the last sentence, which remains unchanged indicates that if
TP DM OAM or other measure of path delay is different, then that must
be indicated in the TE-LSDB.  That would only be possible if a FA was
created for the LSP.  If the LSP was rerouted, then the measurement no
longer exists and the FA should also disappear.  That would create a
chance for oscillation.

A better approach would be to create a zero bandwidth LSP on a
potential path specifically for the purpose of measuring end-to-end
delay at high priority (high TC value is using diffserv CS service -
IP Prec) and run DM OAM or other measurement of delay on the LSP.
Since the LSP never goes away, there is no chance of oscillation.
These LSP would only exist on path that were either in use, or were
feasible based on the sum of the delays, but were infeasible based on
the path measurements.

> As for the RSVP-TE path related requirements, these drafts do go much
> much farther than draft-atlas-mpls-te-express-path does.  This is the
> ocean-boiling (perhaps pond) of handling inter-area, inter-as, and
> LSPs without explicit routes completely computed by the ingress.
> Those aspects may be addressing very real use-cases - but that is much
> more extensive and complicated than a simple informational draft
> showing how the extended TE metrics can be useful today with minimal
> effort.

Yes.  I absolutely agree that tackling intra-AS first and any other
inter-domain later (if ever) is a better approach.  I have made
comments of this sort to the authors of draft-fuxh-mpls-delay-loss-*
but have not received a reply.  If that situation continues (no reply
from the authors) then perhaps we should consider
draft-fuxh-mpls-delay-loss-* to be abandonned.

> Right, the intent is that the information could come from many places,
> and John Drake can quote the RFC that is a good example.

OK John.  You are on stage next.  I don't know which RFC Alia is
referring to.

I think we are in agreement that the measurement should be link delay,
specifically exclude queuing delay, and optionally include any
relatively fixed (and generally extremely small) router delay due to
such things as the fabric in with the link delay.  Note that fabric
delays are generally a fairly small number of microseconds, as long as
any VOQ related delays are excluded.

> If you look at draft-ietf-ospf-te-metric-extensions, the delay can
> range from 1 micro-second to 16.777215 seconds.  Do we really need
> more dynamic range and associated complexity?

The hysteresis should be configurable.  If the hysteresis is such that
we are sending an updated TLV for any given link only when there is a
substantial change, then the range will still be in the tens or
hundreds of microseconds.  What most people don't realize is that for
packet data, even the number of FEC corrected bit-errors affects the
number of FEC passes and the latency.  This can be a few multiples of
the TDM (ie: SONET or OTN) superframe time.  With very few or no
correctable bit errors (ie: very clean and one FEC pass), for packet
data there is still a jitter of about a superframe time, as I
explained in my last email to John.

I'm fine with excluding queuing delay.  If so, then providing a range
of delay is far less important.  This would be fine for high priority
services that have small queuing delay (no more than a few packet
times).

If anything the resolution is too high.  In a datacenter where there
is negligible geographic delay, then microseconds may matter.  In a
microsecond light travels 200m in fiber (300m in vacume), so for most
datacenters at most only a few microseconds are added when going
between buildings.  Serialization of 1500B at 1Gb/s is 12 usec (which
is why cut-through is popular in datacenter switches and routers).  It
seems to be only the datacenter and campus networks where microsecond
resolution is useful.

> draft-previdi-isis-te-metric-extensions meet the composite-link
> requirements, that is useful; it's always nice to solve multiple
> problems at the same time.

Yes.  And I think there is very little that needs to be changed to
accomplish that.

> (1),(2), and (3) It's not measuring queuing delay.  I'll work on improved

Though it was not clear from a read of your document, it is clear from
your email and John's email that you intended to exclude queuing
delay.

What about a MPLS server layer which is introducing delay?  That would
be a variable delay coming from the server layer.

Also for CL where heterogeneous component links are desired, there is
a need to advertise a range for a group, so that we don't end up with
one group per component link (destroying any benefit of grouping).
Therefore I think a range would be good.

Long term drift in the measured delay could be reported by way of the
range.  Any relatively short term variation in the delay would be
reported as jitter.  For real time applications the jitter would
indicate the size of jitter buffer needed, with drift compensation
through very slight speedup or slow down of the playback (dropping or
interpolating data points typically rather than clock adjustment).

> The draft states that it should be an average.  Why would this
> information WANT to be communicated?  It seems the type of thing that
> would be documented - perhaps configurable on a router, but I can't
> see a reason for signaling it.

The word average does not appear in te-express-path.

> I can see that there might be an argument for using a 98th percentile
> vs.  an average.

I kinda like using a range.  It also has a use for CL if we are
reporting values in microseconds and there is a slight variation in
the length of parallel fibers.

> Why not?  What is being measured and reported does not change as a
> result of traffic load.  It's based on characteristics of the link and
> not queuing.  Perhaps one should put in some words about future study
> for stability for virtual links, whose measurements may include
> queuing delay in them?  Perhaps we should should say that these values
> shouldn't be advertised for such links?  Or have a way of measuring
> them that assumes maximum queuing delay at each hop on the virtual
> link's path?

If you are excluding queuing delay, then you can cite that as a reason
that oscillations should not occur.

You should also point out that if the underlying server layer, such as
an MPLS server layer, has an inherent queuing delay, then there is a
chance from oscillation and even instability, if offerred load is high
enough to impact congestion at the measured PHB.

> Again, the draft-fuxh-set is trying to do much more complicated stuff
> for computation; I can see those authors writing a suitable draft to
> explain how to compute LSPs between a min delay and a max delay.

The working path is computed with a max path delay.  Once a working
path is determined, the protect path is computed with a max path
delay.  If both paths are within range, then they are set up and
rerouting (preferably of the protect path) may be required if the
working path is still within its max path delay but the delay
difference between working and protect is too great as a result of a
change.

> I've included ISIS back as well, because the concerns were raised
> there about draft-previdi-isis-te-metric-extensions and the
> application and usage discussed is important for ISIS to draw
> conclusions.
>  
> Alia

If you agree that both te-metric-extension drafts need to await
resolving issues in te-express-path, then we don't need to cross
post.  Cross posting is a bad practice.

I don't think it will take long to resolve issues with
te-express-path.

Curtis

---

From: Alia Atlas <[email-masked]>
Date: Sat, 23 Feb 2013 13:52:12 -0500

Curtis,

I am snipping this thread (for the ISIS WG) to the relevant changes to
draft-previdi-te-isis-metric-extensions and draft-ietf-ospf-te-metric
extensions.  I will have a separate reply for the comments relevant to
draft-atlas-mpls-te-express-path to the mpls WG list.

> which is relatively constant per interface.

[Alia] Yes, I usually consider queuing delay to be a potential component of
link delay.  draft-fuxh-* however referred to it the other way and since I
was addressing that, I used their terminology.

> stated in te-express-path and in both te-metric-extension drafts.

[Alia]  I have suggested the following wording to my co-authors that will
hopefully make it in before IETF.  In the last paragraph of the
introduction to go to:

"Note that the mechanisms described in this document only disseminate
performance
information. The methods for initially gathering that performance
information, such as [Frost], or acting on it once it is distributed are
outside the scope of this document.  Example mechanisms to measure latency,
delay variation, and loss in an MPLS network are given in [RFC6374].  While
this document does not specify how the performance information should be
obtained, the measurement of delay SHOULD NOT vary significantly based upon
the offered traffic load.  Thus, queuing delays SHOULD NOT be included in
the delay measurement.  For links, such as Forwarding Adjacencies, care
must be taken that measurement of the associated delay avoids significant
queuing delay; that could be accomplished in a variety of ways, including
either by measuring with a traffic class that experiences minimal queuing
or by summing the measured link delays of the components of the link’s
path."

> OK John.  You are on stage next.  I don't know which RFC Alia is

[Alia] It is RF 6374 and the reference is given in the above new paragraph.

> explained in my last email to John.

[Alia] For the hysteresis being configurable, the drafts currently say:

"The values advertised in all sub-TLVs MUST be controlled using an exponential
filter (i.e. a rolling average) with a configurable measurement interval
and filter coefficient.

Implementations are expected to provide separately configurable advertisement
thresholds. All thresholds MUST be configurable on a per sub-TLV basis.

The announcement of all sub-TLVs that do not include the A bit SHOULD be
controlled by variation thresholds that govern when they are sent."
 and

"When link performance average values change, but fall under the threshold
that would cause the announcement of a sub-TLV with the A bit set,
implementations MAY suppress or throttle sub-TLV announcements. All
suppression features and thresholds SHOULD be configurable."

For the concerns about network stability, the drafts clearly state the
advertised values MUST NOT be instantaneous, but rather based upon a
configurable rolling period.

We already have configurable periods, no more frequent announcements than
1/sec, configurable advertisement thresholds, and configurable inter-update
throttle timers.   What else would you want to be configurable??

> seems to be only the datacenter and campus networks where microsecond

[Alia] Sure - but we have the bits and there are use-cases for microsecond
resolution.
So I think there's no need to change that.

> be a variable delay coming from the server layer.

[Alia] This is a good point.  I addressed it in my changed paragraph in the
introduction.

> Therefore I think a range would be good.

[Alia] For CL, the question is whether each member is advertised (as Tony
Li had recommended) or bundled in some fashion.  The existing TE sub-TLVs
don't address providing a range.  Rather than try to change the individual
sub-TLVs, it may make sense to define a max-TE-TLV and a min-TE-TLV that
can include the already existing sub-TLVs without modification and be used
for CL.   I do think this aspect is CL specific; it would be good to think
of the best way to handle those requirements, but I don't think that is by
modifying these sub-TLVs.

> interpolating data points typically rather than clock adjustment).

[Alia] We have delay variation.  As Spencer explained, this is for networks
whose underlying links may change their latency in sudden step-function
fashion due to regrooming or re-routing.  I don't really see the benefit of
a range here, except to make it more complicated.

> The word average does not appear in te-express-path.

[Alia] Yes - because it is using the extensions defined in the associated
OSPF and ISIS drafts.
That said, I'm not so convinced that average is the one true value to
advertise, particularly if one is trying to avoid exceeding a bound; I know
that 98th percentile is commonly used in looking at SLAs and performance.
 To provide more flexibility, we could add a 2-bit field that would allow
indication of what type of statistic is being sent.  I've suggested to my
co-authors:

"The M field indicates how the value sent is calculated.  The M field is 2
bits long and has the following meanings:

       00:  Average

       01:  98th percentile

       02:  Network Custom – a different statistic is sent.  The specific
meaning is dependent upon the network and configuration.

       03:  RESERVED: A router SHOULD NOT send this value.  If it is
received, then a router SHOULD treat it as 02 (Network Custom)

"

> the length of parallel fibers.

[Alia] A range doesn't describe the frequency and I don't think we want to
be sending the full set of samples or a probablility distribution function!
 There's also the question of how to use the information - and range
doesn't give enough data.  I think the CL concerns should be solved in a
different way that allows the backwards use of all existing TE sub-TLVs.

> that oscillations should not occur.

[Alia] Right - these drafts now explicitly exclude queuing delay as in the
modified introduction paragraph.

> enough to impact congestion at the measured PHB.

[Alia] Instead, I said that the measurements shouldn't include queuing
delay.

> post.  Cross posting is a bad practice.

[Alia] I'm not convinced that the te-metric-extensions draft need to await
resolving issues in the mpls-te-express-path, but I'm happy to work on
resolving issues rapidly in the mpls-te-express-path as well.

Alia

---

From: "Mcdysan, David E" <[email-masked]>
Date: Thu, 21 Feb 2013 16:48:34 -0500

Some responses in line below related to draft-fuxh-mpls-delay-loss-te-problem-statement and -framework in line below. Cross posting to camp since the mols wg and the AD's decided in Atlanta that these drafts should be discussed in camp since they include multi-layer aspects.
Dave
From: Alia Atlas <[email-masked]<mailto:[email-masked]>>
Date: Thursday, February 21, 2013 3:56 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Mpls <[email-masked]<mailto:[email-masked]>>, Nabil Bitar <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Alia Atlas <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [mpls] ISIS TE Metric Extension Discussion (IETF-85 Followup)

Hi Curtis,

On Tue, Feb 19, 2013 at 7:56 PM, Curtis Villamizar <[email-masked]<mailto:[email-masked]>> wrote:

Alia,

In general, we should avoid duplication.  Where there is strong disagreement about the requirements and usage, or where there isn't sufficient maturity on all the aspects, then it makes sense to think about the problem.

There's also the difference between trying to make a cup of tea and boiling the ocean.

Finally, it is good to get things done - particularly when drafts are sitting for 1+ years without comments...

I think that mpls-te-express-path and the igp drafts cover case 1 of Section 5.3 of draft-fuxh-mpls-delay-loss-te-problem-statement-01

1.  The case of few nodes with large geographic separation, (e.g.,trans-oceanic), where link delay alone would be a good approximation.

Would like to hear from others if the data center and mix of link/node delay for cases 2 and 3 is of strong interest to anyone.

draft-fuxh-mpls-delay-loss-te-framework documented our understanding of what requirements the express path igp and mpls drafts met. Suggest that the IGP and MPLS draft authors review that draft to see if anything in there is covered, or could be easily added/changed.

There are some  requirements in draft-fuxh-mpls-delay-loss which are concerning, such as the desire to
include node queuing delay and use that as a criteria for paths without addressing any of the stability concerns
that started this conversation.

In the -01 version we tried to state requirements that would address these stability concerns, for example in section 5.2, draft-fuxh-mpls-delay-loss-te-problem-statement-01

The solution SHOULD provide a means to dynamically measure and/or estimate the one-way link and node performance parameters for delay, loss and delay variation.
As defined in section 2.2. , the estimation interval for the performance parameters is assumed to be on the order of minutes. The solution MUST not impact stability nor significantly increase convergence time if performance parameters change over a timeframe on the order of minutes.

 I have NO interest in trying to meet that requirement  without someone doing the necessary
convincing studies, analysis & convincing.
In my opinion, the mandatory requirement is to allow configuration of the parameters, which would enable an operator to measure parameters and configure them that could be reported on in an analysis/study as stated in 5.2.

The solution MUST provide a means to configure the one-way link and node performance parameters for delay, loss and delay variation.

 Someone else can define a new sub-TLV to advertise node delays and related information.   In this case, IMHO, it's useful to separate out the fairly static link delay from the traffic-dependent queuing delay.
So - no duplication there.   It needs significant research and analysis to prove it's useful; I don't see a reason to wait on the draft-ietf-ospf-te-metric-extensions or draft-previdi-isis-te-metric-extensions for that.

Agreed, Section 5.3 of draft-fuxh-mpls-delay-loss-te-problem-statement-01 states that node level delay is a separate requirement that is desirable (SHOULD), not mandatory.

As for the RSVP-TE path related requirements, these drafts do go much much farther than draft-atlas-mpls-te-express-path does.   This is the ocean-boiling (perhaps pond) of handling inter-area, inter-as, and LSPs without explicit routes completely computed by the ingress.  Those aspects may be addressing very real use-cases - but that is much more extensive and complicated than a simple informational draft showing how the extended TE metrics can be useful today with minimal effort.

Right, the intent is that the information could come from many places, and John Drake
can quote the RFC that is a good example.

If you look at draft-ietf-ospf-te-metric-extensions, the delay can range from 1 micro-second to
16.777215 seconds.  Do we really need more dynamic range and associated complexity?

Certainly, if draft-ietf-ospf-te-metric-extensions and draft-previdi-isis-te-metric-extensions meet
the composite-link requirements, that is useful; it's always nice to solve multiple problems at the
same time.

(1),(2), and (3) It's not measuring queuing delay.  I'll work on improved text to clarify this.

The draft states that it should be an average.  Why would this information WANT to be
communicated?  It seems the type of thing that would be documented - perhaps
configurable on a router, but I can't see a reason for signaling it.

In my opinion, the mandatory requirement from fuxh-problem-statement in section 5.2 is to allow configuration of the parameters, which would enable an operator to change these based upon static or dynamic measurements.

The solution MUST provide a means to configure the one-way link and node performance parameters for delay, loss and delay variation.
I think that the IGP drafts support this.

 I can see that there might be an argument for using a 98th percentile vs. an average.

Why not?  What is being measured and reported does not change as a result of
traffic load.  It's based on characteristics of the link and not queuing.  Perhaps one should
put in some words about future study for stability for virtual links, whose measurements may
include queuing delay in them?  Perhaps we should should say that these values shouldn't
be advertised for such links?  Or have a way of measuring them that assumes maximum
queuing delay at each hop on the virtual link's path?

This is a fair summary. Note that the requirement is desirable (SHOULD).

Again, the draft-fuxh-set is trying to do much more complicated stuff for computation;
I can see those authors writing a suitable draft to explain how to compute LSPs
between a min delay and a max delay.

If you don't mind, I think we should take this thread to MPLS.

At the last IETF, the mpls wg and the AD's decided that the fuxh drafts should go to camp. I missed the deadline to submit a -00 draft to camp and they won't get discussed there. Would really like to get some feedback on what aspects are more important than others. I made a number of the requirements SHOULD/MAY since IMO they are of lower priority. There are still a number of Open MUST requirements. Cross posting this thread to camp for this reason.

 I

I've included ISIS back as well, because the concerns were raised there about
draft-previdi-isis-te-metric-extensions and the application and usage discussed
is important for ISIS to draw conclusions.

Alia

Curtis

In message <[email-masked]<mailto:[email-masked]>>
Alia Atlas writes:

Curtis,

are UNCONNECTED

I would say not completely unconnected, the drafts mentioned below meet some of the requirements in   draft-fuxh-mpls-delay-loss-te-problem-statement-01 as documented in  draft-fuxh-mpls-delay-loss-te-framework draft.

to the drafts that we are discussing, which are:

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, February 19, 2013 1:57 AM
To: Alia Atlas
Cc: Lou Berger; [email-masked]<mailto:[email-masked]>; John E Drake; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Alia,

Right now we have:

Would the authors of the set of documents above please combine them.
Would they also please consider explicitly addressing the requirments in draft-ietf-rtgwg-cl-requirements, such as stability, keeping the amount of churn down, supporting a grouping of CL component links.  If you do this, would you also please include me in the discussion.
 draft-fuxh-mpls-delay-loss-te-problem-statement-01 tried to explicitly state the CL requirements which overlap with it.

If I missed something, please identify and I can update -- if the community believes collecting these requirements (near and longer term, straightforward and controversial in one place) is of value, I would like to receive some affirmation before investing more time updating draft-fuxh-mpls-delay-loss-te-problem-statement and -framework.

Curtis

_______________________________________________
mpls mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/mpls

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 22 Feb 2013 00:08:59 +0000

Dear Alia, et al.,
"What is being measured and reported does not change as a result of
traffic load.  It's based on characteristics of the link and not queuing.  Perhaps one should
put in some words about future study for stability for virtual links, whose measurements may
include queuing delay in them?  Perhaps we should should say that these values shouldn't
be advertised for such links?  Or have a way of measuring them that assumes maximum
queuing delay at each hop on the virtual link's path?"
I think that uncertainty of whether it is possible to measure delay without queing requires careful consideration  since:

  *    if in fact such measurements are not plausible then delay effectively becomes propagation delay and is, for practical purposes, static metric.
  *   but if delay metric is dynamically measureable, then, I believe, performance measurement methodology must be consistent throughout the IGP domain (test packet TC setting, packet length, inter-paket interval etc.)

And, piggyabcking on, IGP-TE question:

  *   how Unreserved Bandwidth is different from Residiual BW proposed in IGP TE documents as new sub-TLV.

    Regards,
        Greg

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Alia Atlas
Sent: Thursday, February 21, 2013 12:57 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Samita Chakrabarti; Acee Lindem; Alia Atlas; [email-masked]; [email-masked]
Subject: Re: [mpls] ISIS TE Metric Extension Discussion (IETF-85 Followup)

Hi Curtis,

Alia,

There are some  requirements in draft-fuxh-mpls-delay-loss which are concerning, such as the desire to
include node queuing delay and use that as a criteria for paths without addressing any of the stability concerns
that started this conversation.  I have NO interest in trying to meet that requirement  without someone doing the necessary
convincing studies, analysis & convincing.  Someone else can define a new sub-TLV to advertise node delays and related information.   In this case, IMHO, it's useful to separate out the fairly static link delay from the traffic-dependent queuing delay.
So - no duplication there.   It needs significant research and analysis to prove it's useful; I don't see a reason to wait on the draft-ietf-ospf-te-metric-extensions or draft-previdi-isis-te-metric-extensions for that.

Alia

Curtis

Curtis,

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, February 19, 2013 1:57 AM
To: Alia Atlas
Cc: Lou Berger; [email-masked]<mailto:[email-masked]>; John E Drake; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: ISIS TE Metric Extension Discussion (IETF-85 Followup)

Alia,

Right now we have:

Curtis