Subject: Re: [alto] Review of draft-ietf-alto-deployments-08#appendix-A

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Thu, 13 Feb 2014 17:39:36 +0000

Hi Qin Wu,

The new version of the draft includes an entirely re-written section replacing the former appendix A:

http://tools.ietf.org/html/draft-ietf-alto-deployments-09#section-3.4

Please let me know if this does not address your concerns or if you have suggestions for improvements.

Thanks

Michael

From: alto [mailto:[email-masked]] On Behalf Of Scharf, Michael (Michael)
Sent: Friday, January 10, 2014 10:43 AM
To: Qin Wu; IETF ALTO
Subject: Re: [alto] Review of draft-ietf-alto-deployments-08#appendix-A

Hi Qin Wu,

Thanks for your feedback. You raise similar issues like a performance directorate review of the same part of the document (cf. http://www.ietf.org/mail-archive/web/pm-dir/current/msg00610.html).

Addressing this review is still TBD. Version -08 of the draft contains numerous other edits, and since I focused on those other parts of the document, I just moved this problematic section into appendix A. My thinking (see also the slides at the last meeting) is that the current text in appendix A will have to be almost completely removed.

I plan to publish -09 before the next meeting, and addressing the performance directorate review is a high-priority item. If you have suggestions for specific text to be contributed to -09, please let me know.

Thanks

Michael

From: alto [mailto:[email-masked]] On Behalf Of Qin Wu
Sent: Tuesday, December 24, 2013 10:06 AM
To: IETF ALTO
Subject: [alto] Review of draft-ietf-alto-deployments-08#appendix-A

Hi,
I have reviewed appendix A of draft-ietf-alto-deployments-08.
It appear to me this appendix A is outdated and need to be update or separated as another draft.
A few comments below:
1. Appendix A paragraph 1 said:
"
In addition to providing configuration, an ISP providing ALTO may
want to deploy a monitoring infrastructure to assess the benefits of
ALTO and adjust its ALTO configuration according to the results of
the monitoring.

"
It looks something is disconnected when we say "in addition to providing configuration", where
Configuration providing is discussed in this draft?

Also I think monitoring infrastructure is not limited to assess the benefits of ALTO, I think the more important
thing is performance metrics can be injected into ALTO server to provide fine granularity of cost map and network map,
alto server can leverage these information to decide which is the best endpoint to connect.

2. Appendix ,paragraph 3 said:
"
   [Editor's note: Is there a relationship to the IPPM working group at
   the IETF?]

"
Sure, for most QoS metrics like loss, delay, delay variation, etc.,
standard IPPM definitions exist.  In case such metrics are reported,
the IPPM standard definition should be used.

3. Appendix A.1

Monitoring Metrics is not limited to the list given in the Appendix A.1,
Also it doesn't make sense to enumerate all the performance metrics.

4. Appendix A.2
If you support routing protocol, you can gather peformance metrics from routing protocol,
In this case, you don't need to deploy any data source to collect data.

On the other hand, there are many ways in which the performance of an data folw can be
monitored.  These include generic MIBs, NetFlow, IP Flow Information Export
(IPFIX),syslog, and so on.

Regards!
-Qin

---

From: Qin Wu <[email-masked]>
Date: Sat, 15 Feb 2014 03:57:10 +0000

Hi, Michael:
Thanks for re-work and updating to monitoring section.
I have a few comments to section 3.4 you updated.
1. The monitoring infrastructure can be third party,Or can be integrated into ALTO server? If it is the third party based, I think the trust relationship between monitoring system and alto server needs to be established, otherwise, security concern may be raised.
2.  Section 4.3 says:
"
An NSP offering ALTO could consider the impact on (or integration with) traffic
   engineering and the deployment of a monitoring service to observe the
   effects of ALTO operations.
"

Traffic engineering is related to monitoring ALTO performance? I think you may gather traffic engineering information from routing protocol, these traffic engineering information can be used to reflect network performance and provide fine granularity of cost map or network map? However it has nothing to do with performance of ALTO?
Performance of ALTO is more about how many request or response can be handled,
3. Section 3.4.2 says:
"
Total amount and distribution of traffic: ALTO enables ISPs to
influence and localize traffic of applications that use the ALTO
service.  An ISP may therefore be interested in analyzing the
  impact on the traffic, i.e., whether network traffic patterns are
shifted.  For instance, if ALTO shall be used to reduce the inter-
  domain P2P traffic, it makes sense to evaluate the total amount of
inter-domain traffic of an ISP.
"
How traffic distribution is related to ALTO performance monitoring, I can not understand this.

4. Section 3.4.2 says:
"

Application performance: The objective of ALTO is improve
      application performance.  ALTO can be used by very different types
      applications, with different communication characteristics and
      requirements.  For instance, if ALTO guidance achieves traffic
      localization, one would expect that applications achieve a higher
      throughput and/or smaller delays to retrieve data.  Application-
      specific performance characteristics (e.g., video or audio
      quality) can be useful as well.  In addition, selected statistics
      from the TCP/IP stack in hosts can be useful, e.g., the number of
      retransmitted TCP segments.
"
It is not clear to me whether you measure traffic performance or ALTO performance?
I think application performance you described here has nothing to do with ALTO performance.
Only ALTO system performance is related to ALTO performance. I feel you mix traffic performance
With ALTO performance. What am I missing?

Regards!
-Qin
From: Scharf, Michael (Michael) [mailto:[email-masked]]
Sent: Friday, February 14, 2014 1:40 AM
To: Qin Wu
Cc: IETF ALTO
Subject: RE: Review of draft-ietf-alto-deployments-08#appendix-A

Hi Qin Wu,

Thanks

Michael

Hi Qin Wu,

Thanks

Michael

Hi,

"

"

3. Appendix A.1

4. Appendix A.2

Regards!
-Qin

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sat, 15 Feb 2014 19:54:41 +0000

Hi Qin,

Please find some comments inline <MSC/>.

From: Qin Wu [mailto:[email-masked]] 
Sent: Saturday, February 15, 2014 4:57 AM
To: Scharf, Michael (Michael)
Cc: IETF ALTO
Subject: RE: Review of draft-ietf-alto-deployments-08#appendix-A

Hi, Michael:

<MSC>
I think the most realistic case it that it is neither third party, nor an integrated function of the ALTO server. Instead, a network service provider would probably monitor the benefits of its ALTO service by a monitoring infrastructure, which could be an extension of the general network monitoring / OAM. It could be possible that third parties (e.g., academic researchers) want to monitor the benefits as well; in that case, there is indeed a question whether they would have access to network service provider data beyond what the ALTO service provider exposes. However, since the IETF does not mandate how to implement an ALTO server, an implementer could obviously also come up with a huge all-in-one ALTO server implementation that realizes many other functions beyond providing the ALTO service itself.

I could add a sentence like "Monitoring the benefits of an ALTO service could also be realized by third parties. In this case, insight into ALTO data may require a trust relationship between the monitoring system operator and the network service provider offering an ALTO service." Would this address your comment? 
</MSC>

2.  Section 4.3 says:
"

<MSC>
This sentence is taken from Section 16.1.4 of the ALTO protocol spec (http://tools.ietf.org/html/draft-ietf-alto-protocol-25#page-81). Actually, the whole scope of Section 3.4 in draft-ietf-alto-deployment is explained in Section 16.1.4 of the ALTO protocol spec. Given that explanation there, I am not sure if I understand your question and/or concern. Specifically, the whole section 3.4 of draft-ietf-alto-deployment deals NOT with how to retrieve routing protocol information (including e.g. TE information from IGP) and how to put it into network and cost maps. This is discussed in Section 3.2.1.

Let me try to explain how I understand draft-ietf-alto-protocol-25: This sentence deals with understanding what impact the ALTO guidance has on applications, and whether the ALTO service operation is efficient and effectively improving "better-than-random" resource selections by applications. I understand here "traffic engineering" in a very general way. For instance, an NSP offering an ALTO service for P2P applications would typically expect that the amount of aggregated inter-domain traffic is reduced. If ALTO is effective, the NSP can probably slow down upgrades of Internet-peering-point capacities; this is traffic engineering, but that kind of information may not be carried in, or retrieved from routing protocols, and it may not be used to calculate ALTO maps. If traffic is kept local in access network domains, the capacity planning and traffic engineering of core network links can be adapted, because they may have less utilization, e.g., bandwidth allocations of core links could be adapt. This can be part of the typical mid-/long-term capacity planning processes of a network service provider.

Furthermore, please note the ALTO protocol spec has another section (Section 16.2.5. Performance Management) that describes examples for system metrics characterizing the performance of an ALTO server (e.g., number of requests, CPU/memory utilization, etc.).

If you struggle with the term "ALTO performance" in the headline, we could perhaps better align the section headlines to draft-ietf-alto-protocol-25, i.e.:

OLD:
3.4. Monitoring the Performance of ALTO
3.4.1. Supervising the Benefits of ALTO
3.4.2. How to Monitor ALTO Performance

NEW:
3.4. Monitoring the Impact of ALTO
3.4.1. Supervising the Benefits of ALTO
3.4.2. How to Observe the Impact of ALTO

Would this address your question?
</MSC>

3. Section 3.4.2 says:
"

<MSC>
Again, please refer to Section 16.1.4 in draft-ietf-alto-protocol-25. That document states "Deployment of an ALTO Server may shift network traffic patterns.". This paragraph just explains examples how one can monitor whether ALTO achieves the shift of traffic distribution that an NSP wants to achieve by ALTO.
</MSS>

4. Section 3.4.2 says:
"

<MSC>
The draft explains the categories "total amount and distribution of traffic", "application performance", and "ALTO system performance", and they all refer to different aspects. For the headline, I was looking for a short term that encompasses all different aspects, and I came up with "ALTO performance" (obviously, this is not the ALTO system performance). If you expand it as "application-layer topology optimization performance", in my understanding "ALTO performance" refers to how efficient and effective the application topologies are optimized by deploying an ALTO service.

However, as already mentioned, if the term "ALTO performance" should be confusing, I could use the term "impact of ALTO" instead of "ALTO performance", i.e., rename the section to "3.4.2. How to Observe the Impact of ALTO". In that case, it should be clear that it deals with much more than ALTO system performance, "traffic performance", etc. Would this solve your question?
</MSC>

Thanks

Michael

---

From: Qin Wu <[email-masked]>
Date: Mon, 17 Feb 2014 08:12:44 +0000

Hi,Michael:
Thanks for your clarification, please see my followup comments.

Regards!
-Qin
-----Original Message-----
>>Sent: Sunday, February 16, 2014 3:55 AM
>>To: Qin Wu
>>Cc: IETF ALTO

>>1. The monitoring infrastructure can be third party,Or can be integrated into ALTO server? If it is the third party based, I think the trust 
>>relationship between monitoring system and alto server needs to be established, otherwise, security concern may be raised.

<MSC>
>I think the most realistic case it that it is neither third party, nor an integrated function of the ALTO server. Instead, a network service provider 
>would probably monitor the benefits of its ALTO service by a monitoring infrastructure, which could be an extension of the general network monitoring 
>OAM. It could be possible that third parties (e.g., academic researchers) want to monitor the benefits as well; in that case, there is indeed a question 
>whether they would have access to network service provider data beyond what the ALTO service provider exposes. However, since the IETF does not mandate 
>how to implement an ALTO server, an implementer could obviously also come up with a huge all-in-one ALTO server implementation that realizes many other 
>functions beyond providing the ALTO service itself.

>I could add a sentence like "Monitoring the benefits of an ALTO service could also be realized by third parties. In this case, insight into ALTO data 
>may require a trust relationship between the monitoring system operator and the network service provider offering an ALTO service." Would this address 
>your comment? 

</MSC>

[Qin]:Yes, thanks.

"
"

>>Traffic engineering is related to monitoring ALTO performance? I think you may gather traffic engineering information from routing protocol, these 
>>traffic engineering information can be used to reflect network performance and provide fine granularity of cost map or network map? However it has 
>>nothing to do with performance of ALTO?

<MSC>
>This sentence is taken from Section 16.1.4 of the ALTO protocol spec (http://tools.ietf.org/html/draft-ietf-alto-protocol-25#page-81). Actually, the 
>whole scope of Section 3.4 in draft-ietf-alto-deployment is explained in Section 16.1.4 of the ALTO protocol spec. Given that explanation there, I am 
>not sure if I understand your question and/or concern. Specifically, the whole section 3.4 of draft-ietf-alto-deployment deals NOT with how to retrieve 
>routing protocol information (including e.g. TE information from IGP) and how to put it into network and cost maps. This is discussed in Section 3.2.1.

[Qin]: The confusion comes from the title of section 3.4 of draft-ietf-alto-deployments-08,i.e.," Monitoring the Performance of ALTO " since ALTO performance is more about resource oriented objective, e.g, CPU,memory, request/responses to be handled, ALTO performance has nothing to do traffic oriented objective.

>Let me try to explain how I understand draft-ietf-alto-protocol-25: This sentence deals with understanding what impact the ALTO guidance has on 
>applications, and whether the ALTO service operation is efficient and effectively improving "better-than-random" resource selections by applications. I 
>understand here "traffic engineering" in a very general way. For instance, an NSP offering an ALTO service for P2P applications would typically expect 
>that the amount of aggregated inter-domain traffic is reduced. 

[Qin]: The goal of Traffic engineering is to facilitate efficient and reliable network operation while simultaneously optimization network resource
Utilization and traffic performance. Inter-domain traffic engineering in some cases can be used to reduce traffic between domain, but in most of cases,
Traffic engineering will be use for traffic performance optimization or load balancing of the traffic. I am not sure traffic localization is related to
Traffic engineering. If it is, I think it is one aspect of traffic performance optimization.

>If ALTO is effective, the NSP can probably slow down upgrades of Internet-peering-point capacities; this is traffic engineering, but that kind of 
>information may not be carried in, or retrieved from routing protocols, and it may not be used to calculate ALTO maps.

[Qin]: If you are referring to bandwidth capability, it definitely carried using routing protocol, e.g.,ISIS-TE, OSPF-TE,
Can you tell me what kind of information you are referred to? 

> If traffic is kept local in access network domains, the capacity planning and traffic engineering of core network links can be adapted, because they 
>may have less utilization, e.g., bandwidth allocations of core links could be adapt. This can be part of the typical mid-/long-term capacity planning 
>processes of a network service provider.

>Furthermore, please note the ALTO protocol spec has another section (Section 16.2.5. Performance Management) that describes examples for system metrics 
>characterizing the performance of an ALTO server (e.g., number of requests, CPU/memory utilization, etc.).

[Qin]: I think systems metrics described in example of section 16.2.5 of ALTO protocol spec are the only metrics that characterize the performance of an ALTO server and make sense to me.

>Would this address your question?

[Qin]:Not enough, if you change " Performance of ALTO " to " Impact of ALTO ", I am wondering what is impacted by the ALTO server, then
I think the last two bullets in the section 3.4.2 should be moved to somewhere else or removed,i.e.,
"
   o  ALTO system performance: As mentioned in [I-D.ietf-alto-protocol],
      there are a number of interesting parameters that can be measured
      at an ALTO server, including the Requests and responses for each
      service listed in a Information Directory (total counts and size
      in bytes) or the CPU and memory utilization.  Also, the
      characteristics of the ALTO maps can be monitored as well, e.g.,
      regarding the frequency of ALTO map updates, the number of PIDs,
      or the ALTO map sizes (in-memory size, encoded size, number of
      entries).

   o  ALTO service utilization: Of potential interet can be the share of
      applications or customers that actually use an offered ALTO
      service, i.e., the degree of adoption.

"
I don't think ALTO system performance is impacted by using a ALTO sever.
ALTO service utilization seems just show the popularity of ALTO service deployment.

If you still use the old title for the section 3.4,i.e.," Monitoring the Performance of ALTO ",
I think the first two bullets and the last bullet should be removed.

"
"

<MSC>
>Again, please refer to Section 16.1.4 in draft-ietf-alto-protocol-25. That document states "Deployment of an ALTO Server may shift network traffic
>patterns.". This paragraph just explains examples how one can monitor whether ALTO achieves the shift of traffic distribution that an NSP wants to 
>achieve by ALTO.

[Qin]: See above.

</MSS>

"

"

<MSC>
>The draft explains the categories "total amount and distribution of traffic", "application performance", and "ALTO system performance", and they all 
>refer to different aspects. For the headline, I was looking for a short term that encompasses all different aspects, and I came up with "ALTO 
>performance" (obviously, this is not the ALTO system performance). If you expand it as "application-layer topology optimization performance", in my 
>understanding "ALTO performance" refers to how efficient and effective the application topologies are optimized by deploying an ALTO service.

>However, as already mentioned, if the term "ALTO performance" should be confusing, I could use the term "impact of ALTO" instead of "ALTO performance", 
>i.e., rename the section to "3.4.2. How to Observe the Impact of ALTO". In that case, it should be clear that it deals with much more than ALTO system 
>performance, "traffic performance", etc. Would this solve your question?

[Qin]:See above.

</MSC>

Thanks

Michael

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 17 Feb 2014 18:27:07 +0000

Hi Qui,

inline.

> Sent: Monday, February 17, 2014 9:13 AM
> >>1. The monitoring infrastructure can be third party,Or can be
> integrated into ALTO server? If it is the third party based, I think
> the trust
> >>relationship between monitoring system and alto server needs to be
> established, otherwise, security concern may be raised.
> 
> <MSC>
> >I think the most realistic case it that it is neither third party, nor
> an integrated function of the ALTO server. Instead, a network service
> provider
> >would probably monitor the benefits of its ALTO service by a
> monitoring infrastructure, which could be an extension of the general
> network monitoring
> >OAM. It could be possible that third parties (e.g., academic
> researchers) want to monitor the benefits as well; in that case, there
> is indeed a question
> >whether they would have access to network service provider data beyond
> what the ALTO service provider exposes. However, since the IETF does
> not mandate
> >how to implement an ALTO server, an implementer could obviously also
> come up with a huge all-in-one ALTO server implementation that realizes
> many other
> >I could add a sentence like "Monitoring the benefits of an ALTO
> service could also be realized by third parties. In this case, insight
> into ALTO data
> >may require a trust relationship between the monitoring system
> operator and the network service provider offering an ALTO service."
> Would this address
> >your comment?
> 
> </MSC>
> 
> [Qin]:Yes, thanks.

OK

> >>Traffic engineering is related to monitoring ALTO performance? I
> think you may gather traffic engineering information from routing
> protocol, these
> >>traffic engineering information can be used to reflect network
> performance and provide fine granularity of cost map or network map?
> However it has
> >This sentence is taken from Section 16.1.4 of the ALTO protocol spec
> (http://tools.ietf.org/html/draft-ietf-alto-protocol-25#page-81).
> Actually, the
> >whole scope of Section 3.4 in draft-ietf-alto-deployment is explained
> in Section 16.1.4 of the ALTO protocol spec. Given that explanation
> there, I am
> >not sure if I understand your question and/or concern. Specifically,
> the whole section 3.4 of draft-ietf-alto-deployment deals NOT with how
> to retrieve
> >routing protocol information (including e.g. TE information from IGP)
> and how to put it into network and cost maps. This is discussed in
> Section 3.2.1.
> 
> [Qin]: The confusion comes from the title of section 3.4 of draft-ietf-
> alto-deployments-08,i.e.," Monitoring the Performance of ALTO " since
> ALTO performance is more about resource oriented objective, e.g,
> CPU,memory, request/responses to be handled, ALTO performance has
> nothing to do traffic oriented objective.

Well, as I already explained, ALTO expands as Application-layer Topology Optimization, which is an abstract concept, not its implementation in a server.

In my understanding the term "Performance of ALTO" is *NOT* the same like "Performance of an ALTO server/service", and the text in that section explains that multiple times.

Having said this, I already offered to change this term.

> >Let me try to explain how I understand draft-ietf-alto-protocol-25:
> This sentence deals with understanding what impact the ALTO guidance
> has on
> >applications, and whether the ALTO service operation is efficient and
> effectively improving "better-than-random" resource selections by
> applications. I
> >understand here "traffic engineering" in a very general way. For
> instance, an NSP offering an ALTO service for P2P applications would
> typically expect
> [Qin]: The goal of Traffic engineering is to facilitate efficient and
> reliable network operation while simultaneously optimization network
> resource
> Utilization and traffic performance. Inter-domain traffic engineering
> in some cases can be used to reduce traffic between domain, but in most
> of cases,
> Traffic engineering will be use for traffic performance optimization or
> load balancing of the traffic. I am not sure traffic localization is
> related to

Thus, what do you suggest to change specifically in draft-ietf-alto-deployments-09?

I only copied the sentence from draft-ietf-alto-protocol-25. I think your definition of traffic engineering "facilitate efficient and reliable network operation while simultaneously optimization network resource Utilization and traffic performance" perfectly describes the key objectives of deploying an ALTO server, e.g., for P2P overlay optimization:

- "Network efficiency" is improved (e.g., by less inter-domain traffic, less peering costs, etc.)

- "Network reliability" is improved (e.g., by less risk of congesting the network by suboptimal P2P overlay topologies)

- "Network utilization" is optimized (e.g., by less network load on core links)

- "Traffic performance" is improved (e.g., higher P2P download throughput 

With all aspects of your definition of "traffic engineering", the sentence in draft-ietf-alto-protocol-25 "Thus, an ALTO Service Provider should consider impacts on (or integration with) traffic engineering and the deployment of a monitoring service to observe the effects of ALTO operations.  Note that ALTO-specific monitoring and metrics are discussed in [I-D.ietf-alto-deployments]." Makes perfectly sense.

What do I miss?

> >If ALTO is effective, the NSP can probably slow down upgrades of
> Internet-peering-point capacities; this is traffic engineering, but
> that kind of
> >information may not be carried in, or retrieved from routing
> protocols, and it may not be used to calculate ALTO maps.
> 
> [Qin]: If you are referring to bandwidth capability, it definitely
> carried using routing protocol, e.g.,ISIS-TE, OSPF-TE,

I might be wrong, I think the root cause of our mis-understanding is that you apparently assume that "traffic engineering" is equivalent to RSVP-TE, ISIS-TE, OSPF-TE, etc.

An Internet Service Provider can run an entire IP network without enabling ISIS-TE, OSPF-TE, BGP-TE, etc., right? For instance, if that network doesn't use any MPLS? In that case, as far as I understand routing protocols, no "bandwidth capabilities" will be carried in the routing protocols, right? That ISP can still optimize routing protocol link weights, modify the routing import/export policies, turn up/down interfaces, etc. In other words, an ALTO service provider has many means to influence how traffic is routed in his network, and he doesn't have to rely on ISIS-TE, OSPF-TE, etc. for that.

Also, please note that bandwidth information can come from a centralized inventory system/database instead of ISIS-TE, OSPF-TE, etc. As example, have a look at draft-farrkingel-pce-abno-architecture-07 Section 2.3.1.8.1. And I might be wrong, but the SDN research community argues right now that one build traffic engineering solutions without using RSVP-TE, ISIS-TE, OSPF-TE, BGP-TE, etc.

In summary: "traffic engineering" != data in routing protocols, right?

> > If traffic is kept local in access network domains, the capacity
> planning and traffic engineering of core network links can be adapted,
> because they
> >may have less utilization, e.g., bandwidth allocations of core links
> could be adapt. This can be part of the typical mid-/long-term capacity
> planning
> >Furthermore, please note the ALTO protocol spec has another section
> (Section 16.2.5. Performance Management) that describes examples for
> system metrics
> [Qin]: I think systems metrics described in example of section 16.2.5
> of ALTO protocol spec are the only metrics that characterize the
> performance of an ALTO server and make sense to me.
> 
> >If you struggle with the term "ALTO performance" in the headline, we
> could perhaps better align the section headlines to draft-ietf-alto-
> protocol-25,
> >i.e.:
> 
> >OLD:
> [Qin]:Not enough, if you change " Performance of ALTO " to " Impact of
> ALTO ", I am wondering what is impacted by the ALTO server, then

I do not agree that a removal makes sense.
 
There are ALTO system performance metrics and ALTO service utilization metrics that allow characterizing the "impact" of ALTO on a network. For instance, assume that an ISP runs several access networks, but he suspects that ALTO is not effectively used in one of them (e.g., because there is much less traffic locality in this access network than in others). How would he probably dig into that problem? If he runs the ALTO server, he could easily look into the distribution of requests and their mapping to access networks (ALTO system performance data, right?). If he observes that from one access network only few requests arrive, he can confirm that ALTO has hardly has any "impact" in that specific access network. Maybe there is a connectivity problem to the ALTO server in that network. Or maybe there is another problem, e.g., the users in those network don't opt-in to using said ALTO server. In any case, in this example, the NSP has used ALTO system performance data to understand why the ALTO deployment and its "impact" is not as efficient as expected.
 
I do not claim that the existing text is already perfect. Specifically, the text may not be comprehensive right now. But please suggest additional specific text if you think that the wording is not clear, instead of asking for removal of monitoring methods that are already introduced and motivated in draft-ietf-alto-protocol. 

Michael

> >Again, please refer to Section 16.1.4 in draft-ietf-alto-protocol-25.
> That document states "Deployment of an ALTO Server may shift network
> traffic
> >patterns.". This paragraph just explains examples how one can monitor
> whether ALTO achieves the shift of traffic distribution that an NSP
> wants to
> >achieve by ALTO.
> 
> [Qin]: See above.
> 
> </MSS>
> 
> 
> >>4. Section 3.4.2 says:
> "
> 
> >The draft explains the categories "total amount and distribution of
> traffic", "application performance", and "ALTO system performance", and
> they all
> >refer to different aspects. For the headline, I was looking for a
> short term that encompasses all different aspects, and I came up with
> "ALTO
> >performance" (obviously, this is not the ALTO system performance). If
> you expand it as "application-layer topology optimization performance",
> in my
> >understanding "ALTO performance" refers to how efficient and effective
> the application topologies are optimized by deploying an ALTO service.
> 
> >However, as already mentioned, if the term "ALTO performance" should
> be confusing, I could use the term "impact of ALTO" instead of "ALTO
> performance",
> >i.e., rename the section to "3.4.2. How to Observe the Impact of
> ALTO". In that case, it should be clear that it deals with much more
> than ALTO system

---

From: Qin Wu <[email-masked]>
Date: Tue, 18 Feb 2014 05:54:45 +0000

Hi, Michael again:

-----Original Message-----
From: Scharf, Michael (Michael) [mailto:[email-masked]] 
Sent: Tuesday, February 18, 2014 2:27 AM
To: Qin Wu
Cc: IETF ALTO
Subject: RE: Review of draft-ietf-alto-deployments-08#appendix-A

Hi Qui,

inline.

OK

[Qin]: When we says " Monitoring the Impact of ALTO", I am wondering what is impacted by who? I think who is "ALTO service provided by ALTO server", what that is impacted is traffic between the domains and application performance.
When we say " Monitoring the performance of ALTO ", I think ALTO performance is only referred to CPU, memory, the number of request or response that can be handled.
ALTO performance is not part of ALTO impact on traffic and application.
ALTO impact can not be applied to ALTO performance.
When we talking about ALTO performance, e.g., ALTO server 1 who can deal with 1000 request/response exchange, ALTO server 2 who can deal with 50000 request/response exchanges. I would say Alto Server 2 has better than ALTO server 1.

Mixing ALTO performance with ALTO impact make me difficult to understand. Apply ALTO impact to ALTO ALTO performance also make me confused.

What do I miss?

[Qin]: See above.

[Qin]: You misunderstand. I am talking about internet traffic engineering, Traffic engineering is not only applied to mpls-TE network but also applied to
Best effort network(e.g.,internet) and IP network.

An Internet Service Provider can run an entire IP network without enabling ISIS-TE, OSPF-TE, BGP-TE, etc., right? For instance, if that network doesn't use any MPLS? In that case, as far as I understand routing protocols, no "bandwidth capabilities" will be carried in the routing protocols, right? 

[Qin]:no, "bandwidth capability" can be carried in the routing protocol, as I said earlier, traffic engineering is also applied to best effort network,
e.g., maximum bandwidth attribute, carried in RFC5305 and RFC3784 is also applied to best effort network and equivalent to bandwidth capacity.
Utilized bandwidth attribute carried in isis-te-extension or ospf-te-extension draft is also applied to best effort network.

That ISP can still optimize routing protocol link weights, modify the routing import/export policies, turn up/down interfaces, etc. In other words, an ALTO service provider has many means to influence how traffic is routed in his network, and he doesn't have to rely on ISIS-TE, OSPF-TE, etc. for that.

[Qin]:Sure, as described in ALTO base protocol, ALTO information can be gather not only from routing protocol, but also from NMS/OSS.

Also, please note that bandwidth information can come from a centralized inventory system/database instead of ISIS-TE, OSPF-TE, etc. As example, have a look at draft-farrkingel-pce-abno-architecture-07 Section 2.3.1.8.1. And I might be wrong, 

[Qin]:Yes, you misunderstand. Just like how we gather ALTO information from various sources, TED can be built by gathering information not only from routing protocol, but also from NMS/OSS. However most of TED information come from routing protocols, e.g., IGP, BGP.
Section 2.3.1.8.1 Traffic Engineering Database (TED)of draft-farrkingel-pce-abno-architecture-07 says:
"
   The Traffic Engineering Database (TED) is a data store of topology
   information about a network that may be enhanced with capability
   data (such as metrics or bandwidth capacity) and active status
   information (such as up/down status or residual unreserved
   bandwidth).

   The TED may be built from information supplied by the network or
   from data (such as inventory details) sourced through the NMS/OSS.

"
What The first paragraph says is bandwidth capacity, redisual unreserved bandwidth can be gathered by using attributes defined in draft-ietf-isis-te-metric-extensions or draft-ietf-ospf-te-metric-extensions.

What the second paragraph says inventory information(e.g., Enterprise Number,or other vendor specific information) can be gathered using SNMP MIB,
And then NMS/OSS as path computation client communicate these information with path computation element(PCE).

but the SDN research community argues right now that one build traffic engineering solutions without using RSVP-TE, ISIS-TE, OSPF-TE, BGP-TE, etc.

[Qin]:sure, SDN can be used to deal with non MPLS-TE case, e.g., IP network case.

[Qin]: Yes, as I said earlier, traffic engineering information can be gathered from different sources just like how ALTO server gather information from
Different sources.

 

[Qin]: I think you can separate "Monitoring the Performance of ALTO" part from "Monitoring the impact of ALTO" section, that will address my comments.

Michael

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 25 Feb 2014 20:53:26 +0000

Hi,

In order to sort out this discussion regarding terminology, I have now ...

1. Reverted the main headline of the section to what it was in -08 and before ("Monitoring ALTO"), 

2. Aligned the wording with draft-ietf-alto-protocol-26 Section 16.1.4 and Section 16.2.5, under the assumption that the content and wording therein is WG consensus

3. Added the desired sentence on third parties and trust issues to Section 3.4.1

4. Separated the discussion of "impact" and "performance" into two sections, as suggested in this thread

Below is a copy of the updated, pre-10 text.

Please let me know if I missed something.

Thanks

Michael

*** draft-ietf-alto-deployments-010pre ***

3.4.  Monitoring ALTO

3.4.1.  Impact and Observation on Network Operation

   ALTO presents a new opportunity for managing network traffic by
   providing additional information to clients.  In particular, the
   deployment of an ALTO Server may shift network traffic patterns, and
   the potential impact to network operation can be large.  An ISP
   providing ALTO may want to assess the benefits of ALTO as part of the
   management and operations (cf. [I-D.ietf-alto-protocol]).  For
   instance, the ISP might be interested in understanding whether the
   provided ALTO maps are effective, and in order to decide whether an
   adjustment of the ALTO configuration would be useful.  Such insight
   can be obtained from a monitoring infrastructure.  An NSP offering
   ALTO could consider the impact on (or integration with) traffic
   engineering and the deployment of a monitoring service to observe the
   effects of ALTO operations.  The measurement of impacts can be
   challenging because ALTO-enabled applications may not provide related
   information back to the ALTO Service Provider.

   To construct an effective monitoring infrastructure, the ALTO Service
   Provider should decide how to monitor the performance of ALTO and
   identify and deploy data sources to collect data to compute the
   performance metrics.  In certain trusted deployment environments, it
   may be possible to collect information directly from ALTO clients.
   It may also be possible to vary or selectively disable ALTO guidance
   for a portion of ALTO clients either by time, geographical region, or
   some other criteria to compare the network traffic characteristics
   with and without ALTO.  Monitoring an ALTO service could also be
   realized by third parties.  In this case, insight into ALTO data may
   require a trust relationship between the monitoring system operator
   and the network service provider offering an ALTO service.

   The required monitoring depends on the network infrastructure and the
   use of ALTO, and an exhaustive description is outside the scope of
   this document.

3.4.2.  Measurement of the Impact

   ALTO realizes an interface between the network and applications.
   This implies that an effective monitoring infrastructure may have to
   deal with both network and application performance metrics.  This
   document does not comprehensively list all performance metrics that
   could be relevant, nor does it formally specify metrics.

   The impact of ALTO can be classified regarding a number of different
   criteria:

   o  Total amount and distribution of traffic: ALTO enables ISPs to
      influence and localize traffic of applications that use the ALTO
      service.  An ISP may therefore be interested in analyzing the
      impact on the traffic, i.e., whether network traffic patterns are
      shifted.  For instance, if ALTO shall be used to reduce the inter-
      domain P2P traffic, it makes sense to evaluate the total amount of
      inter-domain traffic of an ISP.  Then, one possibility is to study
      how the introduction of ALTO reduces the total inter-domain
      traffic (inbound and/our outbound).  If the ISPs intention is to
      localize the traffic inside his network, the network-internal
      traffic distribution will be of interest.  Effectiveness of
      localization can be quantified in different ways, e.g., by the
      load on core routers and backbone links, or by considering more
      advanced effects, such as the average number of hops that traffic
      traverses inside a domain.

   o  Application performance: The objective of ALTO is improve

   Of potential interest can also be the share of applications or
   customers that actually use an offered ALTO service, i.e., the
   adoption of the service.

   Monitoring statistics can be aggregated, averaged, and normalized in
   different ways.  This document does not mandate specific ways how to
   calculate metrics.

3.4.3.  System and Service Performance

   A number of interesting parameters can be measured at the ALTO
   server.  [I-D.ietf-alto-protocol] suggests certain ALTO-specific
   metrics to be monitored:

   o  Requests and responses for each service listed in a Information
      Directory (total counts and size in bytes).

   o  CPU and memory utilization

   o  ALTO map updates

   o  Number of PIDs

   o  ALTO map sizes (in-memory size, encoded size, number of entries)

   This data characterizes the workload, the system performance as well
   as the map data.  Obviously, such data will depend on the
   implementation and the actual deployment of the ALTO service.

3.4.4.  Monitoring Infrastructure

   Understanding the impact of ALTO may require interaction between
   different systems, operating at different layers.  Some information
   discussed in the preceding sections is only visible to an ISP, while
   application-level performance can hardly be measured inside the
   network.  It is possible that not all information of potential
   interest can directly be measured, either because no corresponding
   monitoring infrastructure or measurement method exists, or because it
   is not easily accessible.

   One way to quantify the benefit of deploying ALTO is to measure
   before and after enabling the ALTO service.  In addition to passive
   monitoring, some data could also be obtained by active measurements,
   but due to the resulting overhead, the latter should be used with
   care.  Yet, in all monitoring activities an ALTO service provider has
   to take into account that ALTO Clients are not bound to ALTO Server
   guidance as ALTO is only one source of information, and any
   measurement result may thus be biased.

   Potential sources for monitoring the use of ALTO include:

   o  Network Operations, Administration, and Maintenance (OAM) systems:
      Many ISPs deploy OAM systems to monitor the network traffic, which
      may have insight into traffic volumes, network topology, and
      bandwidth information inside the management area.  Data can be
      obtained by SNMP, Netflow, IP Flow Information Export (IPFIX),
      syslog, etc.

   o  Applications/clients: Relevant data could be obtained by
      instrumentation of applications.

   o  ALTO server: If available, log files or other statistics data
      could be analyzed.

   o  Other application entities: In several use cases, there are other
      application entities that could provide data as well.  For
      instance, there may be centralized log servers that collect data.

   In many ALTO use cases some data sources are located within an ISP
   while some other data is gathered at application level.  Correlation
   of data would require a collaboration agreement between the ISP and
   an application owner, including agreements of data interchange
   formats, methods of delivery, etc.  In practice, such a collaboration
   may not be possible in all use cases of ALTO, because the monitoring
   data can be sensitive, and because the interacting entities may have
   different priorities.  Details of how to build an over-arching
   monitoring system for evaluating the benefits of ALTO are outside the
   scope of this memo.