Subject: [alto] Parameterized ALTO cost-types

From: Wendy Roome <[email-masked]>
Date: Tue, 28 Jul 2015 09:33:26 -0400

Recently, draft-wydrych-alto-paths-00 &
draft-gao-alto-routing-state-abstraction-00 have proposed extending ALTO
with path-based costs. I'm going to outline a considerably simpler way to
accomplish the same goal. I would like all of you -- and especially the
authors of those drafts! -- to let me know what those extensions provide
that this doesn't.

The Problem:
The ALTO cost model assumes costs depend only on the source & destination
addresses. But modern networks route on other attributes (protocol,
dest-port, diffserv, etc) as well as destination address. The result is
that the cost for downloading a file may depend on the protocol: rtp,
tcp/port80, tcp/other-port, etc.

What's worse, those protocol-dependent costs might not "track." That is,
if you stream a video via rtp, ServerA might be faster than ServerB, but
if you download it via http, ServerB might be faster.

The current ALTO cost model cannot convey those distinctions.

Observation #1:
While paths may be the cause of the cost differences, ALTO clients -- end
user applications -- do not care about the paths their packets take. ALTO
clients just care about the end-to-end costs.

Feel free to describe a use-case that contradicts this claim!

I realize that multi-flow clients need to know about paths, to determine
how much the routes to several different endpoints overlap. But those are
not the clients I am talking about here.

Observation #2:
Clients who are concerned about protocol-dependent costs are only
interested a small number of different protocol/port/etc combinations.
That is, an ALTO client might be interested in the costs for downloading
via rtp vs http. But that is all; the client does not care about the costs
for other protocols & ports.

Again, feel free to point out a use-case that contradicts that claim.

My Proposal:
Instead of introducing a path abstraction, suppose we extend the cost-type
with parameters that describe the packet type, such as protocol, src-port,
dst-port & diffserv. Then a client request for a Filtered Cost Map might
look like this:

   {
      "cost-type" : {
         "cost-metric" : "routingcost",
         "cost-mode" : "numerical",
         "src-port" : 80,
         ³protocol" : 6
      },
      "pids" : {
         "srcs" : [ "PID1", "PID2", "PID3" ],
         "dsts" : [ "PID1" ]
      }
   }

The ALTO server returns the numerical routing costs from the srcs to the
dsts, for packets with the indicated port & protocol. If there is no
special route for that packet flavor, the ALTO server simply returns the
general routingcost for the src & dst.

If a client wants costs for a small number of different protocols, the
client can use the multi-cost extension, draft-ietf-alto-multi-cost-00.

My Question:
What do the proposed path-based cost extensions provide that this does not?

	- Wendy Roome

---

From: "Bertz, Lyle T [CTO]" <[email-masked]>
Date: Tue, 28 Jul 2015 15:00:33 +0000

Wendy,

A few thoughts.

With respect to Observation #1 a client *does* care about the congestion state of a single path in a non-multipath scenario.  It is for this very reason in the mobile world that the phones 'Connection Optimizers' and radio access network packet schedules begin looking at other radio access technology or channel alternatives.

With respect to Observation #2 is correct if we consider 'small number' say less than 6.   I know that is a nit but it I find it to be all about perspective.

I like your proposal but it is often not enough to consider the top protocol and you need to look at the stack itself.  For instance,

http/tcp
http/mptcp
and http/dash (which can include eMBMS on a LTE network)
http/quic

yield wildly different results in terms of uplink performance and, in turn downlink performance at high speeds due to the ACK speeds and windowing.

Finally, the path extensions make sense.  I liken PID to PID metrics at the measurements of one link so we can understand the cost of egressing a PID and entering another.  This is no different that L2/3 QoS.  However, the next question is what is the cost of entering PID A and traversing internal links to then make the connection to PID B.  It is not the cost of PID A to B over a L2/L3 link.   How would we represent the difference?

We had the same issue 20 years ago for QoS in general.   Layer 2 tech does well for QoS across Layer 2 but did not give priority to QoS within the router itself - hence DiffServ.   I consider Cost maps the Layer 2 of metrics but we are still missing the internal one.  If the cost of transiting a PID to any other PID is constant than Cost (PIDA => PIDA) = X is sufficient.  If not then we have a representation issue that ALTO does not currently address.

Finally there is a question of hierarchical aggregation when presenting a metric value.   How would we measure the Cost from PID A to PID B when there are 3 endpoints in each PID fully meshed to each other but the links have slightly different measurements due to our ability to precisely measure the metric which may lead to several significant digits.

Which path or paths represents the cost in Costmap?  How did this get noted as the summary?  How can this be tracked when an Operator needs to understand how the server came to this value?

I am not saying path representation answer all or any of these questions but I do understand the value it can provide in terms of giving further information.

Also, as far as metrics go packet-count, flow-count and ECN packets observed over a time window are very useful to operators and may be useful in ALTO.   It was such a driver that changes were made to Diameter to address this -  https://tools.ietf.org/html/draft-ietf-dime-congestion-flow-attributes-02

Lyle

-----Original Message-----
From: alto [mailto:[email-masked]] On Behalf Of Wendy Roome
Sent: Tuesday, July 28, 2015 8:33 AM
To: IETF ALTO <[email-masked]>
Subject: [alto] Parameterized ALTO cost-types

Recently, draft-wydrych-alto-paths-00 &
draft-gao-alto-routing-state-abstraction-00 have proposed extending ALTO with path-based costs. I'm going to outline a considerably simpler way to accomplish the same goal. I would like all of you -- and especially the authors of those drafts! -- to let me know what those extensions provide that this doesn't.

The Problem:
The ALTO cost model assumes costs depend only on the source & destination addresses. But modern networks route on other attributes (protocol, dest-port, diffserv, etc) as well as destination address. The result is that the cost for downloading a file may depend on the protocol: rtp, tcp/port80, tcp/other-port, etc.

What's worse, those protocol-dependent costs might not "track." That is, if you stream a video via rtp, ServerA might be faster than ServerB, but if you download it via http, ServerB might be faster.

Observation #1:
While paths may be the cause of the cost differences, ALTO clients -- end user applications -- do not care about the paths their packets take. ALTO clients just care about the end-to-end costs.

I realize that multi-flow clients need to know about paths, to determine how much the routes to several different endpoints overlap. But those are not the clients I am talking about here.

Observation #2:
Clients who are concerned about protocol-dependent costs are only interested a small number of different protocol/port/etc combinations.
That is, an ALTO client might be interested in the costs for downloading via rtp vs http. But that is all; the client does not care about the costs for other protocols & ports.

My Proposal:
Instead of introducing a path abstraction, suppose we extend the cost-type with parameters that describe the packet type, such as protocol, src-port, dst-port & diffserv. Then a client request for a Filtered Cost Map might look like this:

   {
      "cost-type" : {

The ALTO server returns the numerical routing costs from the srcs to the dsts, for packets with the indicated port & protocol. If there is no special route for that packet flavor, the ALTO server simply returns the general routingcost for the src & dst.

If a client wants costs for a small number of different protocols, the client can use the multi-cost extension, draft-ietf-alto-multi-cost-00.

My Question:

- Wendy Roome

_______________________________________________
alto mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/alto

________________________________

This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.

---

From: Wendy Roome <[email-masked]>
Date: Tue, 28 Jul 2015 14:16:26 -0400

Lyle,

Thanks for your comments. This is exactly the sort of discussion I hoped
to start. My comments in-line.

On 07/28/2015, 11:00, "Bertz, Lyle T [CTO]" <[email-masked]>
wrote:

>Wendy,
>
>A few thoughts.
>
>With respect to Observation #1 a client *does* care about the congestion
>state of a single path in a non-multipath scenario.  It is for this very
>reason in the mobile world that the phones 'Connection Optimizers' and
>radio access network packet schedules begin looking at other radio access
>technology or channel alternatives.

Can the congestion state and other problems be represented just by
increased cost? Or does the client really need to know the details about
the congestion?

Also, does ³other radio access technology or channel alternatives² mean
that the client adjusts the wireless device¹s access point, channel, power
level, etc? I don¹t believe those fit into the current ALTO network model,
so we would need extensions to get those concepts into ALTO.

>With respect to Observation #2 is correct if we consider 'small number'
>say less than 6.   I know that is a nit but it I find it to be all about
>perspective.

6 should be fine. I really meant ³small enough to be feasible with the
multi-cost extension².

>I like your proposal but it is often not enough to consider the top
>protocol and you need to look at the stack itself.  For instance,
>
>http/tcp
>http/mptcp
>yield wildly different results in terms of uplink performance and, in
>turn downlink performance at high speeds due to the ACK speeds and
>windowing.

Can those be represented as additional parameters on a cost-type, or would
that require a more fundamental change?

>Finally, the path extensions make sense.  I liken PID to PID metrics at
>the measurements of one link so we can understand the cost of egressing a
>PID and entering another.  This is no different that L2/3 QoS.  However,
>the next question is what is the cost of entering PID A and traversing
>internal links to then make the connection to PID B.  It is not the cost
>of PID A to B over a L2/L3 link.   How would we represent the difference?

Yes, but I think you are assuming that links are always between PIDs. I
think it is more likely that the links between PIDs will involve non-PID
entities (Abstract Network Elements). Eg, paths might be PID1 => link1 =>
router1 => link2 => PID2, and PID3=>link3=>router1=>link4=>PID4. That is,
the paths from PID1=>PID2 and PID3=>PID4 share a common element, router.
Since the router doesn¹t have any user endpoints, there is no need for it
to be a PID.

>We had the same issue 20 years ago for QoS in general.   Layer 2 tech
>does well for QoS across Layer 2 but did not give priority to QoS within
>the router itself - hence DiffServ.   I consider Cost maps the Layer 2 of
>metrics but we are still missing the internal one.  If the cost of
>transiting a PID to any other PID is constant than Cost (PIDA => PIDA) =
>X is sufficient.  If not then we have a representation issue that ALTO
>does not currently address.
>
>Finally there is a question of hierarchical aggregation when presenting a
>metric value.   How would we measure the Cost from PID A to PID B when
>there are 3 endpoints in each PID fully meshed to each other but the
>links have slightly different measurements due to our ability to
>precisely measure the metric which may lead to several significant digits.

In that case, should those 3 endpoints really be in the same PID? I always
thought an informal definition of a PID was a set of endpoints such the
cost between any two endpoints in the PID was significantly less than the
cost between either endpoint and all endpoints outside the PID.

>Which path or paths represents the cost in Costmap?  How did this get
>noted as the summary?  How can this be tracked when an Operator needs to
>understand how the server came to this value?
>
>I am not saying path representation answer all or any of these questions
>but I do understand the value it can provide in terms of giving further
>information.
>
>Also, as far as metrics go packet-count, flow-count and ECN packets
>observed over a time window are very useful to operators and may be
>useful in ALTO.   It was such a driver that changes were made to Diameter
>to address this - 
>https://tools.ietf.org/html/draft-ietf-dime-congestion-flow-attributes-02
>
>Lyle

One more consideration: what about ECS? I think ECS is likely to be the
most popular ALTO service. After all, clients are really interested in
costs between endpoints; PIDs are just a necessary abstraction. The
parameterized cost-type model fits nicely into ECS. Unless I missed
something, I think path-based costs won¹t work with ECS.

Here is one additional concern I have with path-based costs. My
understanding is that ALTO clients are end-user applications, and they
cannot select paths directly. Instead, clients select the output interface
(if the device has more than one), port, protocol, etc. The OS & network
then select the path based on that. So path-based costs require an
additional mechanism to tell the client how the network maps flows to
paths. That is possible, of course, but I haven¹t seen a concrete example
yet.

Thanks again for you comments and for starting the discussion!

	- Wendy Roome

---

From: "Bertz, Lyle T [CTO]" <[email-masked]>
Date: Wed, 29 Jul 2015 18:01:06 +0000

Wendy,

Comments below.

> From: Wendy Roome [mailto:[email-masked]]
> Sent: Tuesday, July 28, 2015 1:16 PM
> To: Bertz, Lyle T [CTO] <[email-masked]>; IETF ALTO
> <[email-masked]>
> Subject: Re: Parameterized ALTO cost-types
>
> Lyle,
>
> Thanks for your comments. This is exactly the sort of discussion I hoped to
> start. My comments in-line.
>
> >With respect to Observation #1 a client *does* care about the
> >congestion state of a single path in a non-multipath scenario.  It is
> >for this very reason in the mobile world that the phones 'Connection
> >Optimizers' and radio access network packet schedules begin looking at
> >other radio access technology or channel alternatives.
>
> increased cost? Or does the client really need to know the details about the
> congestion?

In order to maximize IETF standards in place, Congestion is best measured by the Unidirectional CE (ECN Congestion Experienced) count and the reverse path count ECE (Echoed CE).  When you combine this with other metrics such as packet count and Flow count (AVPs we added to Diameter Filters) you can provide a profile of the congestion type, large fat flow congestion, web congestion, etc.   This is the best you can do and are direct counters from the IP and TCP information bases in the network's elements.   In other words we don't have to make anything up!  The other benefit is that it occurs in general outside the encryption mechanism and can be applied without the need for DPI.

ECN markings are unidirectional though.  Which brings back the question of directionality in the maps.  If you see a cost (a=>b) and (b=>a) you can infer that the measurements are unidirectional but what if in the same map I only see a b=>c then how do I interpret?  Should all maps assume the use of general edges, e.g. cost(a,b) = cost(b,a) and we can flag a map as say
unidirectional - if you see a b=>a and a=>b consider this in your calculations but if you only see b=>c *assume* the same value applies for c=>b
unidirectional-strict - if you see b=>a and no a=>b is present the value of a=>b is undefined.

I think that there is also a large assumption about what you mean by client.  Imo, there are many intermediate ALTO server layers that aggregate, reconcile and consolidate information on behalf of the ultimate consumers which I think of as ALTO clients.   These intermediate layers are themselves ALTO clients of the source information tier.  If this is not the case then we have a real problem.  In the coversations I have with suppliers of equipment it is quite clear that in heavy networking scenarios I will have no more than 8-10 racks of equipment managed per NFVI and, therefore, I will have 1 to 2 active controllers for every 8 to 10 racks of equipment.   In the case of a mobile carrier with say, 54 million users, I would expect to have 60 to 120 controllers.   I cannot expose that much information to simple ALTO clients.  I will clearly have multiple layers.

In the case of CDNI, or at least the draft that has been presented wrt ALTO for CDNI, the exposure of the public map from the carrier is a direct result of consolidating the routes across all of those controllers.

>
> Also, does ³other radio access technology or channel alternatives² mean that
> the client adjusts the wireless device¹s access point, channel, power level,
> etc? I don¹t believe those fit into the current ALTO network model, so we
> would need extensions to get those concepts into ALTO.

Once again, I think you are looking at ultimate consumers of ALTO information as Clients and I am concerned about intermediate tiers.  If you look at ultimate client / consumers you are correct.  For intermediate tiers the information is important.

> 6 should be fine. I really meant ³small enough to be feasible with the multi-
> cost extension².
>

Very good - 6 it is ;). It is here that we can point to consensus and ultimately the moment we made the wrong decision on the number and it should have been 7 but I will wait until we realize it before we point back to this thread :D - just kidding!

> >and http/dash (which can include eMBMS on a LTE network) http/quic
> >

I think adding it as another parameter works for me.   A mere string type with a separator between the protocols and then noting something like ' the protocol in the string is the protocol encapsulated by  subsequent protocols in the string ' should be, hopefully, clear enough.   This would avoid silly situations such as seeing 'http/tcp' (correct) and 'tcp/http' (incorrect but probably worthy of an april fool's paper) in the JSON information.  Wildcarding or some form of don't care about the lower layers, i.e. the string does not further specify  should also be considered.

> >the measurements of one link so we can understand the cost of egressing
> >a PID and entering another.  This is no different that L2/3 QoS.
> >However, the next question is what is the cost of entering PID A and
> >traversing internal links to then make the connection to PID B.  It is not the
> cost
> Yes, but I think you are assuming that links are always between PIDs. I think it
> is more likely that the links between PIDs will involve non-PID entities
> (Abstract Network Elements). Eg, paths might be PID1 => link1 =>
> Since the router doesn¹t have any user endpoints, there is no need for it to be
> a PID.
>

I believe you are right that I am making an assumption here in the sense that links between PIDs involve non-PID entities, however, for at least an operator with their own backbone I can say those entities are L1/L2 (optical) whose measures are well understood and as L2 transit (long haul) we would not expose in ALTO.   I understand that not every operator has this situation but some do.

If there is a router in a site we will put it in if we intend to understand costs.  This is the price of owning everything from the ground up as is the case in some operators.

> >transiting a PID to any other PID is constant than Cost (PIDA => PIDA)
> >= X is sufficient.  If not then we have a representation issue that
> >ALTO does not currently address.
> >
> thought an informal definition of a PID was a set of endpoints such the cost
> between any two endpoints in the PID was significantly less than the cost
> between either endpoint and all endpoints outside the PID.

Yes, especially if they are using multi-cost in the IGP or EGP.  This is a common issue in LAN for LAG and not just specific for ECMP as well.

> >noted as the summary?  How can this be tracked when an Operator needs
> >to understand how the server came to this value?
> >
> >I am not saying path representation answer all or any of these
> >questions but I do understand the value it can provide in terms of
> >giving further information.
> >
> One more consideration: what about ECS? I think ECS is likely to be the most
> popular ALTO service. After all, clients are really interested in costs between
> endpoints; PIDs are just a necessary abstraction. The parameterized cost-
> type model fits nicely into ECS. Unless I missed something, I think path-based
> costs won¹t work with ECS.

Once again I would agree that the ultimate consumer of ALTO would be looking for a simple answer.  The intermediate tiers would be more interested in how to arrive to a good decision.

> understanding is that ALTO clients are end-user applications, and they cannot
> select paths directly. Instead, clients select the output interface (if the device
> has more than one), port, protocol, etc. The OS & network then select the
> path based on that. So path-based costs require an additional mechanism to
> tell the client how the network maps flows to paths. That is possible, of
> course, but I haven¹t seen a concrete example yet.

I would agree that if you stick to the end-user application being the only consumer of applications then you are correct.  However, I question how good calculations are made in a scalable manner when there are multiple sources.   And finally, I think the draft "CDNI Footprint and Capabilities Advertisement using ALTO" may contradict this a bit.  I would suggest we ask one of the co-authors who *if* we happen to know them :D

> Thanks again for you comments and for starting the discussion!

No problem.