Subject: Re: [arch-d] [Apn] Questions for APN: Q#5

From: Lars Eggert <[email-masked]>
Date: Tue, 22 Sep 2020 09:44:45 +0300

Hi,

On 2020-9-22, at 9:02, [email-masked] wrote:
> [Shuping] Again APN is aimed to work within a controlled and limited operators’ network domain not for Internet.

that significantly limits the attractiveness of this proposal to application vendors. They can either buy into APN and ship an app that only works in some (initially, very few to none) operator networks. Or, they can ship a slightly less optimal app that works on the entire Internet with its billions of users.

More broadly, I'd like to point out that during the entire history of the Internet there were application classes that the deployed networks at the time were struggling to support. There were always claims that something like APN was needed, i.e, solutions that were intending to improve network performance and quality but that were also adding significant complexity and often required application changes.

What always happened so far was that Moore's law solved these problems, by improving the performance and quality of the general Internet so that best effort was sufficient to support all these "special" applications a few years down the road.

The same will happen with the use cases presented to motivate the need for APN. The general Internet may struggle to support them now, but in five years or so - which is about the time horizon for any APN enablement of any operator network to happen - these will just work.

Thanks,
Lars

---

From: Christian Huitema <[email-masked]>
Date: Tue, 22 Sep 2020 08:29:12 -0700

> More broadly, I'd like to point out that during the entire history of the Internet there were application classes that the deployed networks at the time were struggling to support. There were alway…
Yes indeed. In the 90's the argument was around video conferencing,
which "obviously" could not work on a plain best effort service. And
then we got Skype.
> What always happened so far was that Moore's law solved these problems, by improving the performance and quality of the general Internet so that best effort was sufficient to support all these "spe…
Yes. Indeed the whole history of wireless is driven by Moore's law. For
example, the theory of MIMO has been known for some time, put the
application and the use of large number of input and output is only
possible because of progress in electronics.
> The same will happen with the use cases presented to motivate the need for APN. The general Internet may struggle to support them now, but in five years or so - which is about the time horizon for…

More bandwidth solves a lot of these issues. But we can also expect
improvement in transport protocols and in queue management. Active Queue
Management techniques can isolate different data streams and prevent
stupid issues like "my daughter playing video games is degrading my
VPN". New congestion control protocols like BBR actively manage queuing
delays, resulting in significantly lower latency overall. And then
application just get smarter in their use of caches, redundancy, gap
filling, etc. All that is guaranteed to improve over time.

-- Christian Huitema

---

From: "Yiannis Yiakoumis" <[email-masked]>
Date: Fri, 25 Sep 2020 23:22:31 +0000

late in the discussion, but wanted to share some thoughts:

One of the first successful proof-points for SDN was Google claiming ( https://cs144.github.io/handouts/cs144_mogul_2019_nov.pdf ) that they moved their WAN utilization to ~100%, instead of over-provisioning. It saved them lots of money. Two of the design principles that led them there were

i) moving from "all packets are equally important" to "allocate resources based on application-level priorities" and

ii) moving from "TCP flows regulated by "fair share" mechanisms" to "measure demands, and shape flows at the endpoints".

I can imagine offering low-latency SLA for 1% of the traffic (say video calls), is much cheaper and easier than doing this for all Internet traffic (with the cost varying whether you are in Europe/US or an emerging market, rural or urban, a WISP, a low-orbit satellite network, etc). *Could anyone from operator-land comment on the economics of oversubscription/overprovision/SLAs in the mobile world?*

(btw: we already have SLAs today --- VoLTE is IP traffic routed through dedicated radio and wired resources and has materially better network quality and reliability than plain VoIP, and when you sign-up for a broadband connection you can get 10Mbps/100Mbps/1Gbps etc).

If over-provisioning wins (in 10 years) it will likely be the outcome of the network community's  failure to find a good solution for traffic differentiation, rather than a verdict that these use cases were not important enough.

FWIW: there are many issues to debate around fine-grained traffic differentiation beyond overprovisioning (like security, scalability, privacy, net neutrality), and there are different approaches one could take (APN and network tokens have different design principles and priorities on these trade-offs).

Best,

Yiannis

=====================
Yiannis Yiakoumis
Co-Founder & CEO
https://selfienetworks.com | +1-650-644-7857

On Tue, Sep 22, 2020 at 8:29 AM, Christian Huitema < [email-masked] > wrote:

>> On 2020-9-22, at 9:02, zhangs366@ chinaunicom. cn (
>> [email-masked] ) wrote:
>> 
>> 
>>> 
>>> 
>>> [Shuping] Again APN is aimed to work within a controlled and limited
>>> operators’ network domain not for Internet.
>>> 
>>> 
>> 
>> 
>> 
>> that significantly limits the attractiveness of this proposal to
>> application vendors. They can either buy into APN and ship an app that
>> only works in some (initially, very few to none) operator networks. Or,
>> they can ship a slightly less optimal app that works on the entire
>> Internet with its billions of users.
>> 
>> 
>> 
>> More broadly, I'd like to point out that during the entire history of the
>> Internet there were application classes that the deployed networks at the
>> time were struggling to support. There were always claims that something
>> like APN was needed, i.e, solutions that were intending to improve network
>> performance and quality but that were also adding significant complexity
>> and often required application changes.
>> 
>> 
> 
> 
> 
> Yes indeed. In the 90's the argument was around video conferencing, which
> "obviously" could not work on a plain best effort service. And then we got
> Skype.
> 
> 
>> 
>> 
>> What always happened so far was that Moore's law solved these problems, by
>> improving the performance and quality of the general Internet so that best
>> effort was sufficient to support all these "special" applications a few
>> years down the road.
>> 
>> 
> 
> 
> 
>> The same will happen with the use cases presented to motivate the need for
>> APN. The general Internet may struggle to support them now, but in five
>> years or so - which is about the time horizon for any APN enablement of
>> any operator network to happen - these will just work.
>> 
>> 
> 
> 
> 
> stupid issues like "my daughter playing video games is degrading my VPN".
> New congestion control protocols like BBR actively manage queuing delays,
> resulting in significantly lower latency overall. And then application
> just get smarter in their use of caches, redundancy, gap filling, etc. All
> that is guaranteed to improve over time.
> 
> 
> 
> -- Christian Huitema
> 
> 
> 
> --
> Network-tokens mailing list
> Network-tokens@ ietf. org ( [email-masked] )
> https:/ / www. ietf. org/ mailman/ listinfo/ network-tokens (
> https://www.ietf.org/mailman/listinfo/network-tokens )
> 
> 
>

---

From: "Pengshuping (Peng Shuping)" <[email-masked]>
Date: Tue, 29 Sep 2020 01:36:16 +0000

Well said, Yiannis. Thank you!

Best regards,
Shuping

From: Network-tokens [mailto:[email-masked]] On Behalf Of Yiannis Yiakoumis
Sent: Saturday, September 26, 2020 7:23 AM
To: Christian Huitema <[email-masked]>
Cc: [email-masked]; 曹畅 <[email-masked]>; [email-masked]; [email-masked]; apn <[email-masked]>; Lars Eggert <[email-masked]>; Pengshuping (Peng Shuping) <[email-masked]>
Subject: Re: [Network-tokens] [Apn] [arch-d] Questions for APN: Q#5

One of the first successful proof-points for SDN was Google claiming<https://cs144.github.io/handouts/cs144_mogul_2019_nov.pdf> that they moved their WAN utilization to ~100%, instead of over-provisioning. It saved them lots of money. Two of the design principles that led them there were
i) moving from "all packets are equally important" to "allocate resources based on application-level priorities" and
ii) moving from "TCP flows regulated by "fair share" mechanisms" to "measure demands, and shape flows at the endpoints".

I can imagine offering low-latency SLA for 1% of the traffic (say video calls), is much cheaper and easier than doing this for all Internet traffic (with the cost varying whether you are in Europe/US or an emerging market, rural or urban, a WISP, a low-orbit satellite network, etc). Could anyone from operator-land comment on the economics of oversubscription/overprovision/SLAs in the mobile world?

Best,
Yiannis

=====================
Yiannis Yiakoumis
Co-Founder & CEO

On Tue, Sep 22, 2020 at 8:29 AM, Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

Hi,

On 2020-9-22, at 9:02, [email-masked]<mailto:[email-masked]> wrote:

Yes indeed. In the 90's the argument was around video conferencing, which "obviously" could not work on a plain best effort service. And then we got Skype.

Yes. Indeed the whole history of wireless is driven by Moore's law. For example, the theory of MIMO has been known for some time, put the application and the use of large number of input and output is only possible because of progress in electronics.

More bandwidth solves a lot of these issues. But we can also expect improvement in transport protocols and in queue management. Active Queue Management techniques can isolate different data streams and prevent stupid issues like "my daughter playing video games is degrading my VPN". New congestion control protocols like BBR actively manage queuing delays, resulting in significantly lower latency overall. And then application just get smarter in their use of caches, redundancy, gap filling, etc. All that is guaranteed to improve over time.

-- Christian Huitema

--
Network-tokens mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/network-tokens

---

From: 刘 鹏 <[email-masked]>
Date: Wed, 30 Sep 2020 01:14:03 +0000

Hi Yiannis,

I agree that offering low-latency SLA for 1% of the traffic is easier and cheaper than all internet traffic. It's better to guarantee the SLA of some critical traffic first. However, it should also strike a balance between benefits and costs.

Some new technologies such as network slice, APN may help to solve the problem and meet the demand of fine-grained traffic. Therefore, I think fine-grained traffic requirements are necessary, which can help to save network resources, and over configuration may waste network resources.

Regards,
Peng (China Mobile)

Peng Liu | 刘鹏
China Mobile | 移动研究院
mobile phone：13810146105
email:  [email-masked]/[email-masked]

________________________________
发件人: Apn <[email-masked]> 代表 Yiannis Yiakoumis <[email-masked]>
发送时间: 2020年9月26日 7:22
收件人: Christian Huitema <[email-masked]>
抄送: [email-masked] <[email-masked]>; 曹畅 <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; apn <[email-masked]>; Lars Eggert <[email-masked]>; pengshuping <[email-masked]>
主题: Re: [Apn] [Network-tokens] [arch-d] Questions for APN: Q#5

Best,
Yiannis

=====================
Yiannis Yiakoumis
Co-Founder & CEO

Hi,

-- Christian Huitema

---

From: Lars Eggert <[email-masked]>
Date: Wed, 30 Sep 2020 10:28:01 +0300

Hi,

On 2020-9-26, at 2:22, Yiannis Yiakoumis <[email-masked]> wrote:
> One of the first successful proof-points for SDN was Google claiming that they moved their WAN utilization to ~100%, instead of over-provisioning. It saved them lots of money. Two of the design principles that led them there were

I think these claims were made for Google's private B4 network, right? AFAIK that is a *very* different network from the public Internet. From what I recall, each transfer over B4 is scheduled over the course of a day based on various flow properties that are required to be specified, following some giant minmax-like computation.

> I can imagine offering low-latency SLA for 1% of the traffic (say video calls), is much cheaper and easier than doing this for all Internet traffic (with the cost varying whether you are in Europe/…

Sure. But that could be done with existing mechanisms, like DSCP marking. Remarking/bleaching does happen as traffic gets routed between networks, but since the stated APN use case here is within an operator network, that is under that operator's control.

> (btw: we already have SLAs today --- VoLTE is IP traffic routed through dedicated radio and wired resources and has materially better network quality and reliability than plain VoIP, and when you s…

Yup. And that happens without APN in the picture.

> If over-provisioning wins (in 10 years) it will likely be the outcome of the network community's  failure to find a good solution for traffic differentiation, rather than a verdict that these use c…

The problem isn't that we don't have good solutions for traffic differentiation. We have tons. The problem is that none of the existing ones have seen much deployment - but not because they don't work, because of other issues. I don't see APN to offer materially different/better deployment incentives than what we already have, and so IMO it would share the same fate.

Thanks,
Lars

---

From: "Yiannis Yiakoumis" <[email-masked]>
Date: Wed, 30 Sep 2020 18:07:44 +0000

Hi Lars,

On Wed, Sep 30, 2020 at 12:28 AM, Lars Eggert < [email-masked] > wrote:

> 
> 
> 
> Hi,
> 
> 
> 
> 
> On 2020-9-26, at 2:22, Yiannis Yiakoumis < yiannis@ selfienetworks. com (
> [email-masked] ) > wrote:
> 
> 
> 
>> 
>> 
>> One of the first successful proof-points for SDN was Google claiming that
>> they moved their WAN utilization to ~100%, instead of over-provisioning.
>> It saved them lots of money. Two of the design principles that led them
>> there were
>> i) moving from "all packets are equally important" to "allocate resources
>> based on application-level priorities" and ii) moving from "TCP flows
>> regulated by "fair share" mechanisms" to "measure demands, and shape flows
>> at the endpoints".
>> 
>> 
> 
> 
> 
> I think these claims were made for Google's private B4 network, right?
> AFAIK that is a *very* different network from the public Internet. From
> what I recall, each transfer over B4 is scheduled over the course of a day
> based on various flow properties that are required to be specified,
> following some giant minmax-like computation.
> 
> 
> 
> 

Yes, this is for B4. While the network is different, the idea that you can build a much more cost/performance efficient WAN still stands.

> 
> 
>> 
>> 
>> I can imagine offering low-latency SLA for 1% of the traffic (say video
>> calls), is much cheaper and easier than doing this for all Internet
>> traffic (with the cost varying whether you are in Europe/US or an emerging
>> market, rural or urban, a WISP, a low-orbit satellite network, etc). Could
>> anyone from operator-land comment on the economics of
>> oversubscription/overprovision/SLAs in the mobile world?
>> 
>> 
>> 
> 
> 
> 
> Sure. But that could be done with existing mechanisms, like DSCP marking.
> Remarking/bleaching does happen as traffic gets routed between networks,
> but since the stated APN use case here is within an operator network, that
> is under that operator's control.
> 
> 
> 
> 

DSCP marking works within a single administrative domain where you have full control of endpoints and the network. The moment you want to incorporate end user devices and or third-party servers that sit out of the network, it doesn't work well. My comments are not per APN usecases per se, but rather the bigger traffic differentiation picture. (if I understand correctly, APN tries to limit things within an administrative domain to avoid having to deal with trust, security, neutrality, etc).

> 
> 
>> 
>> 
>> (btw: we already have SLAs today --- VoLTE is IP traffic routed through
>> dedicated radio and wired resources and has materially better network
>> quality and reliability than plain VoIP, and when you sign-up for a
>> broadband connection you can get 10Mbps/100Mbps/1Gbps etc).
>> 
>> 
>> 
> 
> 
> 

This happens by having vertical integration for a specific use case (phone app, phone driver, network control plane, network data plane ). I would really like to get VoLTE's reliability into my Zoom/FaceTime/WhatsApp calls, but it is not possible.

> 
> 
>> 
>> 
>> If over-provisioning wins (in 10 years) it will likely be the outcome of
>> the network community's failure to find a good solution for traffic
>> differentiation, rather than a verdict that these use cases were not
>> important enough.
>> 
>> 
>> 
> 
> 
> 
> The problem isn't that we don't have good solutions for traffic
> differentiation. We have tons. The problem is that none of the existing
> ones have seen much deployment - but not because they don't work, because
> of other issues. I don't see APN to offer materially different/better
> deployment incentives than what we already have, and so IMO it would share
> the same fate.
> 
> 
> 
> 

I would disagree here. We do have several traffic differentiation deployments that use DPI and application signatures to map traffic into SLAs at a fine granularity. It is largely inefficient cost and performance-wise, error-prone, and violates privacy and neutrality in certain occasions. But that's the one solution mostly used, for the lack of better alternatives.

Best,

Yiannis

> 
> 
> 
> Thanks,
> Lars
> 
> 
>> 
>> 
>> FWIW: there are many issues to debate around fine-grained traffic
>> differentiation beyond overprovisioning (like security, scalability,
>> privacy, net neutrality), and there are different approaches one could
>> take (APN and network tokens have different design principles and
>> priorities on these trade-offs).
>> 
>> 
>> 
> 
> 
>

---

From: Christian Huitema <[email-masked]>
Date: Wed, 30 Sep 2020 13:58:09 -0700

On 9/30/2020 11:07 AM, Yiannis Yiakoumis wrote:
> Hi Lars,
>
> On Wed, Sep 30, 2020 at 12:28 AM, Lars Eggert <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi,
>
>     On 2020-9-26, at 2:22, Yiannis Yiakoumis
>     <[email-masked] <mailto:[email-masked]>>
>     wrote:
>
>         One of the first successful proof-points for SDN was Google
>         claiming that they moved their WAN utilization to ~100%,
>         instead of over-provisioning. It saved them lots of money. Two
>         of the design principles that led them there were
>         i) moving from "all packets are equally important" to
>         "allocate resources based on application-level priorities" and
>         ii) moving from "TCP flows regulated by "fair share"
>         mechanisms" to "measure demands, and shape flows at the
>         endpoints".
>
>     I think these claims were made for Google's private B4 network,
>     right? AFAIK that is a *very* different network from the public
>     Internet. From what I recall, each transfer over B4 is scheduled
>     over the course of a day based on various flow properties that are
>     required to be specified, following some giant minmax-like
>     computation.
>
>
> Yes, this is for B4. While the network is different, the idea that you
> can build a much more cost/performance efficient WAN still stands.

What do you call WAN? Until now, APN was proposed for "limited domains",
which are typically access networks. WAN would imply going further, to
long distance networks used for connections between access providers. Is
that really what you have in mind?

-- Christian Huitema