Subject: Re: [5gangip] Notes from today's meeting

From: "FIGURELLE, TERRY F" <[email-masked]>
Date: Tue, 17 Apr 2018 04:46:09 +0000

I’ve been absent for these threads because I was on short term disability (in ICU for 4 weeks with machines keeping me alive). It was a close call but I did pull thru. I may  not be so lucky next time (autoimmune disease similar to MS).

It feels like people are making this more complex that it needs to be.

Operators prefer open standards because it makes interop easier when you can point to standards or popularity based pseudo standards. It’s easier to refer to 3GPP, IETF, OMA, etc. when working with other businesses than it is to come up with something. Plenty of RFC’s were initially driven by Intranet not Internet but I don’t think it matters. When it comes to an entity like AT&T the difference between Intranet and Internet blurs.

Is Netflix an Intranet or Internet service?

Most of the large ISP’s have direct connections engineered to handle the volume of Netflix traffic and other high volume services. End users don’t know how the traffic gets from point A to point B. They buy “Internet” services from the available ISP’s in their area based on their needs.

Most of the companies that offer their employees company paid or user paid cell phones thru AT&T support end-to-end QoS to the company’s Intranet and for most of their employees “Internet” usage.

Most of the sports arenas/stadiums in North and South America including cell phones from other mobile operators goes over AT&T’s network. This includes Wi-Fi, 3GPP and 3GPP2 RAN. We also support most of the cruise ship traffic.

Many ASP’s don’t even know how their traffic goes from point A to point B. They may be using Amazon’s cloud to host their application. Worldwide the majority of Amazon’s traffic is over AT&T’s network from Amazon hosting facilities to at least ISP peering points. In the US it’s a larger majority and its 100% when using AT&T’s home/business/virtual-network-operator Internet or AT&T/Cricket/MVNO mobiles. Many would say this in intranet traffic because it is on AT&T’s network end-to-end and never mixes with other Internet traffic. But end users would call this Internet traffic.

I think the whole should IETF be involved is moot. This stuff is all so intertwined when you start talking about the big tier 1 ISP’s and the big ASP’s/CDN’s like Akamai, Amazon, Apple, AT&T, Google, Facebook, IBM, Microsoft, Netflix, etc….

See “terry>” below…

Terry Figurelle
Lead Member of Technical Staff
Wireless Network Arch. & Design
Mobility Core
AT&T Services
[email-masked]<mailto:[email-masked]>
cell - (206)601-5157

From: 5gangip [mailto:[email-masked]] On Behalf Of Jon Crowcroft
Sent: Friday, March 23, 2018 3:47 AM
To: [email-masked]
Cc: 5GANGIP <[email-masked]>; Behcet Sarikaya <[email-masked]>; [email-masked]
Subject: Re: [5gangip] Notes from today's meeting

so to make something fly as an IETF working group concern, its going to have to have end-to-end-ness about it - sure we have all these properies local to the cellular data service provider, but if packets don't leave that domain, then the IETF probably doesn't have a stake.....imho

terry> Is DirecTV from AT&T when bundled with AT&T fiber service an Intranet or Internet service?
If Comcast is their ISP and they are streaming over Comcast’s network to AT&T’s DirecTV service is that Intranet of Internet?

so while there might be things like 9 air-defined interface levels, unless there's a corresponding list of 9 diffserv services an ISP could offer,
or when there's some desiderata for delay bounding IoT traffic, unless (like most actual IoT applications) it goes off over the backhaul, into the WAN and possibly off into a data center/cloud service, (e.g. see
https://arxiv.org/abs/1803.05368<https://urldefense.proofpoint.com/v2/url?u=https-3A__arxiv.org_abs_1803..05368&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=FfqnZoQQ-8cbnp-lPfR7VAuw5TsDqJEat0_9YUmYTcs&s=mEID5VrQFcIUSym-f6BhAW-yhY-zeS5umczEJSDLXBk&e=>
for some measurements)
then its outside he IETF's bailiwick i believe...

There seems to be a lot of momentum behind OneM2M API which allows an Application Service Provider to interact with a Wireless Services Provider. People care about the services they want not some complex analysis that it actually is an Intranet not Internet path or meets some definition of IoT. So when you stream music in your new car using an App icon that looks the same as your smartphone App does it matter that it’s an IoT solution?

All of our solutions for automotive allow Rx, API or manifest based end-to-end QoS control. We support more automotive companies and total cars on the road than all the other players combined. While we support complete control of packet markings these companies prefer an out of band solution like API gateways. But we support end-to-end layer2/layer3 control for both telemetry, entertainment (e.g. Apple Music or Pandora) and vehicle Wi-Fi Internet services.

We even support this for fleet services so updates to the vehicle software can be done over-the-air when vehicles are parked on the lot in the airport (one of many use cases that are live in the field today).

On Fri, Mar 23, 2018 at 9:18 AM, <[email-masked]<mailto:[email-masked]>> wrote:
No reason at all why we shouldn't consider all and every technology touted as being mobile friendly.   The question I have is whether that should done in a standards track or somewhere else (IAB Study Group, IRTF group, etc).   It would seem to me that IETF WGs are more suited to formalisation where there is a known problem and a set of potential solutions...

To the specific points (and with apologies for this being so long but I strongly feel we have a lot to discuss):

1) In 4G and proposed in 5G, there is tight integration of RAN capabilities and user service.    The best way to explain this is to look at the only service which offers a guaranteed level of quality today in mobile networks - VoLTE.   By using two bearers between the UE and the base-station and monitoring that very closely, IP traffic associated with the voice call is re-directed across a specific set of radio parameters which have a different numerology to the "default" (read Internet) traffic bearer.   The LTE core will monitor all handovers to ensure that the new serving site is capable of handling the required traffic pattern.  As voice is a known-quantity, an exact amount of RF can be allocated to the UE (within usual operational bounds).   When the VoIP traffic reaches the breakout point (PGW in 4G), it is directed to the IMS core over a dedicated SGi interface.   It is also marked/policed as you would mark/police VoIP traffic in a commercial network (i.e. NOT the Internet 😊 ).

terry>
While VoLTE is the only service define by 3GPP which leverages tiered QCI, it is not the only service in production today that allows for API driven Internet or Intranet (choice made by account owner via purchase).

AT&T offers open API or Rx via an API-GW or signed certificate (manifest) for QoS management which does include the TOS/TS field and 3GPP bearer (default and/or dedicated) QoS. We can honor DiffServ markings of Intranet connections that don’t require address translation. We use a set QCI:DiffServ marking but a business can choose different values for SGi. We expose ECN to Intranet and are working to expose that to Internet (not honored by ISP peering). The challenge is passing this thru proxies and NAT. IPv6 is easier since we don’t have the same limitations that come with giving the mobile a private IPv4 address.

Currently, the view is that supporting this thru ISP peering would violate Net Neutrality. So it’s not a technical limitation but legal red tape that prevents this.

For most Internet sites we have direct to data center links (Apple, Facebook, Google, Microsoft, Netflix, etc.) so the markings survive. We map this to our MPLS backbone plus to Metro-E. The granularity of some layers is less than 3GPP 9 (extended is larger) levels. So this is true end-to-end QoS including markings on outside GTP and inside IP packets plus support thru Internet content on AT&T’s network (MIS/GMIS = managed Internet service where AT&T is the ISP). Currently markings are not honored at ISP peering points. About 60-80% of content accessed by mobiles does support end-to-end QoS. We also support this over IPX and it is honored by most of our roaming partners including QCI 1, 2, 5, 6-9. We don’t have a customer yet for QCI 3 or 4 but there are use cases we have discussed with business customers.

For IMS services we support roaming and LBO (local break out) and we have several API-GW’s. We are working to support standard API’s including OneM2M which targets IoT. We will support all API’s defined by 3GPP and OMA (already supported). Plus major IETF standards. We strongly support open source efforts and participate in a number of efforts.

2) This tight coupling between RAN and IP layers allows up to 9 levels of defined air-interface quality at varying bandwidths.   The operation of the PCRF is essentially an admission system for required outcomes.  However, in 4G, the domain of this outcome is limited to the MNO - there is no way an external party could declare the "need" for a particular quality and have that managed across multiple cells.

terry>
See my response to #1 but this is clearly not true since we support open API’s or Rx via API-GW’s that are exposed to the Internet plus we support manifest based signed certifications (AT&T is root signing authority for this service where the protected portion limits to the purchased service levels and business uses their signing key for the list of services in the manifest they want to provide non-default QoS. If the business doesn’t want to host their certificate server then AT&T will host it but give them control to manage their services.

3) In 5G, the concept of the PCRF/PCEF is extended to the PCF which in turn may be influenced by MULTIPLE, EXTERNAL AF (Application Functions).  This is important because it is the first time in mobile networks that the idea of allowing external actors to influence and determine packet treatment all the way to the UE has been considered.  You can see a situation where, for example, BBC iPlayer would be able to signal a 4k HD data stream for a specific user including making sure that the OTA numerology was set to support such a stream to the UE and packets directed over that bearer.   Likewise, should the next cell not be able to handle the stream, this can be signalled to the CDN which could then down-rate the video.    There is then a revenue chain possible - I pay the BBC to delivery me high quality video; they signal to the MNO that I want that; the MNO charges an uplift to the BBC to deliver at 4k.   This is a settlement model not possible in 4G.

terry>
AF’s were defined before 5G and we already have them as API-GW’s including support for Rx to third parties. We support OMA API which does work inter-carrier for messaging today for LTE and are working on a number of open API’s include OneM2M which we think fits IoT. We already support several interfaces to IMS services but I don’t work in that area so I don’t want to list those interfaces but I know we have this working in production. We do support this over IPX and their were public announcements when this first rolled out between AT&T and Verizon. We also support this internationally but it seems US carriers and Asian carriers are ahead of UK and western Europe. So it might not have made the press in the UK.

We already support this for DirecTV traffic and are more than willing to support it for other stream services (Amazon Prime, Netflix, etc.).

4) Densification of wireless is already being seen as a sticky problem in 5G.   Predictions suggest a 20 fold increase in radio coverage but in a mixture of mmWave (high density, low coverage distance, high bandwidth) and low-bands (lower density, high coverage, low bandwidth).  The challenge for the industry is in deploying in the current competitive model of multiple MNOs per country. At least across Europe, cell site installation and planning is a very hot-topic and unpopular with the public.  Network sharing, Neutral Host, single national carriers with MvNOs - all of these models are starting to appear in order to cope with the challenge of having to deploy vast amounts of expensive coverage in a climate of decreasing ARPUs.   The need to manage radio spectrum in the most optimal way possible is real - governments around the world have seen spectrum as a cash-cow....

terry>
We already support a number of MVNO’s and have partners around the world for RAN. This started years ago and was driven as a business  opportunity and expanded for IoT including automotive where AT&T is #1. It had nothing to do with ARPU’s. This was just typical business planning and 5G wasn’t in the picture years ago when we launched these services. The cost of coverage has been dropping for many years and isn’t as big a factor as you are stating even for 5G. Spectrum has always been a scarce resource and always will be.

Cell sites aren’t that big of a challenge and something we excel at. There is a lot that goes into acquiring new cell sites and installing new antennas at existing sites but with good planning and execution it is something the best operators are good at. It’s the devices that are the most significant challenge for both IoT and 5G. Each band and RAN technology added increases the cost of the device and the complexity of designing the device. There are some real geniuses working on this and I am impressed by what they are doing to drive device costs where they need to be. But this is all true with or without 5G.

Infrastructure sharing is not new and it’s a model that fits some business models, operators and countries better than others. But it isn’t a factor in early 5G announcements. Availability of completed standards and time for vendors to work out their products has been the biggest factor early on but that is pretty typically with something new.

There were many factors in Europe lagging other parts of the world for LTE and those may play out for 5G too. The use of mobile Internet exploded in US with 3G and has just gone up from there. IoT has followed with amazing growth and we are adding capacity and functionality as fast as the teams can work. We see this same trend in Asia and to a lesser extent in other places around the world.

5) Whilst from a network engineering perspective path-optimisation may seem like the best thing to do, there are cases in mobile networks where it isn’t.   A good example of this would again be the current VoLTE service:

        - Mobile networks are highly sub-divided by nature.   Cells connect to pre-aggregation points, which connect to S-GWs which connect to P-GW(s - country size dependent).
        - Handing over between cells can either be between the cells directly (direct fibre link, microwave link, tunnel via pre-agg site) or with reference back to the S-GW if shared between eNBs or back as far as the P-GW if served by different S-GWs).
        - Operators will try to build their networks so that the majority of handovers take place on the quickest path possible.  There are usually KPIs about maximum handover times, number of allowed lost frames for the various bearer types, etc.
        - In terms of voice, max round trip time is 150ms.   Packetisation is usually 20ms - overheads take this up to about 35ms leaving around 40ms acceptable gap per leg.  The air interface used for VoLTE relies on a re-transmission scheme to guarantee delivery and this has a operational time of 8ms.   So, only 6 re-transmissions are allowed per voice packet.
        - If the voice service is anchored on the P-GW there is a known, reasonably constant delay - the distance from any eNB to the P-GW will be roughly the same in terms of transport speeds where-ever I am in the country.  In large countries, multiple P-GWs are used and strange things happen as you move between eNBs on different P-GWs but this is usually engineered such that the number of people this would affect is very small.
        - However, latencies on the other paths (S1 handover, X2) will be highly variable as the physical distances between them will be more varied.   You can easily envisage a scenario where a voice call handover would cause a path change length which would become material in terms of the VoIP packet size.   The simple answer is anchor such that UE<->UE media always takes the mostly fixed, longer path.

6) Looking ahead to 5G and mission-critical services, sub-1ms IoT, massive increase in mobile broadband and the VoLTE 75ms requirement becomes a walk-in-the-park.  We need to ensure that any 5G transport network is able to understand not just the packets in-flight but what is required from the RAN in order to support the use-case.

terry> We’ll see about the sub-1ms IoT.

David

-----Original Message-----
From: Jon Crowcroft [mailto:[email-masked]<mailto:[email-masked]>]
Sent: 23 March 2018 08:08
To: [email-masked]<mailto:[email-masked]>; Behcet Sarikaya <[email-masked]<mailto:[email-masked]>>
Cc: 5GANGIP <[email-masked]<mailto:[email-masked]>>; Lake D Mr (PG/R - Elec Electronic Eng) <[email-masked]<mailto:[email-masked]>>; Jon Crowcroft <[email-masked]<mailto:[email-masked]>>
Subject: Re: [5gangip] Notes from today's meeting

is there a technical reason we can'tjust re-visit MIPv6 in 5g in 2018?
> Thanks Dave for the detailed notes.
>
> I extracted this:
>
>
> We did MIPv6 in 2004, and when a binding update works, you eliminate
> anchor points. Does not work in radio access networks as they have
> their own numerology.  Qualcomm did not implement MIPv6.. that killed
> it.
>
> Looking at a 20-30 fold densification of radio access.   No idea how we do
> that.
>
>
>
> On the above point David Lake had stated a requirement for access network.
> David can you please remind it? I am trying to collect and document
> the requirements as know them. We know them.
>
>
> Also Lorenzo commented on the BU message but I am not sure if it was
> as the one above?
>
>
> Regards
>
> _______________________________________________
> 5gangip mailing list
> [email-masked]<mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/5gangip<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_5gangip&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=FfqnZoQQ-8cbnp-lPfR7VAuw5TsDqJEat0_9YUmYTcs&s=6tJRXeKJJ411eGlhREk3TWW660wWFDGRRZ20x2OfLl4&e=>
>