Subject: Re: [5gangip] request of packet dump of GTPUDP/IPv6

From: Alexandre Petrescu <[email-masked]>
Date: Mon, 14 May 2018 13:52:29 +0200

Hi Vojislav,

Please allow me to resurrect this because of some doubt I have.

Le 24/01/2018 à 18:51, Vojislav Vucetic a écrit :
> Let me just add to Terry's email:
> 
> Alex, could you please stop with the statement such as "*Cellular 
> operator does have answers about why they continue using IPv4 to 
> transport IPv6 (instead of using IPv6 to transport IPv6*."
> 
> Leading US mobile carriers are ONLY using IPv6 to transport GTP-U (they 
> do not have IPv4 transport network between EPC elements, so it is 
> "difficult" to use IPv4). Let me repeat one more time: *GTPUDP/IPv6*.

This is the doubt: some months ago I talked in person to a couple of 
people at some operators.  I felt I understood that there is no GTP 
UDP/IPv6 anywhere.  I was shown a few packet dumps at some operators 
that do GTP UDP/IPv4 even though it _does_ carry IPv6 inside.

So let me ask you: can you please provide a packet dump of a GTPUDP/IPv6 
packet so I analyse it?

An example of GTPUDP/IPv4 packet dump is visible on IPv6 at 
http://petrescu.sandelman.ca

   > INBOUND>>>>> 15:47:48:883 Eventid:142004(3)
   > GTPU Rx PDU, from [IPv4-address]:2152 to [IPv4-address-2]:2152 (129)
   > TEID: 0x80072183, Message type: GTP_TPDU_MSG (0xFF)
   > Sequence Number:: NA
   > GTP HEADER FOLLOWS:
   > [snip]
   > GTP HEADER ENDS.
   > Payload protocol: IPv6
   > PROTOCOL PAYLOAD FOLLOWS:
   > fe80::b9d9:c10a:2c2d:3c3b.546 > ff02::2.546: [udp sum ok]
   > IPv6 Header Follows:
   > [snip]
   > PROTOCOL PAYLOAD ENDS.

Alex

> 
> You heard the same statement from others (i.e., ATT), but you constantly 
> and consistently ignoring that and keep repeating the same misleading 
> statements. There are carriers that are using IPv4 to transport GTP-u, 
> but that is their choice.
> 
> Vojislav
> 
> On Wed, Jan 24, 2018 at 4:58 PM, FIGURELLE, TERRY F <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     We have deployed more than 20k eNB’s that use IPv6 only transport
>     for S1-U and all of our SGW’s and PGW’s support IPv6 transport for
>     GTP-U and GTP-C. We actually are working with Syniverse for IPv6 IPX
>     for S8. We already have inter lab over production Syniverse IPX
>     between AT&T and six other major telecoms (Bell Canada, Rogers,
>     DoCoMo, etc.). ____
> 
>     __ __
> 
>     We are running  production VoLTE (ims APN) using IPv6 GTP-U|GTP-C
>     with UE IPv6 only and the common general purpose APN (nxtgenphone)
>     supports IPv4 only, IPv6 only and currently IPv4v6 (deprecated by
>     3GPP). This includes two UE operating systems which cover 97% of the
>     market share. Only handful of devices where certified for IPv4v6.
>     All devices AT&T has certified since the launch or UMS support IPv4
>     only, IPv6 only and “IPv4 or IPv6”. AT&T’s device requirements are
>     considered the gold standard in the mobile industry. All of the
>     major vendors for EPC support IPv6 GTP transport and we have 42
>     million active subscribers running on a virtualize EPC (Intel COTS
>     hardware, hypervisor, etc.) that supports IPv6 GTP. Verizon,
>     T-Mobile (USA) and Sprint also use IPv6.____
> 
>     __ __
> 
>     GM’s OnStar uses IPv6 only VoLTE over IPv6 GTP-U in North America
>     they use UE IPv4 only for the other APN’s but those still use IPv6
>     GTP-U in North America. BMW and other auto companies also support
>     IPv6 GTP-U in North America. AT&T was the first to support LTE
>     roaming and we helped Bell Canada and Rogers support it. We can use
>     direct connections (IPX bypass) or Syniverse IPX for LTE roaming.
>     There are a number of country specific IPX but Syniverse now
>     supports interop with a large number of telecoms via their IPX or in
>     conjunction with country specific IPX.____
> 
>     __ __
> 
>     Ericson, Huawei, Samsung and Nokia all support IPv6 GTP-U eNB’s.
>     Spirent, Tektronix and other test tool vendors all support IPv6
>     GTP-C|GTP-U. IPv6 GTP has been supported for several years in
>     production networks. It’s very common in Asian countries too.____
> 
>     __ __
> 
>     Hopefully this ends the IPv6 GTP thread!!!____
> 
>     __ __
> 
>     *Terry Figurelle*____
> 
>     *Lead Member of Technical Staff____*
> 
>     *Wireless Network Arch. & Design____*
> 
>     *Mobility Core____*
> 
>     *AT&T Services____*
> 
>     [email-masked]* <mailto:[email-masked]>____
> 
>     *cell - (206)601-5157 <tel:(206)%20601-5157>____*
> 
>     __ __
> 
>     *From:*5gangip [mailto:[email-masked]
>     <mailto:[email-masked]>] *On Behalf Of *Ca By
>     *Sent:* Wednesday, January 17, 2018 5:13 AM
>     *To:* [email-masked] <mailto:[email-masked]>
>     *Cc:* [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     FIGURELLE, TERRY F <[email-masked] <mailto:[email-masked]>>;
>     [email-masked] <mailto:[email-masked]>
>     *Subject:* Re: [5gangip] Problems I see____
> 
>     __ __
> 
>     __ __
> 
>     On Wed, Jan 17, 2018 at 12:33 AM <[email-masked]
>     <mailto:[email-masked]>> wrote:____
> 
>         <dl> more ... </dl>____
> 
>         __ __
> 
>         Sent from my iPhone____
> 
>         _____________________________
>         From: Alexandre Petrescu <[email-masked]
>         <mailto:[email-masked]>>
>         Sent: Wednesday, January 17, 2018 8:08 am____
> 
> 
>         Subject: Re: [5gangip] Problems I see____
> 
>         To: Lake D Mr (PG/R - Elec Electronic Eng) <[email-masked]
>         <mailto:[email-masked]>>
>         Cc: <[email-masked] <mailto:[email-masked]>>, <[email-masked]
>         <mailto:[email-masked]>>, Nick HEATLEY <[email-masked]
>         <mailto:[email-masked]>>____
> 
> 
> 
> 
>         Le 17/01/2018 à 08:51, [email-masked]
>         <mailto:[email-masked]> a écrit :
>         [...]
> 
>          > (I am not asking whether they offer IPv6 service to
>         smartphone - I do
>          > know many operators offer IPv6 service to smartphone)
>          >
>          > <dl> Utterly non-scientific test involving 5 mobile contracts
>         from 4
>          > of the UK operators (3 MNOs, one MVNO) shows not one support
>         IPv6 to
>          > the UE. </dl>
> 
>         So, I learn in UK there is no IPv6 as a service offered to
>         smartphones.
> 
>         However, there may be some trials ongoing. One could ask, if I may
>         refer, Nick HEATLEY from bt.com
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__bt.com&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=7w99vYkJ60WM1FBGgy4J7mqT1275YbjBb8JT-wb3mnQ&s=Qbn7h7dkjelNV6G3WyrCCIkkGpi185TxTqR2v3Jnlas&e=>,
>         who some times posts in v6ops WG. He
>         often talks about IPv6.____
> 
>         __ __
> 
>         <dl> quick call to friend at UK MvNO - they offer a fixed IPv6
>         service at a cost - this allows a /64 to be allocated against an
>         endpoint and apparently is used for IOT applications which
>         require two-way connectivity.  This is a separate APN. </dl>____
> 
> 
>          >> and for CUPS in the lab.
>          >
>          > What is CUPS?
>          >
>          > <dl> Control User-Plane Split.
> 
>         Thanks for the definition.
> 
>          > Fundamental building-block of 5G core architecture based which
>          > allows different user-planes associated with a single control
>         plane,
>          > thereby allowing differing "services" to be offered across the
>          > user-planes. HOWEVER, this is where the arguments start (see IETF
>          > Netslices proto-WG list for nearly 2 years of arguments and
>          > counter-arguments) as no-one is actually sure what the "services"
>          > are. One theory is that we could offer different protocol support
>          > on different user-plane slices, but this is problematic now
>         that N3
>          > and N6 have been defined. The other is that we could offer
>         different
>          > grades of service but this presupposes that we can split
>         traffic from
>          > the UE on a "session by session" or "application by application"
>          > basis.
>          >
>          > CUPS in 5G is the action of multiple UPFs against single
>         AMF/SMF (or
>          > some combination of AMF+SMF leading to a single control-plane
>          > instance). This allows scaling/placement of the UPF
>         independently of
>          > the control-plane elements.
>          >
>          > CUPS is also being retro-fitted to 4G with the concept of
>         SGWc/PGWc
>          > in the control plane and multiple SGWu/PGWu in the
>         user-plane. </dl>
> 
>         Is GTP part of control plane or of data plane?____
> 
>         __ __
> 
>         <dl> For 4G, Both.   GTP-u used on the S1 user-plane leg; GTP-c
>         used between MME and SGW (S11 interface)   S1-MME (eNB to MME
>         for control plane uses SCTP.____
> 
>         __ __
> 
>         In 5G, the interface from gNB to AMF is N2 which I haven’t seen
>         defined as yet (but will probably be SCTP as per current
>         S1-MME).____
> 
>         __ __
> 
>         However, there is no functional equivalent to S11 in 5G as the
>         MME function has been broken out to AMF and SMF elements and it
>         is the SMF which communicates to the multiple UPF over N4.____
> 
>         __ __
> 
>         The protocol chosen for N4 is PFCP carrying dataplane
>         programming verbs (eg OpenFlow). ____
> 
>         __ __
> 
>         At least, this is my current understanding and I remain to be
>         corrected by anyone who knows more detail/recent information. ____
> 
>         __ __
> 
>         I do not attend 3gpp so I rely on collecting information from
>         those around me who do - it is such a vast organisation that it
>         is sometime very difficult to keep up-to-date.  </dl>____
> 
>     __ __
> 
>     It’s somewhat like trying to contribute to rocket science by
>     attending a cricket game. ____
> 
>     __ __
> 
>     Both  rockets and cricket use physics, and anyone can comment how it
>     might work. ____
> 
>     __ __
> 
>     __ __
> 
>     __ __
> 
> 
>         Alex
> 
>         ____
> 
>         _______________________________________________
>         5gangip mailing list
>         [email-masked] <mailto:[email-masked]>
>         https://www.ietf.org/mailman/listinfo/5gangip
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_5gangip&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=7w99vYkJ60WM1FBGgy4J7mqT1275YbjBb8JT-wb3mnQ&s=PBB_TnejRlFpVc-8l3HLn7XqqqUGonk0Rug-1KLLim0&e=>____
> 
> 
>     _______________________________________________
>     5gangip mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/5gangip
>     <https://www.ietf.org/mailman/listinfo/5gangip>
> 
>

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 14 May 2018 10:14:27 -0500

Hi Alex,

snipped and also simplified cc list.

Alex: GTP in LTE starts at eNB not at UE, so I don't think IPv6 problem is
because of this. So maybe if eNB is dual stack, it uses IPv4 if UE has IPv4.

> http://petrescu.sandelman.ca
Folks, anyone who has access to this document, please post it on the list.

Thanks,
Behcet

> [email-masked]

---

From: "FIGURELLE, TERRY F" <[email-masked]>
Date: Mon, 14 May 2018 15:46:28 +0000

The flip is more common IPv6 transport and IPv5 inside packet in US. We basically share a lot of towers but not everybody in a mobility company knows how backhaul is done. Sprint basically hired E/// to run their network. Here in Redmond WA we steal T-Mo USA employees and they steal ours. We also share vendors for RAN with Verizon and we both drove features for LTE and VoLTE.

I hope I cleaned up the distro correctly. Apparently it was causing issues to put the distro in CC.

Unless you want to partition or segment the network it makes sense to use IPv6 for transport. We have too many nodes in too large (both traffic and physical distances) to do the re-use private IP space. We started LTE with IPv4 then switched once we got all of backhaul equipment upgraded because some equipment was too slow (old) when using IPv6.

We work a lot with Bell and Rogers in Canada including support for local breakout for VoLTE needed by Connected Car. Not too surprising that cars are so mobile but we do not hold a call up during transition from one mobile operator to the other. 

While we use UE IPv6 only for IMS APN (as required), we use IPv4 10/8 with very high reuse for the general purpose APN (nxtgenphone). That APN name is what happens when somebody invites marketing to a meeting. The general purpose APN is a proxied/NAT experience to the Internet. We don't have enough public IPv4 to use for consumer mobiles. We did get dual stack bearer working where the UE was given both a Global Unique IPv6 address and a IPv4 10/8 address. When 3GPP deprecated dual stack bearer, we and device vendors had to take a step back and rethink the path to IPv6 for UE consumer on general purpose APN. You still need the proxy because of supplementary services (XCAP). Plus our proxy is really a proxy-TDF and fully participates in PCC. We are working on NAT-TDF and FW-TDF.

We have per APN address family for the 4000+ APN's we have active (growing fast). We use this a lot for IoT where each network needs privacy and protection. We are pushing IPv6 for transport and device addressing as much as possible.

The S/P-GW's are all dual stack but if given both addresses will always prefer IPv6. We didn't bother to upgrade 3G to IPv6 transport as the cost wasn't justified. Now we are expanding LTE footprint without 3G below it in places and still growing. AT&T Mexico uses same VoLTE design (reassign people to help). Transport is much more challenged in Mexico and we use  more microwave relay there.

We used S1 handoffs between address families during the eNB conversion to IPv6 but now all continental US and Hawaii LTE macro sites are IPv6. There was some weird transport challenges in parts of Alaska but within three years all of our transport contracts require support for MTU2000 and IPv6 for eNB/NR. Since we are also a transport provider (largest in US), we are very familiar with how other operators want their backhaul configured. Typically we have fiber to cell site and multiple operators  on same tower is common except for the replace a utility pole sites which don't have space. Those tend to be daisy-chained which is common for small cell too.

There will be IPv4 GTP for a long time given the nature of many roaming partners. We did work hard with Syniverse to get IPv6 supported and our three US IPX connections are all active plus can support a failure on one of the IPX sites. We are talking about more IPX sites but this is  mostly about latency. It takes collaboration to not have packets take the longer path since we don't control what the roaming partner does. But that collaboration is often AT&T with IPX suppliers. We have the ability to direct GTP traffic to where we want it handled and so does Syniverse. 

Not sure why this matters since things always move at different paces in different places. Any solution has to be backwards compatible so GTP support will be required for some time. Same goes for moving from IPv4 to IPv6 for various things. Even within a company, nobody is doing a flash conversion to  IPv6. You either build new with IPv6 then transition old or deprecate old. Migrating the entire Internet to IPv6 is a whole can of worms and then some. For mobility it likely will be device capability driven when some new device will only see an IPv6 Internet experience. But to start some form of translation makes sense and the transparent proxies support mixed family inside versus outside TCP session. We actually push all TCP traffic thru proxy where it is a TCP proxy instead of a normal HTTP proxy for non-HTTP/HTPS traffic. IP and UDP are a bit more challenging. Common stuff is supported but you can never be sure what won't working until you hit it.

Plus there is a lot of traffic that does not follow IETF standards including even for TCP. So if they breaks should we make non-standard "tweaks" work? How would we know the intent?

As an FYI, mobiles used to violate standards all of the time. Vendors would get very creative trying to hide latency and save battery life. Microsoft even got it in their head that it was faster to send a RST instead of a FYN because you didn't have the ACK to deal with. Time on for radios is a big pull on battery. A version of Windows Mobile for LTE did make it out there with this Microsoft "tweak". But we forced them to change and updated our mobiles. But many operators including us allow bring your own device so it is what it is.

For 5G I would expect all of the major players that push this into production to use new or forward looking solutions at all layers. The high speed and low latency require tight/tidy transport solutions that can meet those demands. In US it seems like everyone is pushing UPF/SGW-U/PGW-U towards macro cell edge. Because of the user-plane and control-plane split you don't have to distribute the control-plane as much. But there is a point where more locations doesn't help and actually hurts. This depends highly on fiber topology and number of cell sites within each tracking area. Inter handoffs always take longer and always have a higher failure rate often due to timing or packet loss over new/old path when things are not in sync yet.

The ultra-fast user-plane actually can and does cause race conditions particularly for IMS. No matter what you do there is no perfect setting for a timer and vendors dislike multiple flavors of a timer but we did get some of that. Here is where replacing GTP may cause the biggest challenges because it will change timings even if only by a small amount. Plus a lot of functionality needs to be duplicated to handle traffic without encapsulation. I've already talked to probe vendors and they think it will be costly. But vendors always sound alarms, so time will tell.

However the biggest challenge is likely table-size/performance for a mobility solution and how those tables are managed.

> -----Original Message-----
> From: Alexandre Petrescu <[email-masked]>
> Sent: Monday, May 14, 2018 4:52 AM
> To: Vojislav Vucetic <[email-masked]>; FIGURELLE, TERRY F
> <[email-masked]>
> Cc: Ca By <[email-masked]>; [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: request of packet dump of GTPUDP/IPv6
> 
> Hi Vojislav,
> 
> > Leading US mobile carriers are ONLY using IPv6 to transport GTP-U
> > (they do not have IPv4 transport network between EPC elements, so it
> > is "difficult" to use IPv4). Let me repeat one more time: *GTPUDP/IPv6*.
> 
> This is the doubt: some months ago I talked in person to a couple of people at
> some operators.  I felt I understood that there is no GTP
> UDP/IPv6 anywhere.  I was shown a few packet dumps at some operators that
> do GTP UDP/IPv4 even though it _does_ carry IPv6 inside.
> 
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__petrescu.sandelman.ca&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=NMY3tOyFFW5zURaioEl
> TD5HtnC6WawgpebslW0trndU&s=16SMz7bvR4GAWg9HDje_UQcrNfQYzs30hjH
> CMOkpzrk&e=
> 
> > You heard the same statement from others (i.e., ATT), but you
> > constantly and consistently ignoring that and keep repeating the same
> > misleading statements. There are carriers that are using IPv4 to
> > transport GTP-u, but that is their choice.
> >
> > Vojislav
> >
> >         refer, Nick HEATLEY from
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__bt.com&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=NMY3tOyFFW5zURaioEl
> TD5HtnC6WawgpebslW0trndU&s=g7LLZ34zzgxSL0eOYEXzDsbwY8RdKDeu_gX0e
> Y3_be0&e=
> >         <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__bt.com&d=DwMFaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=7w99vYkJ60WM1FBGgy
> 4J7mqT1275YbjBb8JT-
> wb3mnQ&s=Qbn7h7dkjelNV6G3WyrCCIkkGpi185TxTqR2v3Jnlas&e=>,
> >         https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_5gangip&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=NMY3tOyFFW5zURaioEl
> TD5HtnC6WawgpebslW0trndU&s=jnSkbCbCaUUnTdikyKn8VD7ec4koxrfYvXrsPC
> BbVjs&e=
> >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
> > lman_listinfo_5gangip&d=DwMFaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxw
> > l2paJIf4zw&m=7w99vYkJ60WM1FBGgy4J7mqT1275YbjBb8JT-
> wb3mnQ&s=PBB_TnejRlF
> > pVc-8l3HLn7XqqqUGonk0Rug-1KLLim0&e=>____
> >
> >
> >     https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_5gangip&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=NMY3tOyFFW5zURaioEl
> TD5HtnC6WawgpebslW0trndU&s=jnSkbCbCaUUnTdikyKn8VD7ec4koxrfYvXrsPC
> BbVjs&e=
> >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
> > lman_listinfo_5gangip&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxw
> >
> l2paJIf4zw&m=NMY3tOyFFW5zURaioElTD5HtnC6WawgpebslW0trndU&s=jnSkb
> CbCaUU
> > nTdikyKn8VD7ec4koxrfYvXrsPCBbVjs&e=>
> >
> >

---

From: "FIGURELLE, TERRY F" <[email-masked]>
Date: Mon, 14 May 2018 16:16:22 +0000

No, there is no correlation between address family used by UE and transport address family. Plus, unless something has changed drastically, the eNB vendors did not support dual stack interfaces. You need to view eNB’s as statically configured via EMS with one transport address family for GTP.

Another challenge is X2 can’t work between address families and S1 with indirect forwarding is a lot more steps in the call flow. More is never good when it comes to call flows!  On the SGW/PGW/UPF side the IP addresses are in APN DNS so it isn’t hard to support dual stack or convert over to supporting IPv6.

Said another way you really don’t want to mix address families within a tracking area other than during migration towards IPv6. It impacts the success rate of active handoffs and increases signaling.

From: 5gangip <[email-masked]> On Behalf Of Behcet Sarikaya
Sent: Monday, May 14, 2018 8:14 AM
To: Alexandre Petrescu <[email-masked]>
Cc: [email-masked]
Subject: Re: [5gangip] request of packet dump of GTPUDP/IPv6

Hi Alex,

Alex: GTP in LTE starts at eNB not at UE, so I don't think IPv6 problem is because of this. So maybe if eNB is dual stack, it uses IPv4 if UE has IPv4.

An example of GTPUDP/IPv4 packet dump is visible on IPv6 at http://petrescu.sandelman.ca<https://urldefense.proofpoint.com/v2/url?u=http-3A__petrescu.sandelman.ca&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=z0Nuc8RFCzTwlRTusGhgOhgQ3uMpqoG2owSaRIahdd8&s=_wncEzKP5ol0PElytLZRcBk2yqOaMIjuItMjs9E7MhI&e=>

Thanks,
Behcet

Alex

_______________________________________________
5gangip mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/5gangip<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_5gangip&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=tO_sNxa2NTxwl2paJIf4zw&m=z0Nuc8RFCzTwlRTusGhgOhgQ3uMpqoG2owSaRIahdd8&s=aLb9YfipE8oD2yvvXMwlVqeVKgDcsqEYFAvfYITLDKs&e=>

---

From: Alexandre Petrescu <[email-masked]>
Date: Mon, 14 May 2018 19:49:05 +0200

Le 14/05/2018 à 17:46, FIGURELLE, TERRY F a écrit :
[...]

>> https://urldefense.proofpoint.com/v2/url?u=http-

I wonder whether urldefense kicked in simply because the posted URL is 
reachable on IPv6?

I said: http://petrescu.sandelman.ca

I wonder whether urldefense kicks in if I post an IPv4 URL like 
http://www.petrescu.fr

Alex

---

From: Alexandre Petrescu <[email-masked]>
Date: Tue, 15 May 2018 08:13:56 +0200

A packet dump with GTP-over-UDP-over-IPv6 please?

Alex

Le 14/05/2018 à 13:52, Alexandre Petrescu a écrit :
>    > INBOUND>>>>> 15:47:48:883 Eventid:142004(3)
>    > GTPU Rx PDU, from [IPv4-address]:2152 to [IPv4-address-2]:2152 (129)
>    > TEID: 0x80072183, Message type: GTP_TPDU_MSG (0xFF)
>    > Sequence Number:: NA
>    > GTP HEADER FOLLOWS:
>    > [snip]
>    > GTP HEADER ENDS.
>    > Payload protocol: IPv6
>    > PROTOCOL PAYLOAD FOLLOWS:
>    > fe80::b9d9:c10a:2c2d:3c3b.546 > ff02::2.546: [udp sum ok]
>    > IPv6 Header Follows:
>    > [snip]
>    > PROTOCOL PAYLOAD ENDS.
> 
> Alex
> 
> 
>>
>>          > (I am not asking whether they offer IPv6 service to
>>         smartphone - I do
>>          > know many operators offer IPv6 service to smartphone)
>>          >
>>          > <dl> Utterly non-scientific test involving 5 mobile contracts
>>         from 4
>>          > of the UK operators (3 MNOs, one MVNO) shows not one support
>>         IPv6 to
>>          > the UE. </dl>
>>
>>          >> and for CUPS in the lab.
>>          >
>>          > What is CUPS?
>>          >
>>          > <dl> Control User-Plane Split.
>>
>>          > Fundamental building-block of 5G core architecture based which
>>          > allows different user-planes associated with a single control
>>         plane,
>>          > thereby allowing differing "services" to be offered across the
>>          > user-planes. HOWEVER, this is where the arguments start 
>> (see IETF
>>          > Netslices proto-WG list for nearly 2 years of arguments and
>>          > counter-arguments) as no-one is actually sure what the 
>> "services"
>>          > are. One theory is that we could offer different protocol 
>> support
>>          > on different user-plane slices, but this is problematic now
>>         that N3
>>          > and N6 have been defined. The other is that we could offer
>>         different
>>          > grades of service but this presupposes that we can split
>>         traffic from
>>          > the UE on a "session by session" or "application by 
>> application"
>>          > basis.
>>          >
>>          > CUPS in 5G is the action of multiple UPFs against single
>>         AMF/SMF (or
>>          > some combination of AMF+SMF leading to a single control-plane
>>          > instance). This allows scaling/placement of the UPF
>>         independently of
>>          > the control-plane elements.
>>          >
>>          > CUPS is also being retro-fitted to 4G with the concept of
>>         SGWc/PGWc
>>          > in the control plane and multiple SGWu/PGWu in the
>>         user-plane. </dl>
>>

---

From: Alexandre Petrescu <[email-masked]>
Date: Tue, 15 May 2018 20:35:19 +0200

Terry,

There is a lot of experience described in the email below.  I can 
understand some of the points.  I agree with some points, for example, 
about the way to migrate to IPv6 and later deprecate IPv4.

However, I will also reply to you to say that if ever you get a chance 
to ask people with Tektronix test tools to get a hand on a GTP packet 
capture I would be grateful.

Alex

Le 14/05/2018 à 17:46, FIGURELLE, TERRY F a écrit :
> The flip is more common IPv6 transport and IPv5 inside packet in US.
> We basically share a lot of towers but not everybody in a mobility
> company knows how backhaul is done. Sprint basically hired E/// to
> run their network. Here in Redmond WA we steal T-Mo USA employees and
> they steal ours. We also share vendors for RAN with Verizon and we
> both drove features for LTE and VoLTE.
> 
> Unless you want to partition or segment the network it makes sense to
> use IPv6 for transport. We have too many nodes in too large (both
> traffic and physical distances) to do the re-use private IP space. We
> started LTE with IPv4 then switched once we got all of backhaul
> equipment upgraded because some equipment was too slow (old) when
> using IPv6.
> 
> We work a lot with Bell and Rogers in Canada including support for
> local breakout for VoLTE needed by Connected Car. Not too surprising
> that cars are so mobile but we do not hold a call up during
> transition from one mobile operator to the other.
> 
> While we use UE IPv6 only for IMS APN (as required), we use IPv4 10/8
> with very high reuse for the general purpose APN (nxtgenphone). That
> APN name is what happens when somebody invites marketing to a
> meeting. The general purpose APN is a proxied/NAT experience to the
> Internet. We don't have enough public IPv4 to use for consumer
> mobiles. We did get dual stack bearer working where the UE was given
> both a Global Unique IPv6 address and a IPv4 10/8 address. When 3GPP
> deprecated dual stack bearer, we and device vendors had to take a
> step back and rethink the path to IPv6 for UE consumer on general
> purpose APN. You still need the proxy because of supplementary
> services (XCAP). Plus our proxy is really a proxy-TDF and fully
> participates in PCC. We are working on NAT-TDF and FW-TDF.
> 
> We have per APN address family for the 4000+ APN's we have active
> (growing fast). We use this a lot for IoT where each network needs
> privacy and protection. We are pushing IPv6 for transport and device
> addressing as much as possible.
> 
> The S/P-GW's are all dual stack but if given both addresses will
> always prefer IPv6. We didn't bother to upgrade 3G to IPv6 transport
> as the cost wasn't justified. Now we are expanding LTE footprint
> without 3G below it in places and still growing. AT&T Mexico uses
> same VoLTE design (reassign people to help). Transport is much more
> challenged in Mexico and we use  more microwave relay there.
> 
> We used S1 handoffs between address families during the eNB
> conversion to IPv6 but now all continental US and Hawaii LTE macro
> sites are IPv6. There was some weird transport challenges in parts of
> Alaska but within three years all of our transport contracts require
> support for MTU2000 and IPv6 for eNB/NR. Since we are also a
> transport provider (largest in US), we are very familiar with how
> other operators want their backhaul configured. Typically we have
> fiber to cell site and multiple operators  on same tower is common
> except for the replace a utility pole sites which don't have space.
> Those tend to be daisy-chained which is common for small cell too.
> 
> There will be IPv4 GTP for a long time given the nature of many
> roaming partners. We did work hard with Syniverse to get IPv6
> supported and our three US IPX connections are all active plus can
> support a failure on one of the IPX sites. We are talking about more
> IPX sites but this is  mostly about latency. It takes collaboration
> to not have packets take the longer path since we don't control what
> the roaming partner does. But that collaboration is often AT&T with
> IPX suppliers. We have the ability to direct GTP traffic to where we
> want it handled and so does Syniverse.
> 
> Not sure why this matters since things always move at different paces
> in different places. Any solution has to be backwards compatible so
> GTP support will be required for some time. Same goes for moving from
> IPv4 to IPv6 for various things. Even within a company, nobody is
> doing a flash conversion to  IPv6. You either build new with IPv6
> then transition old or deprecate old. Migrating the entire Internet
> to IPv6 is a whole can of worms and then some. For mobility it likely
> will be device capability driven when some new device will only see
> an IPv6 Internet experience. But to start some form of translation
> makes sense and the transparent proxies support mixed family inside
> versus outside TCP session. We actually push all TCP traffic thru
> proxy where it is a TCP proxy instead of a normal HTTP proxy for
> non-HTTP/HTPS traffic. IP and UDP are a bit more challenging. Common
> stuff is supported but you can never be sure what won't working until
> you hit it.
> 
> Plus there is a lot of traffic that does not follow IETF standards
> including even for TCP. So if they breaks should we make non-standard
> "tweaks" work? How would we know the intent?
> 
> As an FYI, mobiles used to violate standards all of the time. Vendors
> would get very creative trying to hide latency and save battery life.
> Microsoft even got it in their head that it was faster to send a RST
> instead of a FYN because you didn't have the ACK to deal with. Time
> on for radios is a big pull on battery. A version of Windows Mobile
> for LTE did make it out there with this Microsoft "tweak". But we
> forced them to change and updated our mobiles. But many operators
> including us allow bring your own device so it is what it is.
> 
> For 5G I would expect all of the major players that push this into
> production to use new or forward looking solutions at all layers. The
> high speed and low latency require tight/tidy transport solutions
> that can meet those demands. In US it seems like everyone is pushing
> UPF/SGW-U/PGW-U towards macro cell edge. Because of the user-plane
> and control-plane split you don't have to distribute the
> control-plane as much. But there is a point where more locations
> doesn't help and actually hurts. This depends highly on fiber
> topology and number of cell sites within each tracking area. Inter
> handoffs always take longer and always have a higher failure rate
> often due to timing or packet loss over new/old path when things are
> not in sync yet.
> 
> The ultra-fast user-plane actually can and does cause race conditions
> particularly for IMS. No matter what you do there is no perfect
> setting for a timer and vendors dislike multiple flavors of a timer
> but we did get some of that. Here is where replacing GTP may cause
> the biggest challenges because it will change timings even if only by
> a small amount. Plus a lot of functionality needs to be duplicated to
> handle traffic without encapsulation. I've already talked to probe
> vendors and they think it will be costly. But vendors always sound
> alarms, so time will tell.
> 
> However the biggest challenge is likely table-size/performance for a
> mobility solution and how those tables are managed.
> 
>> -----Original Message----- From: Alexandre Petrescu
>> <[email-masked]> Sent: Monday, May 14, 2018 4:52 AM 
>> <[email-masked]> Cc: Ca By <[email-masked]>;
>> [email-masked]; [email-masked]; [email-masked] Subject:
>> Re: request of packet dump of GTPUDP/IPv6
>> 
>> Hi Vojislav,
>> 
>>> Alex, could you please stop with the statement such as
>>> "*Cellular operator does have answers about why they continue
>>> using IPv4 to transport IPv6 (instead of using IPv6 to transport
>>> IPv6*."
>>> 
>>> Leading US mobile carriers are ONLY using IPv6 to transport
>>> GTP-U (they do not have IPv4 transport network between EPC
>>> elements, so it is "difficult" to use IPv4). Let me repeat one
>>> more time: *GTPUDP/IPv6*.
>> 
>> This is the doubt: some months ago I talked in person to a couple
>> of people at some operators.  I felt I understood that there is no
>> GTP UDP/IPv6 anywhere.  I was shown a few packet dumps at some
>> operators that do GTP UDP/IPv4 even though it _does_ carry IPv6
>> inside.
>> 
>> So let me ask you: can you please provide a packet dump of a
>> GTPUDP/IPv6 packet so I analyse it?
>> 
>>> INBOUND>>>>> 15:47:48:883 Eventid:142004(3) GTPU Rx PDU, from
>>> [IPv4-address]:2152 to [IPv4-address-2]:2152 (129) TEID:
>>> 0x80072183, Message type: GTP_TPDU_MSG (0xFF) Sequence Number::
>>> NA GTP HEADER FOLLOWS: [snip] GTP HEADER ENDS. Payload protocol:
>>> IPv6 PROTOCOL PAYLOAD FOLLOWS: fe80::b9d9:c10a:2c2d:3c3b.546 >
>>> ff02::2.546: [udp sum ok] IPv6 Header Follows: [snip] PROTOCOL
>>> PAYLOAD ENDS.
>> 
>> Alex
>> 
>> 
>>> 
>>> constantly and consistently ignoring that and keep repeating the
>>> same misleading statements. There are carriers that are using
>>> IPv4 to transport GTP-u, but that is their choice.
>>> 
>>> Vojislav
>>> 
>>> On Wed, Jan 24, 2018 at 4:58 PM, FIGURELLE, TERRY F
>>> <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> We have deployed more than 20k eNB’s that use IPv6 only
>>> transport for S1-U and all of our SGW’s and PGW’s support IPv6
>>> transport for GTP-U and GTP-C. We actually are working with
>>> Syniverse for IPv6 IPX for S8. We already have inter lab over
>>> production Syniverse IPX between AT&T and six other major
>>> telecoms (Bell Canada, Rogers, DoCoMo, etc.). ____
>>> 
>>> __ __
>>> 
>>> We are running  production VoLTE (ims APN) using IPv6
>>> GTP-U|GTP-C with UE IPv6 only and the common general purpose APN
>>> (nxtgenphone) supports IPv4 only, IPv6 only and currently IPv4v6
>>> (deprecated by 3GPP). This includes two UE operating systems
>>> which cover 97% of the market share. Only handful of devices
>>> where certified for IPv4v6. All devices AT&T has certified since
>>> the launch or UMS support IPv4 only, IPv6 only and “IPv4 or
>>> IPv6”. AT&T’s device requirements are considered the gold
>>> standard in the mobile industry. All of the major vendors for EPC
>>> support IPv6 GTP transport and we have 42 million active
>>> subscribers running on a virtualize EPC (Intel COTS hardware,
>>> hypervisor, etc.) that supports IPv6 GTP. Verizon, T-Mobile (USA)
>>> and Sprint also use IPv6.____
>>> 
>>> __ __
>>> 
>>> GM’s OnStar uses IPv6 only VoLTE over IPv6 GTP-U in North
>>> America they use UE IPv4 only for the other APN’s but those still
>>> use IPv6 GTP-U in North America. BMW and other auto companies
>>> also support IPv6 GTP-U in North America. AT&T was the first to
>>> support LTE roaming and we helped Bell Canada and Rogers support
>>> it. We can use direct connections (IPX bypass) or Syniverse IPX
>>> for LTE roaming. There are a number of country specific IPX but
>>> Syniverse now supports interop with a large number of telecoms
>>> via their IPX or in conjunction with country specific IPX.____
>>> 
>>> __ __
>>> 
>>> <mailto:[email-masked]>] *On Behalf Of *Ca By *Sent:*
>>> Wednesday, January 17, 2018 5:13 AM *To:* [email-masked]
>>> <mailto:[email-masked]> *Cc:* [email-masked]
>>> <mailto:[email-masked]>; [email-masked]
>>> <mailto:[email-masked]>; FIGURELLE, TERRY F
>>> <[email-masked] <mailto:[email-masked]>>; [email-masked]
>>> <mailto:[email-masked]> *Subject:* Re: [5gangip] Problems I
>>> see____
>>> 
>>> __ __
>>> 
>>> __ __
>>> 
>>> _____________________________ From: Alexandre Petrescu
>>> <[email-masked] 
>>> <mailto:[email-masked]>> Sent: Wednesday, January
>>> 17, 2018 8:08 am____
>>> 
>>> 
>>> <mailto:[email-masked]>> Cc: <[email-masked]
>>> <mailto:[email-masked]>>, <[email-masked] 
>>> <mailto:[email-masked]> a écrit : [...]
>>> 
>>> However, there may be some trials ongoing. One could ask, if I
>>> may refer, Nick HEATLEY from
>>> who some times posts in v6ops WG. He often talks about IPv6.____
>>> 
>>> __ __
>>> 
>>>> user-planes. HOWEVER, this is where the arguments start (see
>>>> IETF Netslices proto-WG list for nearly 2 years of arguments
>>>> and counter-arguments) as no-one is actually sure what the
>>>> "services" are. One theory is that we could offer different
>>>> protocol support on different user-plane slices, but this is
>>>> problematic now
>>> that N3
>>>> the UE on a "session by session" or "application by
>>>> application" basis.
>>>> 
>>> used between MME and SGW (S11 interface)   S1-MME (eNB to MME for
>>> control plane uses SCTP.____
>>> 
>>> __ __
>>> 
>>> The protocol chosen for N4 is PFCP carrying dataplane programming
>>> verbs (eg OpenFlow). ____
>>> 
>>> __ __
>>> 
>>> Both  rockets and cricket use physics, and anyone can comment how
>>> it might work. ____
>>> 
>>> __ __
>>> 
>>> __ __
>>> 
>>> __ __
>>> 
>>> 
>>> Alex
>>> 
>>> ____
>>> 
>>> _______________________________________________ 5gangip mailing
>>> list [email-masked] <mailto:[email-masked]> 
>>> _______________________________________________ 5gangip mailing
>>> list [email-masked] <mailto:[email-masked]>