Subject: webex broken for public IPv4

From: Michael Richardson <[email-masked]>
Date: Tue, 12 May 2020 06:46:29 -0400

Two weeks ago, webex audio stopped working for me on my desktop.
It had worked great for a few YEARS with the webrtc version on Linux.

I kept seeing:

   [5803:5808:0512/063438.704895:ERROR:socket_udp.cc(219)] Received unexpected data packet from 173.39.232.161:9000 before STUN binding is finished.

on the console/terminal of my browser.  The STUN message suggested to me that
something was broken if I *don't* have NAT44.  I have an IPv4/32 routed to my
desktop out of the /28 that I have.

I might be among the few, but Henk points out that "Global EDUroam"
on his campus also gives out public IPv4, only he can't use that while in
lockdown to validate my bug report.

I added to my routing an exception that routed 173.39.232.161 with a NAT44
src address.  Details somewhat specific to my setup, but happy to give
details.

netstat now shows a mix of connections as they move over to the rfc1918
address, and AUDIO WORKS.   Henk and I now discuss details of German
un-lock-down vs R>1.

tcp        0      0 209.87.249.21:52404     173.39.232.161:5004     TIME_WAIT
tcp        0      0 172.30.2.21:33066       173.39.232.161:80       TIME_WAIT
tcp        0      0 209.87.249.21:49426     173.39.232.161:80       TIME_WAIT
tcp        0      0 172.30.2.21:37582       173.39.232.161:5004     ESTABLISHED
tcp        0      0 172.30.2.21:37558       173.39.232.161:5004     ESTABLISHED
...

I had opened a ticket two weeks, but no acknowledgement at all.

This bug would have shown up **immediately** if they had IPv6 support.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Tue, 12 May 2020 09:56:58 -0400

Michael Richardson <[email-masked]> wrote:
    > I added to my routing an exception that routed 173.39.232.161 with a

Of course, each meeting winds up with a different target server, so I have to
adapt each time until I learn all the /24s that webex lives in.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 12 May 2020 10:01:53 -0400

Michael Richardson <[email-masked]> wrote:
    > 172.30.2.21:33066 173.39.232.161:80 TIME_WAIT tcp 0 0
    > 209.87.249.21:49426 173.39.232.161:80 TIME_WAIT tcp 0 0
    > 172.30.2.21:37582 173.39.232.161:5004 ESTABLISHED tcp 0 0
    > 172.30.2.21:37558 173.39.232.161:5004 ESTABLISHED ....

The port 5004 seems to go away if there is nobody speaking.

It's curious about use of port 80 vs 443, even though the connection to
ietf.webex.com is port-443, once the page downloads, there is no further
connection to that address.

obiwan-[~](2.6.5) mcr 10270 %netstat -tan | grep 172.30.2
tcp        0      0 172.30.2.21:42978       173.243.4.171:80        ESTABLISHED
tcp        0      0 172.30.2.21:43026       173.243.4.171:80        ESTABLISHED
tcp        0      0 172.30.2.21:43014       173.243.4.171:80        ESTABLISHED
tcp        0      0 172.30.2.21:42990       173.243.4.171:80        ESTABLISHED
tcp        0    148 172.30.2.21:43092       173.243.4.171:80        ESTABLISHED
tcp        0      0 172.30.2.21:43002       173.243.4.171:80        ESTABLISHED
tcp        0      0 172.30.2.21:43038       173.243.4.171:80        ESTABLISHED
tcp        0      0 172.30.2.21:43050       173.243.4.171:80        ESTABLISHED

port 5004 did not return this time.
I guess the media is being tunnelled over HTTP.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 12 May 2020 10:21:09 -0400

After starting the RATS Design team webex, we discovered that screen sharing
did not work.  I found a ton more connections to EC2 and other places from
the chrome process running that tab. The UDP sockets aren't connected, so I
can't guess where they go so redirect the routing so that it gets NAT44.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Colin Perkins <[email-masked]>
Date: Tue, 12 May 2020 15:52:30 +0100

> On 12 May 2020, at 11:46, Michael Richardson <[email-masked]> wrote:
> 

It’s consistent with a firewall blocking STUN packets used to check whether the path to your public IPv4 address actually works.

...
> This bug would have shown up **immediately** if they had IPv6 support.

I'd expect most conferencing applications use ICE to check connectivity. ICE sends STUN packets over IPv6, same as it does over IPv4, to see whether it has a usable path. 

-- 
Colin Perkins
https://csperkins.org/

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 12 May 2020 11:58:03 -0400

you can figure out all of the webex service addresses with:
1) what ip today
2) what ASN originates that route (13445, cisco-webex)
3) what other ips does that ASN originate?

AS      | BGP IPv4 Prefix     | AS Name
13445   | 114.29.192.0/19     | 13445, US
13445   | 114.29.200.0/23     | 13445, US
13445   | 114.29.202.0/23     | 13445, US
13445   | 114.29.204.0/22     | 13445, US
13445   | 114.29.208.0/21     | 13445, US
13445   | 150.253.128.0/18    | 13445, US
13445   | 150.253.192.0/19    | 13445, US
13445   | 150.253.224.0/20    | 13445, US
13445   | 170.133.128.0/19    | 13445, US
13445   | 170.133.160.0/20    | 13445, US
13445   | 170.133.176.0/20    | 13445, US
13445   | 170.133.176.0/23    | 13445, US
13445   | 170.133.178.0/23    | 13445, US
13445   | 170.133.190.0/23    | 13445, US
13445   | 173.243.0.0/20      | 13445, US
13445   | 173.39.224.0/19     | 13445, US
13445   | 207.182.160.0/19    | 13445, US
13445   | 207.182.172.0/22    | 13445, US
13445   | 208.253.254.0/24    | 13445, US
13445   | 208.254.114.0/24    | 13445, US
13445   | 209.197.192.0/19    | 13445, US
13445   | 209.197.192.0/21    | 13445, US
13445   | 210.4.192.0/20      | 13445, US
13445   | 210.4.198.0/24      | 13445, US
13445   | 210.4.206.0/23      | 13445, US
13445   | 216.151.128.0/19    | 13445, US
13445   | 216.151.128.0/20    | 13445, US
13445   | 62.109.192.0/18     | 13445, US
13445   | 64.68.104.0/21      | 13445, US
13445   | 64.68.120.0/21      | 13445, US
13445   | 64.68.96.0/19       | 13445, US
13445   | 66.114.160.0/20     | 13445, US
13445   | 66.114.160.0/21     | 13445, US
13445   | 66.114.168.0/21     | 13445, US
13445   | 66.163.32.0/19      | 13445, US
13445   | 66.163.46.0/23      | 13445, US
13445   | 66.163.52.0/24      | 13445, US
13445   | 66.163.53.0/24      | 13445, US
13445   | 69.26.160.0/20      | 13445, US
13445   | 69.26.160.0/24      | 13445, US
13445   | 69.26.176.0/20      | 13445, US
13445   | 69.26.176.0/22      | 13445, US
13445   | 69.26.176.0/24      | 13445, US
13445   | 69.26.180.0/22      | 13445, US
13445   | 69.26.184.0/21      | 13445, US
13445   | 69.26.188.0/24      | 13445, US

AS      | BGP IPv6 Prefix     | AS Name
13445   | 2402:2500:6::/48    | 13445, US
13445   | 2402:2500:7000::/40 | 13445, US
13445   | 2402:2500::/32      | 13445, US
13445   | 2607:fcf0:102::/48  | 13445, US
13445   | 2607:fcf0:103::/48  | 13445, US
13445   | 2607:fcf0:10::/48   | 13445, US
13445   | 2607:fcf0:11::/48   | 13445, US
13445   | 2607:fcf0:1::/48    | 13445, US
13445   | 2607:fcf0:3::/48    | 13445, US
13445   | 2607:fcf0:4::/48    | 13445, US
13445   | 2607:fcf0:502::/48  | 13445, US
13445   | 2607:fcf0:503::/48  | 13445, US
13445   | 2607:fcf0:5::/48    | 13445, US
13445   | 2607:fcf0:9000::/40 | 13445, US
13445   | 2607:fcf0::/32      | 13445, US

good luck fellow travelers.

On Tue, May 12, 2020 at 10:52 AM Colin Perkins <[email-masked]> wrote:
> ....

---

From: Michael Richardson <[email-masked]>
Date: Tue, 12 May 2020 13:46:40 -0400

Colin Perkins <[email-masked]> wrote:
    >> Two weeks ago, webex audio stopped working for me on my desktop.  It
    >> had worked great for a few YEARS with the webrtc version on Linux.
    >>
    >> I kept seeing:
    >>
    >> [5803:5808:0512/063438.704895:ERROR:socket_udp.cc(219)] Received
    >> unexpected data packet from 173.39.232.161:9000 before STUN binding is
    >> finished.
    >>
    >> on the console/terminal of my browser.  The STUN message suggested to
    >> me that something was broken if I *don't* have NAT44.  I have an
    >> IPv4/32 routed to my desktop out of the /28 that I have.

    > It’s consistent with a firewall blocking STUN packets used to check
    > whether the path to your public IPv4 address actually works.

Which firewall? Mine? Or WEBEX?
My firewall/router really hasn't changed in the two weeks since it broke, and
except for blocking port 5060, I have nothing blocked.

I could watch the outside of my firewall for signs of this.
I'll have to figure out where the STUN server is to trace that.

    > I'd expect most conferencing applications use ICE to check
    > connectivity. ICE sends STUN packets over IPv6, same as it does over
    > IPv4, to see whether it has a usable path.

AFAIK, Webex doesn't speak IPv6 at all.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 12 May 2020 13:51:43 -0400

Christopher Morrow <[email-masked]> wrote:
    > AS | BGP IPv4 Prefix | AS Name 13445 | 114.29.192.0/19 | 13445, US

I'm astonished that they need so much address space for this.
I can see needing up to a /24 in two dozen data centers world-wide, but
the number, I can't see putting more than 256 VMs in a single datacenter, and
certainly not a /19 of them!

It seems pretty excessive to me, maybe a clear indication that IPv6 is needed.

    > AS | BGP IPv6 Prefix | AS Name 13445 | 2402:2500:6::/48 | 13445, US

I've seen no evidence that they use IPv6 yet.
Maybe there are tests going on, or they have been assigned IPv6 for all their
data centers in advance of using it.

But, thank you for the lists, it will make working around this nonsense
easier, and maybe I can find the STUN servers easier.

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 12 May 2020 15:06:37 -0400

> certainly not a /19 of them!

sure.  maybe... unclear to me what any of that webex stuff does...
(not really being a user myself)

> It seems pretty excessive to me, maybe a clear indication that IPv6 is needed.

they do have v6 addresses allocated and routed.

> I've seen no evidence that they use IPv6 yet.

maybe not in your view of their DNS ? :) they may just choose to only
serve you over v4?

> data centers in advance of using it.

they seem to have a LOT of v6 though :)

> easier, and maybe I can find the STUN servers easier.

sure! :)
I think a TON of things just use stun.l.google.com ... I have no idea why  :(

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 May 2020 20:45:43 +0200

> lockdown to validate my bug report.

Because I had been using library access, I was “in the Internet” (via VPN that made my NAT invisible) with a global address today when I joined the CoRE Interim.  And, indeed, the audio didn’t work!  Left the VPN, Webex reconnected me, and audio then did work.  Amazing.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 13 May 2020 19:59:21 -0400

> maybe not in your view of their DNS ? :) they may just choose to only

That could be.
ietf.webex.com is v4-only.
All of the other client-specific addresses I have seen are v4-only.
Maybe others have, I would be happy to hear that.

    > sure! :) I think a TON of things just use stun.l.google.com ... I have
    > no idea why :(

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 13 May 2020 21:06:58 -0400

> ietf.webex.com is v4-only.

oh,. that's a bummer, passive-dns seems to agree :(
poo.

> Maybe others have, I would be happy to hear that.

maybe this is something the IETF folk can ask webex? :)
again I'm not really a webex user, but.. I imagine: "Hey, got some a
dat sweet, v6 over there<itch, itch, itch>... how about you share it?"
:) might go a long way.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 14 May 2020 10:56:25 -0400

Christopher Morrow <[email-masked]> wrote:
    >> That could be.  ietf.webex.com is v4-only.

    > oh,. that's a bummer, passive-dns seems to agree :( poo.

    > maybe this is something the IETF folk can ask webex? :) again I'm not
    > really a webex user, but.. I imagine: "Hey, got some a dat sweet, v6
    > over there<itch, itch, itch>... how about you share it?"  :) might go a
    > long way.

Over a period of four or five years, I've been asking about our SLA with webex.

This goes back to when there was no WEBRTC version, and the JAVA client
just barely ran on Mac/Linux, and on Linux, without any audio support.

Why do we continue to use this BS system when we have multiple other options
(JITSI is one, essentially built by IETFers) which leverage the protocols we
create (webrtc).

Because (a) CISCO donated it, so we have support.
        (b) but, it's a donation, so we have no support.

I still don't know.  Now it's seriously broken.

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 14 May 2020 11:51:52 -0400

> I still don't know.  Now it's seriously broken.

it seems to me that this is another in the 'long line' of things that
the world is realizing could be 'done better' during our current
global issue :)
I mean by this that enterprises realized 'suddenly':
  "Hey, our 10% plan for WFH.. is about 900% off on capacityplanning, whoops!"

Enterprises (and others) realized that their deployments on
thin-straws at their remote headquarters wasn't
really helping all their now WFH folks... their thinstraw is now also
a bottleneck :(

Production services (google-meet, zoom, webex, etc) all realized:
  "hey, we got super popular suddenly.. huh! better get .. A CRAPLOAD
MORE capacity!!! NOWNOWNOW!!!"

I think, if we asked for support/reasoning and pointed out:
  1) ietf now does 100% of meetings virtually
  2) there are a  bunch of reasons why 'getting ipv6 to work' would
help (ietf) and general user populations

I imagine that the webex thing is proxied through AMSL? maybe starting
there for this question set and discussion will be fruitfuil?

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 14 May 2020 17:54:17 +0200

> Why do we continue to use this [^&@#] system when we have multiple other options

I have said often enough why it is not my favorite web conferencing system, but Webex still has a relatively high success factor among IETF participants.  And, as I’m fond to say, it also isn’t run by criminals.

If we want to run our own systems:

Jitsi has become better a lot during the last few months.  It is still somewhat stuck on features that are being added to Firefox, but works well on Chrome-based browsers.  It is really easy now to set up instances; but as with all web conferencing systems that do video, some processing power is needed that is scaling with the overall number of simultaneous participants.

There are also systems that are built to support meetings where people collaborate on technical content, such as BigBlueButton (which has FREESWITCH behind it).  Again, easy to set up instances (and even to integrate with a login system like datatracker has).

Finally, meetecho might have something up their sleeves that I don’t know.

So I think we should make a transition at some point, not because Webex is bad, but because we can get something that is more tailored to our needs.

> I still don't know.  Now it's seriously broken.

Well, this is so glaring that I’m sure it will be fixed.

Grüße, Carsten
(who just came out of a non-IETF Webex meeting that started 15 minutes late because the host and one other participant couldn’t get logged in)

---

From: Ted Hardie <[email-masked]>
Date: Thu, 14 May 2020 09:10:53 -0700

On Thu, May 14, 2020 at 8:54 AM Carsten Bormann <[email-masked]> wrote:

>
> So I think we should make a transition at some point, not because Webex is
> bad, but because we can get something that is more tailored to our needs.
>
>
I actually hope we don't focus too much on tailoring things to our needs.
The advantages of COTS (commercial, off-the-shelf) systems in both
financial terms and likely broad-scale testing probably outweigh the
benefits of customized solutions.  If we do need something unique (e.g. a
"humming tool"), we can make that separate from the tools which address
common needs.

This is not an argument that we must stick to one provider for all time,
obviously; it's easier to switch among them when you don't try to tailor
too much, as that means you have competitors who likely share the common
features.

Just my two cents, of course,

Ted

---

From: Henk Birkholz <[email-masked]>
Date: Thu, 14 May 2020 18:22:44 +0200

Webex has seen better days - but there also were less challenging days once.

It is a bit funny, I admit, that IPv4 does not seem to work correctly on 
the client side, while IPv6 does not seem to work correctly on the 
server side. Fair universal deployment of bugs.

On 14.05.20 18:10, Ted Hardie wrote:
> On Thu, May 14, 2020 at 8:54 AM Carsten Bormann <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
>     So I think we should make a transition at some point, not because
>     Webex is bad, but because we can get something that is more tailored
>     to our needs.
> 
> 
> I actually hope we don't focus too much on tailoring things to our 
> needs.  The advantages of COTS (commercial, off-the-shelf) systems in 
> both financial terms and likely broad-scale testing probably outweigh 
> the benefits of customized solutions.  If we do need something unique 
> (e.g. a "humming tool"), we can make that separate from the tools which 
> address common needs.
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 14 May 2020 18:59:21 +0200

Hi Ted,

> On 2020-05-14, at 18:10, Ted Hardie <[email-masked]> wrote:
> 
> I actually hope we don't focus too much on tailoring things to our needs.  The advantages of COTS (commercial, off-the-shelf) systems in both financial terms and likely broad-scale testing probably outweigh the benefits of customized solutions.  If we do need something unique (e.g. a "humming tool"), we can make that separate from the tools which address common needs.  
> 
> This is not an argument that we must stick to one provider for all time, obviously; it's easier to switch among them when you don't try to tailor too much, as that means you have competitors who likely share the common features.

RFC 873 applies here.  Long form of this statement below.

I do agree that “running web conferences” will be a COTS service at some point like “running a web server”.  There are still good reasons to run your own web server instead of using the ones offered by Facebook, and there might be good reasons to run our own web conferences.

Tailoring in the sense of writing major amounts of code is expensive and should only be done if really needed.  I think that is reasonable with a web application like datatracker that maps our processes into the digital domain.  I also think that, if there are processes that are specific to us that run during meetings, the web conferencing system we use for that should be tailored to that.

With BigBlueButton, “tailoring” here may mean as much as “switching on polls” (which are switched on by default anyway).  That was easy tailoring.  Does this make it harder to switch to a different system that doesn’t offer polls?  Yes.  But we are no better off by setting up a bespoke separate system for a process where the web conferencing system we choose already happens to have the right tailorability.  (The actual tailoring here would be to provide an interface between a user management system like that we already have in datatracker to the conferencing server, which is useful for more than humming.  Not hard, but probably more than a day of work with testing and setting up a working monitoring component.)

So, yeah, we should use Apache (*), but there is nothing wrong with configuring it right (= tailoring).  We shouldn’t write a new web server (= make our own cloth).  We also should not limit ourselves to COTS services that run Apache for us (nor to running everything ourselves, of course).

The pandemic gives us choices here that wouldn’t have been available 6 months ago.  
Let’s benefit from them.

I can’t end this without expressing my gratitude to Cisco for having given us Webex.  I’m also not a big fan of going completely uncontrolled as to on what platform official digital meetings of the IETF are held.  So Webex it is for now.

Grüße, Carsten

(*) nginx, actually, but that is a separate discussion.