Subject: Re: [homenet] [Babel-users] Detecting bridges

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 16 Dec 2015 18:01:43 +0100

Added Mikael and the Homenet list to CC.

Homenet, the issue we're dealing with is that babeld performs badly when
there is a transparent wireless bridge connected to a wired interface: the
interface is treated as a lossless wired interface, and if it suffers
packet loss, there is repeated link flapping.  The issue surfaced in the
Slovenian community mesh, but it is likely to occur in a Homenet setting.

> the resulting nanog conversation on detecting wireless bridged ended
> up interesting - with several clever techniques proposed - all
> probably futile.

Yes.  Anoop Ghanwani is the one who gets it:

> bridges are supposed to be "transparent," so there is no way to know
> they are present by using user data frames.

Radia Perlman once said "Bridges don't make sense.  If you do packet
forwarding, you should do it on layer 3."  And then she went on to design
TRILL.

> I fear the default for babel should become etx or rtt as most of the
> world bridges wifi to ethernet.

It's not as bad as you make it.  Babeld automatically detects Linux
software bridges (which takes care of the default OpenWRT configuration)
as well as BATMAN intefaces (which takes care of the Italian community
meshes).  In 1.7.0, I'm planning to add a third interface type to babeld,
the "tunnel" type, and automatically enable it on tun/tap interfaces --
this should take care of OpenVPN tunnels.  (If you're interested, the
tunnel type will use a slightly different link-quality estimator, one that
is not bound to the 802.11 MAC, disable split horizon, and enable RTT
estimation.)

(Why disable split horizon?  People are running Babel over tap interfaces
in a point-to-multipoint topology.  Which is somewhat inefficient but
simpler to configure than multiple point-to-point tuns.)

The question remains about what to do with interfaces that appear as
normal wired interfaces, but could be bridged to wireless.  We currently
treat them as lossless wired interfaces, which gives fast reaction to link
failures (just 1.5 hello intervals on average) and no cost fluctuations,
which reduces the amount of control traffic.  However, this causes very
bad behaviour when there is a wireless bridge in the way -- it causes
repeated link failures, which causes massive instability --, and
suboptimal routing due to split horizon processing.

I can see the following possibilites:

 0. ignore the issue;

 1. use the wireless type by default (as with -w), people who have
    lossless wired links will need to configure them manually; this is bad
    for Homenet, which is expected to use wired links extensively, but
    perhaps it doesn't matter, Homenet can accept 15 seconds instead of 6;

 2. use the tunnel type by default; similar to the above, but perhaps
    slightly less drastic;

 3. find a way to make babeld less sensitive to links flapping in
    non-redundant networks (it already behaves well when the flapping link
    is redundant, but in a non-redundant topology it advertises every link
    flap across the routing domain).

Opinions?

-- Juliusz

---

From: Mitar <[email-masked]>
Date: Thu, 17 Dec 2015 00:42:45 +0100

Hi!

>     flap across the routing domain).

What about detecting that an ethernet link is flapping and in that
case it turns on link-quality measurement?

Because I think it is pretty uncommon for ethernet link to be coming
up and down. The main reason why Babel does not use link quality on
the ethernet link is that, arguably, failures on ethernet often means
a permanent failure and it is good to detect this quick. But if the
link is coming back soon afterwards and then again down and so on,
Babel should decide to instead go for packet loss? I think that would
be the best default.

Mitar

-- 
http://mitar.tnode.com/
https://twitter.com/mitar_m

---

From: Philip Homburg <[email-masked]>
Date: Thu, 17 Dec 2015 14:04:49 +0100

>    perhaps it doesn't matter, Homenet can accept 15 seconds instead of 6;

Is there room in the protocol for a router to announce what link type it
is on? I.e., a router on wifi announces wifi and when a router that is on
wired receives an announcement from a router on wifi it knows that there
a bridge somewhere.

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 17 Dec 2015 16:10:37 +0000

> interface is treated as a lossless wired interface, and if it suffers packet loss,
> there is repeated link flapping.  

I've had a lot of experience with trying to use both Wi-Fi and powerline bridges. Powerline bridges can suffer from a lot of the same flapping as Wi-Fi. 
I'm aware of an IPTV provider who experimented with using 802.11-Ethernet bridges to connect set-top boxes (STBs) to the router. They wanted super-duper good wireless for the STBs, so they used proprietary 802.11-based technology and this 802.11 was dedicated to just the STBs (so it wasn't using the same Wi-Fi as the home network). They ended up building this 802.11 technology into the STBs and into the home router (separate radio from the home Wi-Fi), because the bridged connection just wasn't resilient. I'm also aware of powerline bridges being used in lots of IoT "smart home" deployments. And then there are the powerline to Wi-Fi bridges (Wi-Fi extenders) which I'm seeing more and more of. 

The main problem I've seen with frequently flapping bridges is host IP/Ethernet stacks (where the host -- including router WAN "host" -- thinks it has an Ethernet connection) that give up "quickly" on the connection. Reboots or even just disabling and then enabling the network interface cause the connection to be seen again and IP connectivity re-established. Forced DHCP release/renew doesn't tend to work, which leads me to believe the problem may be in the Ethernet stack, and not with IP and DHCP. I've never bothered trying to figure out the root cause -- instead I just stop using the bridges that don't work, and do something different that does work. I've definitely seen this problem a lot, though.

My take-away from my experiences:
Bridges are incredibly useful -- when they work. The better they work, the more people will use them. Homenets (including hosts) need to be resilient to flapping Ethernet links.
There are host IP/Ethernet interface issues (including router WAN "host") that are prevalent and need to be solved independent of anything Babel does. Issues that cause hosts to decide there is no connectivity on Ethernet links that flap.
The problem is not limited to Wi-Fi bridges. It also exists on powerline.
I'm curious as to the prevalence of LLDP in bridges. If LLDP is being included in bridges, it could be used to detect them. Hmm. I should try to find out.
I don't know right now what the right answer is for Babel.

Barbara

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 18 Dec 2015 10:53:42 +0100

> is on?

This could be carried by a sub-TLV of Hello (or a sub-TLV of IHU if you
want to make it per-host).

> I.e., a router on wifi announces wifi and when a router that is on wired
> receives an announcement from a router on wifi it knows that there
> a bridge somewhere.

Not a bad idea, but I'm a little hesitant to implement that, since it
would require defining a taxonomy of interface types at the protocol
level.  (Currently the taxonomy exists in the implementation, but it
doesn't appear in the protocol -- the protocol only knows about metrics.)

-- Juliusz

---

From: Markus Stenberg <[email-masked]>
Date: Fri, 18 Dec 2015 14:06:03 +0200

On 18.12.2015, at 11.53, Juliusz Chroboczek <[email-masked]> wrote:
> doesn’t appear in the protocol -- the protocol only knows about metrics.)

I am somewhat suspicious about the idea too, although for different reasons. Notably, most things that run routing protocol these days use ethernet plugs for them (+- some wireless ad-hoc people).

Even if I have ‘modern cheapo router’ on both ends of a powerline bridge, odds are, it is just using ethernet plug. So link type autodetection is out of the window, and if I manually need to specify that this ethernet is really powerline, I might as well configure the routing protocol, and I find both ideas non-appealing.

Cheers,

-Markus

---

From: Philip Homburg <[email-masked]>
Date: Fri, 18 Dec 2015 13:37:22 +0100

>doesn't appear in the protocol -- the protocol only knows about metrics.)

It is probably not such a good idea to literally say 'wifi', but if a 
set of metrics could be defined that describe that expected stability of a
link then a router could annouce which it has configured or discovered.

One thing to consider though, in my experience lossy links make for a very
bad user experience.

So if a disconnect happens as a result of user action, say moving a wifi
device from one room to another, then losing the link is not a bad
deal.

If links just come and go then ultimately users will just complain that the
internet is bad. If bad links are unavoidable, it is better to run a 
tunneling protocol on top of those links with forward error correction
or other mechanism to make the link more reliable.

(I.e, it might better if babel degrades gracefully, instead of trying to 
make bad links 'work')