Subject: [homenet] Scaling and standards committees.

From: Jim Gettys <[email-masked]>
Date: Fri, 21 Nov 2014 10:50:14 -1000

At the IETF meeting, I made a fuss about scaling and ensuring we can
transition between protocols. Not only do protocols need to work in small
scale testing, they need to work at sufficient scale for the environment of
use. This concern applies both to HNCP and for Babel vs. ISIS decision the
working group is struggling with.

Here's why I was a PITA:

At One Laptop Per Child we made the attempt to use the 802.11s
not-yet-standard mesh networking (this was about 2006).  We failed
*disastrously*. At some scale between 15-30 laptops, our network "melted"
and became totally unusable, even though our software worked fine "in the
small" when testing at scale 5-10 laptops. As a result we had wonderful
demoware, which could not be deployed....

At the time, we identified three problems;
    1) 802.11s had N squared behavior in a dense mesh (where all 802.11
nodes can hear each other), and by the time N was of order 15-20 laptops,
disaster ensued.  This could be so bad that it entirely defeated carrier
sense and all laptop's WiFi would attempt to access the air simultaneously.
When we looked at the WiFi channel with a signal analyzer, all high
frequencies were gone, and a WiFi packet analyzer showed us that at most 14
bits of the wireless preamble were intact before a different laptop would
attempt to transmit. This was induce-able just by opening the laptops at
the same time, with the ensuing DHCP requests and other initial "chatter"
of the laptops.
   2) we were transmitting keys when hashes would do, causing our secure
distributed state protocol used for our collaborative applications to use
many more bytes than necessary.
    3) Even worse, our collaboration software, in the absence of a server,
used a "reliable" multicast protocol for collaboration, dropping the
effective bandwidth of 802.11 to basic rate. A little multicast can ruin
your whole day on WiFi.

But we completely missed bufferbloat.:

   4) a few years later I ran into bufferbloat at home and started to
understand what we had been seeing.  The OLPC's had 1027 packets of
buffering in the WiFi stack, and I know today how disastrous that was to
performance on the air. You can do the math of how many seconds of
buffering that represents on a network, particularly a busy WiFi network.
Without queue management, you get congestion collapse and an absolute
performance cliff when the link saturates....

How does this apply to homenet?
--------------------------------------------

o The IEEE 802.11s committee had decided that it wanted to rule > 32 nodes
as "out of bounds" for the design of 802.11s, ignoring the use case of a
room full of kids with laptops in a school entirely (not to mention, a room
full of people in a  conference room). As a result, they did not think
about the fact that their layer 2 mesh protocol ended up transmitting N^2
packets to discover the mesh exits.

*Moral*: once it becomes possible, people *will* attempt to use something
in ways that a committee may not have considered, or ruled as "out of
scope" and use it at higher scale than it considered.  But reality has this
nasty way of intervening as things scale up. If it *is possible*, people
*will* try to do it, even if the designers think it is a "bad idea" or "out
of scope".

This is why worrying about scale, and whether there is a viable protocol
transition strategy is so important.

o bufferbloat is well on its way to being solved, though it will take a few
more years to drain the wifi driver swamp.  Dave Taht and I, however, have
successful teleconferences over 5 hop Babel based mesh networks (he in CA,
me in MA), so time has moved on since homenet was started.  Meshes can and
will be over WiFi, despite the working group having ruled it out of scope
when homenet was formed.

o performance over mixed networks of WiFi and ethernet and other
technologies (e.g. power, trickle) will be commonplace, once we make it
easy to plug things in without configuration.

So details like whether it takes one message rather than two to distribute
routing information, and how often such messages must be exchanged, over
how much of the network is key to understanding the scaling of the network.

Transition strategy
-------------------------
Since I do not believe the world is static, and that there is much more to
learn in mesh networking, it seems likely we may face transition from one
generation of protocol to another, possibly sooner than we would like.
Similarly, HNCP may need a revision, for similar reasons. If we ignore such
protocol transitions, it is entirely possible homenet's work will be wasted.

Protocol evaluation
--------------------------
Therefore I think we need to dig into HNCP, and into Babel and ISIS to
understand the scaling and transition issues as we try to finish this up.
Both believed theoretical scaling of the protocols, along with as much real
data as we can get would be ideal.

A "zero configuration" system *will* get plugged together in unexpected
ways.  Making a system "idiot" proof attracts "idiots" of all
intelligence's... Including me.

I am hoping the working group can avoid the pain that caused the scars on
my back, and avoid standards committee fail by being unrealistic about what
people will try.

                            - Jim

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sat, 22 Nov 2014 13:19:16 +0100

> it seems likely we may face transition from one generation of [routing]
> protocol to another, possibly sooner than we would like.  Similarly,
> HNCP may need a revision,

In my (biased) opinion, this is the important bit of your mail.

I think we've already avoided the largest pitfall -- HNCP and the routing
protocol are now disjoint protocols, so that one can be changed with
minimal impact on the other.

But yes, we need to better understand what transition strategies are
possible for both HNCP and The One Routing Protocol.

-- Juliusz

---

From: Steven Barth <[email-masked]>
Date: Sat, 22 Nov 2014 13:48:12 +0100

> possible for both HNCP and The One Routing Protocol.
We have thought about that in terms of HNCP.

In general HNCP can operate behind routers that can give out DHCPv6-PD 
(hipnet and the like)
since that is the same mechanism that ISPs use to delegate prefixes to 
us. Interaction with those "legacy" networks by default would be limited 
i.e. the homenet believes its an uplink and firewalls incoming traffic.
We have specified a sort of "hybrid" category so that an admin can drop 
the firewall if he thinks the link and the legacy CPEs on it are 
reasonably secured. This helps for pure unicast case but doesn't really 
solve bi-directional MDNS or multicast interactions since that would 
require modifying the legacy devices.

Also we cannot really do much about CPEs that don't give out delegated 
prefixes since this would require NAT and I don't think we want to go there.

For the other direction HNCP includes a SHOULD-requirement in terms of 
providing DHCPv6-PD services to "legacy" CPEs like RFC 7084 or hipnet 
routers where the delegating homenet server announces the delegated 
prefixes on behalf of the legacy router and injects appropriate routes 
in the homenet.

Cheers,

Steven

---

From: Michael Richardson <[email-masked]>
Date: Sun, 23 Nov 2014 16:27:42 -0500

Jim Gettys <[email-masked]> wrote:
    > At the time, we identified three problems; 1) 802.11s had N squared
    > behavior in a dense mesh (where all 802.11 nodes can hear each other),
    > and by the time N was of order 15-20 laptops, disaster ensued.  This
    > could be so bad that it entirely defeated carrier sense and all

Aside from the standards process and scope issue, and the lack of, I guess,
running-code; I also wonder if part of the major difficulty is that 802.11s 
has to be implemented in a place (the wifi MAC firmware) which is usually
closed to innovation, and almost impossible to apply patches.

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Jim Gettys <[email-masked]>
Date: Mon, 24 Nov 2014 14:03:00 -0500

> closed to innovation, and almost impossible to apply patches.

​Actually, there is also a non-firmware version of 802.11s for Linux.  OLPC
used a firmware version as we suspended the CPU and let the mesh run
autonomously in the WiFi module, for better power consumption.

But the real point was that the 802.11s committee had ruled scaling it as
"out of scope", and ​

​people do what they will do without paying attention to the pious claims
that it was not "in scope" for the standard.  In short, homenet is creating
an attractive nuisance, and people will try to apply it in ways we don't
anticipate. Once burned, twice shy.
                                   - Jim