Subject: [manet] Subject: DLEP link metric discussion - is "mesh" TLV needed or not

From: [email-masked]
Date: Tue, 23 Apr 2013 14:14:44 -0500

|On Tue, Apr 23, 2013 at 5:57 PM, Rex Buddenberg wrote:
|> On Tue, 2013-04-23 at 11:12 +0200, Henning Rogge wrote:
|>
|>> It seems that many IP radios try to establish a transparent layer-2 
mesh
|>> between the radios on the same channel. The fact that the linklayer
|>> forms a mesh is useful knowledge for a layer-3 MANET routing protocol,
|>> it allows an implementation to optimize its flooding and forwarding
|>> routings (by not using two-hop routes within the layer-2 mesh domain).
|>
|> There are two issues here that need to be disentangled.
|>
|> The first is knowledge of connectivity at layer 2, and feeding it 
upward
|> to the routing table.  OK, I'll live with that.
|>
|> But the second is the MAC.  With a shared media, you need one
|> (conversely, with point-point media you don't).  So capacity of the
|> layer 2 radio network is baseband capacity prorated across all the SS 
in
|> the segment .... minus the overhead used by the MAC.  In the case of
|> ethernet derivatives (like WiFi), that overhead may reach 80%!  And it
|> gets worse as the segment gets more congested ... so one of the metrics
|> you would need to get hold of is how congested the network segment
|> is ... so you don't overload and stall it.
|
|I am not sure how this is connected to a "does/does-not layer-2 mesh"
|TLV for DLEP.

I claim that
1. Performance is sometimes worse when IP routers arbitrarily work around 
multihop (mesh) routing below IP - so router-to-router interfaces, such as 
DLEP, should support mesh wireless networks, and
2. IP routers can make an informed decision based upon metric information 
without having to know if the layer-2 is a mesh radio network or not.

1.  Performance is sometimes worse when IP routers work around multihop 
(mesh) routing below IP - so router-to-router interfaces, such as DLEP, 
should support mesh wireless networks.

All of the IP broadcast (and some of the directional) military ad hoc 
networking waveforms that I know of perform multihop (mesh) networking at 
layer 2 below IP (analogous to how 802.3 bridge is multihop below IP). 
Note that I use the term ?networking waveform? to be equivalent to a radio 
with layer 2 mesh networking and MAC.

The multihop routing below IP is to enable optimized cross-layer 
optimization between the mesh routing and the MAC to minimize interference 
and, in many cases, to optimize spatial reuse of spectrum, especially for 
multicast/broadcast traffic in broadcast (omnidirectional) MACs.

Some example multihop routing/MAC cross-layer techniques include:
- multicast/broadcast spectrum (time slot) allocations to maximize spatial 
reuse capacity depending upon traffic and topology
- network coding from overhead transmissions from multiple nodes
- passive (rather than active) acknowledgement using overheard 
transmission of next radio hop
- adaptive routing on transmission by transmission basis depending upon 
what next few spectrum allocations are coming up

For example, consider a 5 node IP network {A, B, C, D, E} with 2 
networking waveforms radios (i.e. multihop ad hoc layer 2 networks) at 
each node.  The first radio has a narrowband (low data rate) waveform with 
long transmission range with following links {A-B, B-C, C-D, C-E, and 
D-E}.  The second radio has a wideband (high data rate) waveform with 
short transmission range with the following links {C=D, C=E, and D=E}. 
Note that although nodes A and B have wideband radios, they don?t have any 
wideband links due to the long distance.

     Narrowband waveform connectivity:
                              +---+
                              |   |
          +---------+---------+-+-+
          |         |         | | |
          A         B         C D E
          +         +         +=+=+
                              +===+
     Wideband waveform connectivity:

The IP narrowband metric from node X to Y is (x-y).  E.g. the narrowband 
IP link from node C to E is (c-e).

The IP wideband metric from node X to Y is (X-Y).  E.g. the wideband link 
from node C to E is (C-E).

Now the layer 2 networks optimize the shared spectrum through cross layer 
optimization, so that sometimes (a-c) is better than (a-b) and (b-c) 
depending upon scenario.  Similarly, (C=E) is sometimes better than (C=D) 
and (D=E).

However, because the wideband layer 2 network has better performance,
(a-c) and (C=E) is better than (a-e).

Thus, the IP Router at radio node A will select node C as the next IP hop 
in the IP route to E.

2. IP routers can make an informed decision based upon metric information 
without having to know if the layer-2 is a mesh radio network or not.

I think that radio-to-router interfaces, such as DLEP, should allow the 
networking waveform in the radio to advertise the MAC address of the next 
IP neighbor whether than IP neighbor is one or more IP hops away. (From my 
reading of draft-ietf-manet-dlep-04.pdf, I think that DLEP would support 
either IP neighbors that are one RF hop away or multiple RF hops away.)

Thus, in the example above, the IP router at A can get radio link 
information to the IP routers at the other nodes, whether they are one or 
more radio-hops away.  The routers can then look at the metrics to make 
the choice on whether to go in a single IP hop route or multiple IP hop 
route across the networks.

I.e. if (a-c) is better than (a-b) and (b-c), then the router at A would 
route directly to IP router at C rather than routing via IP router at B 
using the knowledge of the metrics without having to know that the radio 
was a mesh network.

Jim Stevens

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Tue, 23 Apr 2013 19:54:06 +0000

Jim,

On Apr 23, 2013, at 3:14 PM, <[email-masked]<mailto:[email-masked]>>
 <[email-masked]<mailto:[email-masked]>> wrote:

|>> It seems that many IP radios try to establish a transparent layer-2 mesh
|> The first is knowledge of connectivity at layer 2, and feeding it upward
|> layer 2 radio network is baseband capacity prorated across all the SS in

I claim that
1. Performance is sometimes worse when IP routers arbitrarily work around multihop (mesh) routing below IP - so router-to-router interfaces, such as DLEP, should support mesh wireless networks, and
2. IP routers can make an informed decision based upon metric information without having to know if the layer-2 is a mesh radio network or not.

1.  Performance is sometimes worse when IP routers work around multihop (mesh) routing below IP - so router-to-router interfaces, such as DLEP, should support mesh wireless networks.

All of the IP broadcast (and some of the directional) military ad hoc networking waveforms that I know of perform multihop (mesh) networking at layer 2 below IP (analogous to how 802.3 bridge is multihop below IP).  Note that I use the term “networking waveform” to be equivalent to a radio with layer 2 mesh networking and MAC.

The multihop routing below IP is to enable optimized cross-layer optimization between the mesh routing and the MAC to minimize interference and, in many cases, to optimize spatial reuse of spectrum, especially for multicast/broadcast traffic in broadcast (omnidirectional) MACs.

Some example multihop routing/MAC cross-layer techniques include:
- multicast/broadcast spectrum (time slot) allocations to maximize spatial reuse capacity depending upon traffic and topology
- network coding from overhead transmissions from multiple nodes
- passive (rather than active) acknowledgement using overheard transmission of next radio hop
- adaptive routing on transmission by transmission basis depending upon what next few spectrum allocations are coming up

For example, consider a 5 node IP network {A, B, C, D, E} with 2 networking waveforms radios (i.e. multihop ad hoc layer 2 networks) at each node.  The first radio has a narrowband (low data rate) waveform with long transmission range with following links {A-B, B-C, C-D, C-E, and D-E}.  The second radio has a wideband (high data rate) waveform with short transmission range with the following links {C=D, C=E, and D=E}.  Note that although nodes A and B have wideband radios, they don’t have any wideband links due to the long distance.

The IP narrowband metric from node X to Y is (x-y).  E.g. the narrowband IP link from node C to E is (c-e).

The IP wideband metric from node X to Y is (X-Y).  E.g. the wideband link from node C to E is (C-E).

Now the layer 2 networks optimize the shared spectrum through cross layer optimization, so that sometimes (a-c) is better than (a-b) and (b-c) depending upon scenario.  Similarly, (C=E) is sometimes better than (C=D) and (D=E).

Thus, the IP Router at radio node A will select node C as the next IP hop in the IP route to E.

2. IP routers can make an informed decision based upon metric information without having to know if the layer-2 is a mesh radio network or not.

I think that radio-to-router interfaces, such as DLEP, should allow the networking waveform in the radio to advertise the MAC address of the next IP neighbor whether than IP neighbor is one or more IP hops away. (From my reading of draft-ietf-manet-dlep-04.pdf, I think that DLEP would support either IP neighbors that are one RF hop away or multiple RF hops away.)

Yes, that's correct. In the current 04 version of DLEP, the router "doesn't know, doesn't care" if the destination MAC is 1 or > 1 RF hops away. At least in the Cisco implementation, there is an implicit trust in the radio to alter metrics accordingly - e.g. if you get into a situation where multiple radio stations funnel through a single device in the middle of your network, the radios SHOULD downgrade metrics so as to avoid network collapse.

Thus, in the example above, the IP router at A can get radio link information to the IP routers at the other nodes, whether they are one or more radio-hops away.  The routers can then look at the metrics to make the choice on whether to go in a single IP hop route or multiple IP hop route across the networks.

I.e. if (a-c) is better than (a-b) and (b-c), then the router at A would route directly to IP router at C rather than routing via IP router at B using the knowledge of the metrics without having to know that the radio was a mesh network.

Jim Stevens

This helps me put my finger on what's bothering me about the proposal for a Mesh TLV - essentially, the radio is working its fanny off to create a topology, and we're trying to give the router to ability to say (in effect), "No, I don't think so", based on the value of this TLV. It just smells like a layer violation to me (or one that's about to happen).  If you're going to trust the radio to carry the traffic, you should trust the radio to do it in the most efficient manner possible, and trust the radio to give you the correct metrics, based on the physical topology.

Regards,
Stan

---

From: Rex Buddenberg <[email-masked]>
Date: Tue, 23 Apr 2013 13:21:38 -0700

Stan,

This triggers some phobias beyond just hop count.  I'm unclear about
whether this is a 'standards' issue or a 'best practices' issues, but we
ought to understand some implications.

The fundamental advantage of IP and routers is that they execute
_reliable_crossover_ from one route to another by design.  This is what
makes packet switching superior to circuit switching (and Paul Baran was
right!)  In textbook, reliable crossover is the second principle of high
availability engineering.

20 years ago, lots of campus networks were bridged networks (with the
only router at the campus <--> outside world spot).  Each building had a
bridge in it.  
     As the network grew, and as reliance on it grew, we started adding
alternate routes to the campus network.  (First principle of high Ao
engineering, btw).  This caused ringing in the bridged ethernets --
frames would circulate forever.  So IEEE 802.1 cooked up the spanning
tree algorithm so that bridges could construct rudimentary routing
tables (at layer 2) and control the ringing -- BY KILLING OFF THE
REDUNDANT CONNECTIONS.  Direct violation of Principle 2.  
     We worked out of this mostly by replacing the bridges with routers
because layer 3 knows better.  

> This helps me put my finger on what's bothering me about the proposal
> for a Mesh TLV - essentially ...
> 
So yeah, I echo your worry here, but for different reasons.

On Tue, 2013-04-23 at 19:54 +0000, Stan Ratliff (sratliff) wrote:
> Jim,  
> 
> On Apr 23, 2013, at 3:14 PM, <[email-masked]>
>  <[email-masked]> wrote:
> 
> > 1. Performance is sometimes worse when IP routers arbitrarily work
> > around multihop (mesh) routing below IP - so router-to-router
> > interfaces, such as DLEP, should support mesh wireless networks,
> > and 
> > 2. IP routers can make an informed decision based upon metric
> > information without having to know if the layer-2 is a mesh radio
> > network or not. 
> > 
> > 
> > 1.  Performance is sometimes worse when IP routers work around
> > multihop (mesh) routing below IP - so router-to-router interfaces,
> > such as DLEP, should support mesh wireless networks. 
> > 
> > All of the IP broadcast (and some of the directional) military ad
> > hoc networking waveforms that I know of perform multihop (mesh)
> > networking at layer 2 below IP (analogous to how 802.3 bridge is
> > multihop below IP).  Note that I use the term “networking waveform”
> > to be equivalent to a radio with layer 2 mesh networking and MAC. 
> > 
> > optimization between the mesh routing and the MAC to minimize
> > interference and, in many cases, to optimize spatial reuse of
> > spectrum, especially for multicast/broadcast traffic in broadcast
> > (omnidirectional) MACs. 
> > 
> > - multicast/broadcast spectrum (time slot) allocations to maximize
> > spatial reuse capacity depending upon traffic and topology 
> > - adaptive routing on transmission by transmission basis depending
> > upon what next few spectrum allocations are coming up 
> > 
> > networking waveforms radios (i.e. multihop ad hoc layer 2 networks)
> > at each node.  The first radio has a narrowband (low data rate)
> > waveform with long transmission range with following links {A-B,
> > B-C, C-D, C-E, and D-E}.  The second radio has a wideband (high data
> > rate) waveform with short transmission range with the following
> > links {C=D, C=E, and D=E}.  Note that although nodes A and B have
> > wideband radios, they don’t have any wideband links due to the long
> > distance. 
> > 
> > The IP narrowband metric from node X to Y is (x-y).  E.g. the
> > narrowband IP link from node C to E is (c-e). 
> > 
> > Now the layer 2 networks optimize the shared spectrum through cross
> > layer optimization, so that sometimes (a-c) is better than (a-b) and
> > (b-c) depending upon scenario.  Similarly, (C=E) is sometimes better
> > than (C=D) and (D=E). 
> > 
> > 2. IP routers can make an informed decision based upon metric
> > information without having to know if the layer-2 is a mesh radio
> > network or not. 
> > 
> > I think that radio-to-router interfaces, such as DLEP, should allow
> > the networking waveform in the radio to advertise the MAC address of
> > the next IP neighbor whether than IP neighbor is one or more IP hops
> > away. (From my reading of draft-ietf-manet-dlep-04.pdf, I think that
> > DLEP would support either IP neighbors that are one RF hop away or
> > multiple RF hops away.) 
> 
> 
> Yes, that's correct. In the current 04 version of DLEP, the router
> "doesn't know, doesn't care" if the destination MAC is 1 or > 1 RF
> hops away. At least in the Cisco implementation, there is an implicit
> trust in the radio to alter metrics accordingly - e.g. if you get into
> a situation where multiple radio stations funnel through a single
> device in the middle of your network, the radios SHOULD downgrade
> metrics so as to avoid network collapse. 
> 
> > 
> > information to the IP routers at the other nodes, whether they are
> > one or more radio-hops away.  The routers can then look at the
> > metrics to make the choice on whether to go in a single IP hop route
> > or multiple IP hop route across the networks. 
> > 
> > I.e. if (a-c) is better than (a-b) and (b-c), then the router at A
> > would route directly to IP router at C rather than routing via IP
> > router at B using the knowledge of the metrics without having to
> > know that the radio was a mesh network. 
> > 
> > 
> > 
> > Jim Stevens 
> > 
> > 
> > 
> 
> 
> This helps me put my finger on what's bothering me about the proposal
> for a Mesh TLV - essentially, the radio is working its fanny off to
> create a topology, and we're trying to give the router to ability to
> say (in effect), "No, I don't think so", based on the value of this
> TLV. It just smells like a layer violation to me (or one that's about
> to happen).  If you're going to trust the radio to carry the traffic,
> you should trust the radio to do it in the most efficient manner
> possible, and trust the radio to give you the correct metrics, based
> on the physical topology. 
> 
> 
> Regards,
> Stan
> 
> 
> _______________________________________________
> manet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/manet

---

From: [email-masked]
Date: Tue, 23 Apr 2013 16:52:17 -0500

|On Tue, Apr 23, 2013 at 04/23/2013 03:21 PM, Rex Buddenberg wrote:
|Stan,
|
|This triggers some phobias beyond just hop count.  I'm unclear about
|whether this is a 'standards' issue or a 'best practices' issues, but we
|ought to understand some implications.
|
|The fundamental advantage of IP and routers is that they execute
|_reliable_crossover_ from one route to another by design.  This is what
|makes packet switching superior to circuit switching (and Paul Baran was
|right!)  In textbook, reliable crossover is the second principle of high
|availability engineering.
|
|20 years ago, lots of campus networks were bridged networks (with the
|only router at the campus <--> outside world spot).  Each building had a
|bridge in it. 
|     As the network grew, and as reliance on it grew, we started adding
|alternate routes to the campus network.  (First principle of high Ao
|engineering, btw).  This caused ringing in the bridged ethernets --
|frames would circulate forever.  So IEEE 802.1 cooked up the spanning
|tree algorithm so that bridges could construct rudimentary routing
|tables (at layer 2) and control the ringing -- BY KILLING OFF THE
|REDUNDANT CONNECTIONS.  Direct violation of Principle 2. 
|     We worked out of this mostly by replacing the bridges with routers
|because layer 3 knows better.
|
|
|> This helps me put my finger on what's bothering me about the proposal
|> for a Mesh TLV - essentially ...
|> 
|So yeah, I echo your worry here, but for different reasons.

Rex,

I think I understand and agree with the different point you are making, 
but I?m not sure I caught or understood them all.

1 I think you are arguing (and I agree) against the problem of flood 
storms that can arise from un- (or under-constrained) forwarding/flooding 
of traffic, such as with pure bridging alone.

2 I think you are arguing (and I agree) that a some routing protocols can 
cause reliability problems by their very design (such as the delay for the 
original IEEE 802.1D spanning tree algorithm to adapt to link changes as 
compared to the more recent IEEE 802.1aq algorithm that supports multiple 
paths).

3 I think the major point you are arguing (and I agree) is that IP routers 
have a more global network view than layer 2 networks do (by definition) 
and can do a better job of global network routing across multiple networks 
(in fact the very basis for the original DARPA CATANET model that became 
the Internet).

4 I?m not sure, but you appear to be arguing against layer 2 routing below 
IP in wireless network, possibly in all cases.

I agree that there are some wireless layer 2s that don?t do a good job of 
ad hoc routing below IP.  I think single radio channel 802.11 in multi-hop 
ad hoc mode is a good example.

However, there are other wireless layer 2s that do a good job because they 
were designed to optimize for specific conditions.  And these wireless 
layer 2s can do a better job of optimizing performance *within* that 
wireless network than IP can without that specific MAC cross-layer 
information.

5.  I?m not sure, but you also seem to imply you have other things or 
implications that I didn't catch.

Regards,
Jim

---

From: Rex Buddenberg <[email-masked]>
Date: Tue, 23 Apr 2013 15:46:42 -0700

Jim,

good catches, imo, in line below:

On Tue, 2013-04-23 at 16:52 -0500, [email-masked] wrote:
> I think I understand and agree with the different point you are
> making, but I’m not sure I caught or understood them all. 
> 
> storms that can arise from un- (or under-constrained)
> forwarding/flooding of traffic, such as with pure bridging alone. 

Almost.  And probably close enough.  ethernet frames have no TTL like IP
datagrams, so they'll get forwarded across bridges forever.  I guess
that's 'unconstrained';-)
> 
> 2 I think you are arguing (and I agree) that a some routing protocols
> can cause reliability problems by their very design (such as the delay
> for the original IEEE 802.1D spanning tree algorithm to adapt to link
> changes as compared to the more recent IEEE 802.1aq algorithm that
> supports multiple paths). 

loud and clear; you got it.
> 
> 3 I think the major point you are arguing (and I agree) is that IP
> routers have a more global network view than layer 2 networks do (by
> definition) and can do a better job of global network routing across
> multiple networks (in fact the very basis for the original DARPA
> CATANET model that became the Internet). 

Yeah, again, we're agreeing.  I like your term 'global view'.  layer 2
forwarding requires much more homogeneity of the hops than routing does.
I'm not sure this impacts the DLEP discussion, but we're agreeing.
> 
> 4 I’m not sure, but you appear to be arguing against layer 2 routing
> below IP in wireless network, possibly in all cases. 

As a matter of best practices, I think that multiple-hop layer 2 is
generally not a good idea (I can be convinced of specific exceptions).  
  As a matter of economics and scale, past attempts have generally not
been wildly successful and durable.  
  As a matter of standards and protocol design, those observations may
not apply.
> 
> I agree that there are some wireless layer 2s that don’t do a good job
> of ad hoc routing below IP.  I think single radio channel 802.11 in
> multi-hop ad hoc mode is a good example. 

Stan made this point.  Obviously there are times when this may be the
only option you have.  But as soon as you understand the
4-orders-of-magnitude argument, you understand that you don't want them
very often.

Actually, I'd be more interested in the industry uptake on multi-hop
802.16.  The standard's there, it's ratified ... anybody selling,
buying, using it?

> 
> However, there are other wireless layer 2s that do a good job because
> they were designed to optimize for specific conditions.  And these
> wireless layer 2s can do a better job of optimizing performance
> *within* that wireless network than IP can without that specific MAC
> cross-layer information. 
> 
> 5.  I’m not sure, but you also seem to imply you have other things or

Jim, I'm not opposed to DLEP.  Contrary.  But the 'color me skeptical'
attitude comes from the fact that we've been here before.  Attempts in
IEEE 802 to enhance the ethertype function have flamed out (one of the
few cases where a PAR has been formally withdrawn).  And several cases
of cross-2/3 layer communication within IETF have not come to fruition
either (some of the non-starts have been within MANET WG if my memory
cells aren't too foggy).  If we're going to do better, we need to learn
from them.  If that's what you're reading that I'm implying, then right
on.