Subject: RE: [Ospf-manet] Simulation Accuracy??

From: "Henderson, Thomas R" <[email-masked]>
Date: Tue, 31 Jan 2006 23:18:32 -0800

> -----Original Message-----
> From: Russ White [mailto:[email-masked]] 
> Sent: Tuesday, January 31, 2006 6:02 PM
> To: [email-masked]
> Subject: [Ospf-manet] Simulation Accuracy??
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Y'all:
> 
> I was recently on a call where some of the limitations of the 
> simulations we're using were outlined. I don't beleive the 
> majority of the people on the list understand these 
> limitations, so I believe it's important to lay them out, 
> since the entire decision on which algorithm to use appears 
> to be set to be decided strictly on these rather limited simulations.

You may have that perception, but I don't think it is accurate or fair
representation of the work and discussion that's gone into this so far. 

> 
> - -- The simulations, as they are currently written, do not 
> emulate many receivers with different radio ranges, or 
> changing radio ranges.
> 
> - -- The simulations, as they are currently written, do not 
> emulate different devices travelling at different speeds.

As I mentioned to Pratik, we (or others) could expose these settings in
the API-- the radio range is pretty trivial, the variable mobility is
slightly more involved but not difficult.  We plan to add this soon, so
we can look at whether these factors are strongly affecting the results.

> 
> I believe some of my other concerns with the simulations, 
> including such things as the density of the nodes in the 
> test, may be addressed by some of the proposals I've been 
> told are forthcoming on list. Still, all in all, I believe 
> analysis for protocol correctness is a much better method of 
> choosing protocol enhancements, rather than simulation results.
> 

Russ, we've put it all on the table-- implementation code, simulation
code, simulation results, analysis.  We are considering all factors.

Tom

_______________________________________________
Ospf-manet mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ospf-manet

---

From: "Spagnolo, Phillip A" <[email-masked]>
Date: Wed, 1 Feb 2006 07:12:42 -0800

All,

> > emulate different devices traveling at different speeds.
> 

Just for clarification.  The about statements are not completely true.  

Radio range:  The current code sets the max radio range for all nodes at
a certain value.  The actual value is probilistically below it.  See our
tech report:  Section 3.2.1 and B.5.
<http://hipserver.mct.phantomworks.org/ietf/ospf/ospf-manet-techrpt.pdf>

Velocity:  The velocity is not the same for each node.  The max value is
set to (Vmax) and each node chooses a value uniformly between 0 and
Vmax.

Phil

---

From: "Bose, Pratik" <[email-masked]>
Date: Wed, 01 Feb 2006 11:50:03 -0500

So, what is the plan going forward then, are we going to have simulation
results with and without tweaking of range/mobility ?

If we are tweaking range and/or mobility, we should discuss the mix of
nodes at different velocity or range once - I have that information -
let me know when things are ready

Towards the discussion of simulation accuracy: This is something we all
know - there are always 'n' things that simulation cannot achieve and
one runs into that often - sometimes protocol correctness helps complete
the picture, sometimes a testbed emulation, sometimes the a-ha moment
arrives only after actual deployment. It might be worthwhile to compile
known 'n' things and weigh them against the known characteristics of the
protocol (information which we already have and have discussed often) as
a safe harbor in the final choice. Some simplistic examples: 

Certain scenarios had 300 nodes which were tailored to 100 - we can say
that right now what is good for 100 is good for 300 and move on or say
something else like that this algorithm seems to scale better under
these conditions

Current simulation does not account for effects for terrain completely -
however as pointed out by Philip, the code allows one to specify a max
for mobility/range and during the scenario a uniform/probablistic value
is chosen upto that max value - maybe that sufficiently accounts for
terrain, maybe not....

What do you guys think ?

> > 
> > - -- The simulations, as they are currently written, do not emulate 
> > many receivers with different radio ranges, or changing 
> radio ranges.
> > 
> > - -- The simulations, as they are currently written, do not emulate 
> > different devices travelling at different speeds.
> 
> As I mentioned to Pratik, we (or others) could expose these 
> settings in the API-- the radio range is pretty trivial, the 
> variable mobility is slightly more involved but not 
> difficult.  We plan to add this soon, so we can look at 
> whether these factors are strongly affecting the results.
> 
> > 
> > I believe some of my other concerns with the simulations, including 
> > such things as the density of the nodes in the test, may be 
> addressed 
> > by some of the proposals I've been told are forthcoming on list. 
> > Still, all in all, I believe analysis for protocol correctness is a 
> > much better method of choosing protocol enhancements, rather than 
> > simulation results.
> > 
> 
> Russ, we've put it all on the table-- implementation code, 
> simulation code, simulation results, analysis.  We are 
> considering all factors.
> 
> Tom
>