Subject: Re: [Ospf-manet] Re: Ospf-manet Digest, Vol 9, Issue 11

From: Philippe Jacquet <[email-masked]>
Date: Sun, 17 Sep 2006 11:20:31 +0200

Dear Richard,

You can imagine the largest number of scenarios in your proposal  and 
complicate it as well, the point is that we have only a very small 
feedback on real experiment or simulation. To my best  knowledge MDR 
have been evaluated only on 2x2 square with random  way point.

It is the first time I see an attempt to push a proposal that has  been 
validated on such a small set of scenarios and which provide little 
improvement and in most case worse performance than a  solution that 
have been validated on a very much larger scale as  you can check on the 
web.

The point on your MDR improvement seems really like an improvement  
dictated by the very specific and narrow case of the 2x2 square  
network. It does not seem to be a good way of design.

Philippe

Richard Ogier wrote:

> Philippe Jacquet wrote:
>
>> Richard Ogier wrote:
>>
>>>
>>>>
>>>>
>>>> However in a purely random network, using the dense connectivity 
>>>> parameters from Dr. Ogier's simulations (30 neighbors per router in 
>>>> 50 node network), the probability of 2 routers having the exact 
>>>> same neighborhood is small (using simple calculations, we can 
>>>> verify it to be something in between (1/3)^12 and (1/3)^13).
>>>
>>>
>>>
>>>
>>> But the probability of one router having a neighborhood that is a 
>>> subset of
>>> another router's neighborhood is much larger, which is more relevant.
>>> For example, if routers are positioned randomly in a circle, then 
>>> routers
>>> that are near the edge of the circle are more likely to have this
>>> property.
>>
>>
>>
>>
>> This case is strongly related to the unit disk graph model. It seems 
>> to me like an optimization specifically taylored to the intensively 
>> simulated 2x2 square where a significant number of neighborhoods are 
>> truncated by the border of the square. I don't think this would 
>> generalize very well regarding real implementations. 
>
>
>
> So you are saying that the optimization is specifically tailored to
> a unit disk or a 2x2 square (where you assume the radio range is
> one unit).  More generally, one might say the optimization is
> tailored to dense networks with a small diameter.
> Such networks are important, as are sparse networks and large
> diameter networks.  (Also, I see that Aniket explained how this
> optimization is important in hybrid networks.)
> You may be correct that this optimization helps performance the most
> in dense networks with a small diameter, but since such networks can
> occur and should be supported, I think that is enough reason to use
> the optimization.  The goal is to achieve good performance in a wide
> variety of scenarios, including dense networks.
>
> Richard
>
>>
>>
>>
>> Best regards,
>> Philippe
>>
>>>
>>
>>
>>
>
>
>

_______________________________________________
Ospf-manet mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ospf-manet

---

From: Russ White <[email-masked]>
Date: Mon, 18 Sep 2006 07:47:59 -0400

>> optimization is important in hybrid networks.)

Would you suggest that the OSPF WG adopt two proposals, one for each
network situation, and some mechanism to move between the two? IMHO, it
would be better to move both of the proposals we have into experimental,
until we have implementations, and some real world data to show us the
best technical solution.

We should never adopt standards based on a set of simulations.

:-)

Russ

- --
[email-masked] CCIE <>< Grace Alone

---

From: "Henderson, Thomas R" <[email-masked]>
Date: Mon, 18 Sep 2006 08:28:49 -0700

> -----Original Message-----
> From: Russ White [mailto:[email-masked]] 
> Sent: Monday, September 18, 2006 4:48 AM
> To: Richard Ogier
> Cc: [email-masked]
> Subject: Re: [Ospf-manet] Re: Ospf-manet Digest, Vol 9, Issue 11
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 

Russ and all,

I don't have a problem with putting OSPF MANET on an experimental
standards track.  Given the maturity of implementations and lack of
field experience, this seems to be the most prudent approach.  However,
I think it is a bad idea for the WG to continually develop two
experimental protocols or a hybrid thereof that are just variations on
the same theme, unless there is strong evidence for it for some reason.
This just causes confusion and complexity for users/implementors.  Bill
Fenner also spoke out against such an outcome at IETF-65.  IETF WGs
should make some engineering choices and not just throw multiple
proposals against the wall and hope that one sticks [1].

If I recall correctly, at the last WG meeting (March) where this was
discussed, although there were several people who said that they had
reached rough consensus to move forward on a design based on the Ogier
draft, there was sentiment expressed to evaluate whether an
simulation/implementation of the more recent INRIA proposal was
competitive with Ogier's.  INRIA agreed to work on this, but I have not
heard of any progress yet.

I think it is unfair to treat both of these proposals as if they are
currently on equal footing; for one proposal, there are two independent
implementations and it has been extensively studied, while there are no
complete implementations or published results for the other, so claims
that it can do X or Y better than Ogier's approach are still
hypothetical.  

Regards,
Tom 

[1] Randy Bush, "Into the future with the internet vendor task force a
very curmudgeonly view or testing spaghetti: a wall's point of view,"
ACM Sigcomm CCR, pp. 67-68, October 2005.

---

From: Richard Ogier <[email-masked]>
Date: Mon, 18 Sep 2006 11:22:55 -0700

Russ White wrote:

>network situation, and some mechanism to move between the two?
>

My point was that the optimization helps performance the most in
networks with a small diameter.  But it does NOT hurt performance
in networks with a large diameter.  So this point cannot be used
to conclude that we need two protcocols.  OSPF-MDR was designed to
perform well in all scenarios.

Richard

>IMHO, it
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v1.4.2.2 (MingW32)
>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
>iD8DBQFFDodvER27sUhU9OQRAmXDAJ979kSRSsfJEGIkh7CNKYe7UJGfuwCfXQ6V
>PcxBIq8xebOA4cVkBDGuXWs=
>=r9nb
>-----END PGP SIGNATURE-----
>
>

---

From: Russ White <[email-masked]>
Date: Mon, 18 Sep 2006 15:45:17 -0400

>> perform well in all scenarios.

So, you believe that there is no requirement for optimization of
non-dense protocols, that accepting a solution that only optimizes one
problem, "as long as it doesn't harm other problems," is acceptable? Do
you have evidence that the other situations are never going to be
required or useful?

And going back to my original point:

It's inappropriate to put something on the standards track that has only
been tested in simulations. If there's not enough customer demand to
actually drive implementations that can be tested in real networks, then
there's not enough drive to move forward with a standards track
document, I would think.

OTOH, if we really want to base standards track on simulations, then we
should go about the work of designing simulation reports to substitute
for implementation reports, and we should also ask for simulation
results for any other proposed modifications to OSPF. Do you care to
make the argument that simulations should always be accepted in lieu of
actual implementations?

:-)

Russ

- --

---

From: "Joe Macker" <[email-masked]>
Date: Mon, 18 Sep 2006 16:11:10 -0400

I agree with Tom different taxonomies at experimental would be one thing but
protocols that are slight variants on a theme.  argh.. not attractive.

We had some very specific goals in the DT of not getting into an
overoptimization exercise with this work.  It was a fairly focused effort
initially with consistency with the OSPF approach a major goal. I feel the
DT accomplished its goals and we came to some good engineering conclusions
about the early approaches.  I personally believe we had rough consensus
enough to move on and had a design that meshed well with OSPF.  Other
Proactive MANET routing work continues with OSLRv2,etc and as Aniket says
that is a different framework with non-reliable flooding so it should not be
directly compared.

We don't standardize on simulation but we should remember one of the
implementations also is running code and has been trialed in various
experimental networks.  Understanding the potential pitfalls, I'll take
simulation over verbal jousting anyday.  But also the running code is there.

-Joe

>From: Henderson, Thomas R [mailto:[email-masked]] 
>Sent: Monday, September 18, 2006 11:29 AM
>To: Russ White
>Cc: [email-masked]
>Subject: RE: [Ospf-manet] Re: Ospf-manet Digest, Vol 9, Issue 11
>
> 
>
>> network situation, and some mechanism to move between the two? IMHO, 
>> it would be better to move both of the proposals we have into 
>> experimental, until we have implementations, and some real 
>world data 
>> to show us the best technical solution.
>> 
>I don't have a problem with putting OSPF MANET on an 
>experimental standards track.  Given the maturity of 
>implementations and lack of field experience, this seems to be 
>the most prudent approach.  However, I think it is a bad idea 
>for the WG to continually develop two experimental protocols 
>or a hybrid thereof that are just variations on the same 
>theme, unless there is strong evidence for it for some reason.
>This just causes confusion and complexity for 
>users/implementors.  Bill Fenner also spoke out against such 
>an outcome at IETF-65.  IETF WGs should make some engineering 
>choices and not just throw multiple proposals against the wall 
>and hope that one sticks [1].
>
>If I recall correctly, at the last WG meeting (March) where 
>this was discussed, although there were several people who 
>said that they had reached rough consensus to move forward on 
>a design based on the Ogier draft, there was sentiment 
>expressed to evaluate whether an simulation/implementation of 
>the more recent INRIA proposal was competitive with Ogier's.  
>INRIA agreed to work on this, but I have not heard of any progress yet.
>
>I think it is unfair to treat both of these proposals as if 
>they are currently on equal footing; for one proposal, there 
>are two independent implementations and it has been 
>extensively studied, while there are no complete 
>implementations or published results for the other, so claims 
>[1] Randy Bush, "Into the future with the internet vendor task 
>force a very curmudgeonly view or testing spaghetti: a wall's 
>point of view,"

---

From: Russ White <[email-masked]>
Date: Mon, 18 Sep 2006 16:14:48 -0400

> simulation over verbal jousting anyday.  But also the running code is there.

Do you believe there's enough customer drive to implement two solutions
in code, and see how they actually operate? I understand the
consideration of accepting simulations over verbal sparring, but,
historically, we've never, in the past, accepted a simulation as a
reason to place something on the standards track, AFAIK.

My point is that if we start now, we should make certain we understand
the implications not only for this work, but also all future work, and
decide how we're going to fit simulations into the process.

In the past, in fact, running code decided the matter, no matter how
theoretically perfect the other options have been.

:-)

Russ

- --

---

From: "Joe Macker" <[email-masked]>
Date: Tue, 19 Sep 2006 08:53:31 -0400

Russ:

No running code arguments from me. In that regard, I believe we had a good
baseline out of the design team work -- the early Boeing implementation had
several solutions to cross compare.   So there was some customer drive to
fund that early analysis and the results looked pretty conclusive with A vs.
B.  Again the running code is there in addition to simulation and there is a
level of confidence in it. Gaining more confidence through more
experimentation is a always a good thing.

I do believe putting together a good node wireless mobile wireless
experiment is not cheap nor conclusive because of the additional performance
variables and scenario dependent issues involved.  The technical community
should use analysis, simulation, emulation (actual implemented code), and
live deployments or experiments to help make good engineering choices in
this realm. Nowadays, many of us adapt actual running code to simulation and
emulation to supplement good practice deployment testing and speed up the
engineering process.

-Joe
>Sent: Monday, September 18, 2006 4:15 PM
>To: Joe Macker
>Cc: 'Henderson, Thomas R'; [email-masked]
>> experimental networks.  Understanding the potential pitfalls, I'll 
>> take simulation over verbal jousting anyday.  But also the 
>running code is there.
>
>Do you believe there's enough customer drive to implement two 
>solutions in code, and see how they actually operate? I 
>understand the consideration of accepting simulations over 
>verbal sparring, but, historically, we've never, in the past, 
>accepted a simulation as a reason to place something on the 
>standards track, AFAIK.
>
>My point is that if we start now, we should make certain we 
>understand the implications not only for this work, but also 
>all future work, and decide how we're going to fit simulations 
>into the process.
>
>In the past, in fact, running code decided the matter, no 
>matter how theoretically perfect the other options have been.
>
>:-)
>
>Russ
>
>- --
>iD8DBQFFDv44ER27sUhU9OQRAqroAJ92u7//2wTq8WolxxGFcFPwxpK16wCeOlAD
>ZjI4dVJk0sAbbsIWKgC3BA8=
>=2HcF

---

From: Philippe Jacquet <[email-masked]>
Date: Thu, 28 Sep 2006 14:33:18 +0200

Dear Russ and all,

I feel that we should not systematically be against simulations in the
standardization process. Simulations greatly help understanding complex
problems. But I don't understand the claim that deploying a wireless OSPF
might be more complex or more costly than deploying a Manet. Moreover, there is a
proposal which has hundreds of options (who has understood the BMDR stuff?)
and most of them really look as if they were taylored for only one set of
scenarios, in an unrealistic environment. This is definitely not a good way
to design.

A few years ago, the ospf WG discovered that usual ospf does not fit wireless
ad hoc networking at all. Not surprising since manet WG has been created exactly for solving this problem

A few set of simulations provided by Boeing proved that a manet-based solution
can solve the problem in a satisfactory way.  This is good news, but not
surprising again, since manet WG has accumulated a considerable experience on the
subject, had expertized a large number of proposal, and is ready to standardize
two main routing schemes which are satisfactory to every one.

A wireless-OSPF design team was then formed. At this time it was clear that
the best, most efficient and fastest process was to select a manet solution and to
let it receive the appropriate "ospfication" in order to receive the WG
blessing.

At this precise moment things went wrong. For a reason that I will never
understand a small subset of the design team, wrote a completely new 
draft directly from scratch which has never been deeply experimented, neither in manet nor in ospf. Now we are wasting time because there is no enough qualitative and quantitative feedback on this proposal which is not mature, and worse: nobody is able to understand how to cope with all its options and complexity issues.

Last year a manet based solution has again been proposed for the crucial problem
of flooding. More robust, less complex, it provides equivalent and even better
performance. And with the considerable advantage of being mature: it has been
tested deeply in the past, in a huge number of scenarios, many of them having
triggered some improvements to the scheme.  

The natural consequence should have been again to have taken this manet
solution and to taylor it for OSPF. But we heard that we should not
over-optimize and therefore we should push the freak. In my mind this is
nonsense, since the MPR solution is not an over-optimization. There is no
intricate options, it has no complexity issue, it just a simple and robust
solution that have been tested over and over. So why shouldn't it be taken as
the foundation of the wireless ospf extension?

Instead of this we hear that manet-ospf has nothing to do with manet and that
we have to solve different problems. I personnaly don't believe this, because a
pure manet solution showed that the problem can be solved and because a
manet-based solution is already solving most of the problems.

Of course there remains the option to exclude the word "MANET" from the name
of the design team, if there is a consensus that it is indeed non related to
manet extensions. 

Best regards,
Philippe

Russ White wrote:

>iD8DBQFFDvdNER27sUhU9OQRArnaAJ9VL/oh/PHoByOlIsAGqKQ+CvesagCfTkzg
>8ywtkusjYGp/tZ9SjJt/KZg=
>=GzXk

---

From: Russ White <[email-masked]>
Date: Thu, 28 Sep 2006 08:58:01 -0400

> problems. 

Agreed--but simulations should not be the basis for a draft standard. In
fact, the entire IETF history has been built around "rough code and
rough consensus," not "simulate and standardize." A simulation is a tool
to understand how something works, not a tool used to claim something
will work better than something else in the real world.

In the end, a simulation can be _very_ useful, but it cannot replace
real world experience.

> draft directly from scratch which has never been deeply experimented,
> neither in manet nor in ospf. Now we are wasting time because there is
> no enough qualitative and quantitative feedback on this proposal which
> is not mature, and worse: nobody is able to understand how to cope with
> all its options and complexity issues.

I would agree with this assessment....

Right now my understanding of our position is:

o There is one implementation of MPRs.
o There is a simulation of MPRs and MDRs.
o There is a great deal of argument about what the simulations actually
show.
o There are questions around the actual efficiency and use of MDRs
outside the simulated environment.
o There are no implementations of MDRs.
o There is no market drive or force to implement any new solution for at
least the next 3 to 5 years.

At this point, if the WG can't iron out how to reach consensus on the
implemented draft, and there's no known drive to implement the alternate
version, then perhaps the best bet is to take both experimental. This
would, IMHO, be better than going draft standard on something that has
not been implemented, and hoping we can iron the problems out in 3 to 5
years when there's some real market for doing the additional work a
second implementation requires.

Unlike some folks, I do not consider a simulation and "implementation."

:-)

Russ

- --

---

From: Acee Lindem <[email-masked]>
Date: Thu, 28 Sep 2006 09:04:38 -0400

Hi Philippe,
On a related topic, have you progressed over the last year in
implement your MPR draft in GTSNets?
Thanks,
Acee

Philippe Jacquet wrote:
> is not mature, and worse: nobody is able to understand how to cope 
> with all its options and complexity issues.
>

---

From: Acee Lindem <[email-masked]>
Date: Thu, 28 Sep 2006 09:16:34 -0400

Russ,

I'm not sure how much actual interoperability testing or deployment 
information
of any OSPF MANET implementation is in the public information domain
or whether such information could be made available.

Thanks,
Acee

Russ White wrote:
> iD8DBQFFG8bZER27sUhU9OQRAg6FAKCjZfTefaQm2x85+U0lYVi/OUGG4wCfU6AF
> +TF2Ie1olV4t+XeWfSMd1ag=
> =ineQ

---

From: Russ White <[email-masked]>
Date: Thu, 28 Sep 2006 09:29:24 -0400

> or whether such information could be made available.

Agreed, but that makes the problem even worse, not better. We have two
proposals:

o For one, any information about how it works in the field is unavailable.

o The other has only been simulated.

The result? "We should move forward with the simulated mechanism as a
draft standard," of course. Sorry if this doesn't make any sense to me....

:-)

Russ

- --

---

From: "Spagnolo, Phillip A" <[email-masked]>
Date: Thu, 28 Sep 2006 08:44:14 -0700

Russ,

> least the next 3 to 5 years.

When you talk about MPRs are you referring to ORs or a solution yet to
fully specified by Inria (OSPF-Inria)?  When people are using the term
MPRs, I think we need to be specific about which one we are talking
about.  ORs and OSPF-Inria are not the same.

Here is the way I see the position of OSPF-MANET.  Do we agree?

                  MDR             OR               OSPF-Inria
-------------------------------------------------------------
Specification      |Yes             Yes              No
Simulation         |GTNetS/OPNET    GTNetS           None
Implementation     |Quagga          Quagga/Cisco     None
Questions about    |                
outside simulation |Yes             Yes              Yes
Market drive       |??              ??               ??

When I say specification above, I mean to a level of detail that is
implemenatable.  When I say simulation, I mean in an OSPF framework.  
Finally, I don't think we can say MDRs, ORs, or OSPF-Inria have been
truly tested sufficiently outside of the simulation environment.  We (at
Boeing) have been focusing our efforts in this direction.

Sincerely,
Phil

Phil Spagnolo 
Network Technology Engineer 
The Boeing Company 
Phone:  425-373-2810

---

From: Richard Ogier <[email-masked]>
Date: Thu, 28 Sep 2006 10:42:51 -0700

>o There are no implementations of MDRs.

Yes there is.  There is an MDR patch for Quagga 0.98.5 at the Boeing site
http://hipserver.mct.phantomworks.org/ietf/ospf/

Mabye this implementation should be advertised better, so that more
people will try it out.

Regarding consensus, I agree with Joe Macker's recent statement:

>I personally believe we had rough consensus
>enough to move on and had a design that meshed well with OSPF.
>

I am not sure why there was a perception that we did not have
rough consensus at the last meeting.  INRIA did not join the
consensus, but I doubt they would ever agree to any
solution that does not include MPRs.

I am glad that Aniket agrees that OSPF-MDR is not complex and
is a natural extension of OSPF, having implemented it in OPNET.

What most of us are waiting for now is a fair comparison
of OSPF-MDR with INRIA's MPR proposal, using the
GTNetS simulator.  INRIA has been given plenty of time to
implement their MPR solution in GTNetS.
As has been discussed, INRIA seems to be performing a
simulation comparison using their own code, but I thought
the plan was for INRIA to cooperate with Boeing to implement
their solution in GTNetS, so that we could have a fair
comparison with credible results.

Richard

---

From: Emmanuel Baccelli <[email-masked]>
Date: Thu, 28 Sep 2006 20:24:46 +0200

Hi Phil,

I disagree with the status you put for OSPF-Inria specification, which by the
way is called MPR-OSPF extension since it has been published as
http://tools.ietf.org/wg/ospf/draft-baccelli-ospf-mpr-ext-01.txt
If you have comments on how it sould be specified further, please let us know
about these details and we'll happilly collaborate with you.

Emmanuel

>                   MDR             OR               OSPF-Inria

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

---

From: Acee Lindem <[email-masked]>
Date: Thu, 28 Sep 2006 14:46:46 -0400

Hi Emmanuel,

I still wouldn't put the OSPF-MPR proposal on equal standing with the other
two implemented drafts. It has been almost a year since it was presented in
Dallas and we don't have an implementation. This doesn't say much for
simplicity as one of the purported benefits.

Thanks,
Acee