Subject: RE: [bmwg] Workplan Proposal for Routing Protcool Testing

From: Nick Ambrose <[email-masked]>
Date: Tue, 19 Jun 2001 12:39:54 -0700

I think perhaps you are mis-understanding our goals, which may be due to a
confusing explanation - apologies to the group if this is the case.

Let me try to clarify:

1.	We will define terms that can apply to all routing protocols. This
does not necessarily address every detail of the protocol, merely the common
elements. This allows us to discuss similar concepts when applied to
different protocols and I believe is useful. Additional terminology
documents on a per-protocol basis may become necessary and useful at some
point to add new terms.

2.	Develop protocol-independent tests. Again, we are not attempting to
address every aspect of testing every protocol, or even imply that that
might be possible in a general way. What we are attemting to do is find
commanality between protocols and define a framework and methodology for
testing those areas. Any are of a protocol that is not common would warrant
further documents (as i hinted at but maybe was not explicit enough in my
workplan proposal)

3.	Develop additional documents to describe application of the
framework to specific protocols (this is where i see the current BGP
convergence draft coming into play). These documents may make use of
additional terminology documents, as well as the framework terminology
document. 

They will have a section that describes application of the framework tests
to that particular protocol and may expand on the scenarios and add extra
protocol details (for example, specifying particular AS-PATH items for BGP,
or specific metrics for OSPF). The second section could add
protocol-specific tests that do not seem to fit into the framework, but can
still reuse the framework terms where possible and add new ones where not.
This still allows us to use the same terms to describe things where
possible.

I think it only makes sense to do it this way, otherwise we will spend ages
adding tests that are essentially identical (i.e. OSPF LSA capacity compared
to ISIS IP Route capacity) but use different terms, different metrics,
require multiple efforts to define and may end up slightly different and
getting out of sync over time.

I think that also, if you have any specific detailed technical concerns with
our proposal (and I acknowledge it may be difficult to come up with them
without seeing a draft of the method doc which we intend to publish soon)
then I would like to hear them, since this is a technical forum, not a
hand-waving "It'll never work, trust me on this one"-type statements which
is all I see in your previous comments. Specific cases where our framework
breaks down will allow us to fix it, or presuade us to abandon the work (as
appropriate)

Debby and I had already brought up the issue of  collaborating with both
groups involved in the other drafts but have not yet met with any positive
response.

Nick

> -----Original Message-----
> From: David Newman [mailto:[email-masked]]
> Sent: Monday, June 18, 2001 4:26 PM
> To: Nick Ambrose; [email-masked]
> Subject: RE: [bmwg] Workplan Proposal for Routing Protcool Testing
> 
> 
> 
> This appears to be an effort to apply one generic set of 
> definitions to ALL
> routing protocols -- or as the authors say, to work in a "non protocol
> specific manner."
> 
> As anyone who's had the pleasure of working with the various 
> protocols can
> attest, the one-size-fits-all approach is highly problematic. 
> There are too
> many protocols, with too many specific *and orthogonal* 
> requirements, for a
> generalized approach to be useful.
> 
> It's also untrue, as suggested here, that even a subset of 
> routing protocols
> share a core of similar behavior. Even among routing 
> protocols with similar
> functions (e.g., IGPs), there are significant differences in 
> terms of table
> update mechanisms,  table update contents, metric 
> calculation, and on and
> on.
> 
> May I respectfully suggest that the authors instead focus on 
> working from
> the specific toward the general? We don't today have tests 
> for specific
> protocols, and we need them.
> 
> dn
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On 
> Behalf Of Nick
> > Ambrose
> > Sent: Monday, June 18, 2001 4:03 PM
> > To: [email-masked]'
> > Subject: [bmwg] Workplan Proposal for Routing Protcool Testing
> >
> >
> > We are proposing that the BMWG accept the following 
> workplan proposal.
> >
> > The Routing Protcol workplan proposal intends to define
> > terminology and test
> > methodology for the testing of devices that implement 
> routing protocols.
> >
> > There are a number of drafts circulating that attempt to address
> > this issue,
> > but are either too narrowly focused
> > (draft-berkowitz-bgpcon-01.txt) or only
> > address data
> > forwarding issues and ignore control plane testing
> > (draft-ietf-bmwg-fib-term-01.txt).
> >
> > Our proposal attempts to address both issues as well as 
> address testing of
> > multiple protocols that have a core of similar behavior. By
> > defining metrics
> >
> > common to a group of protocols, we reduce the amount of effort
> > required to
> > define test methodologies for such protocols as a whole and allow
> > comparisons
> > between certain protocols when more than one implement 
> similar behavior
> > (e.g. ISIS and OSPF).
> >
> > These tests will address testing both the control plane and 
> data plane of
> > such devices but are designed such that a device that does 
> not implement
> > data forwarding can still be tested.
> >
> > The issues we intend to address are:
> >
> > 1.	Define terminology specific to the testing of Routing 
> Protocols in a
> >
> >     non protocol-specific manner
> >
> > 	First draft available as
> >
> > (http://www.ietf.org/internet-drafts/draft-ambrose-routing-protoco
> > l-term-00.
> > txt)
> >
> > 2.	Define a test methodology framework to allow testing of 
> any protocol
> >     that can be used to affect the data forwarding behavior 
> of a device
> >     (the device does not actually have to implement data
> > forwarding in order
> > 	for it to implement such protocols).
> >
> > 	These tests will address testing of both the control 
> plane and data
> > plane of a
> > 	device.
> >
> > 	A partial list of the proposed tests is:
> >     (this is subject to change as we progress through this 
> area of work)
> >
> >     - Peer Capacity
> >     - Routing Protocol Entry Capacity (RIB)
> >     - Routing Protocol Entry Capacity (FIB)
> >     - Peer and Routing Protocol Entry Capacity (RIB)
> >     - Peer and Routing Protocol Entry Capacity (FIB)
> >     - Performance
> >     - Peers accepted Rate
> >     - Routing Protocol Entry's accepted Rate (RIB)
> >     - Routing Protocol Entry's accepted Rate (FIB)
> >     - Forwarding Rate to stable Routes with Route flapping
> >     - Forwarding Rate while establishing new Peers
> >     - Initial Peer convergence (RIB)
> >     - Initial Peer convergence (FIB)
> >     - Convergence for new Routing Protocol Entry (RIB)
> >     - Convergence for new Routing Protocol Entry (FIB)
> >     - Convergence to a backup Routing Protocol Entry (FIB)
> >
> >
> > 3.	Define subsequent RFC documents that describe how to 
> application of
> > the framework
> >     to specific protocols
> >
> > The protocols we intend this work to apply to include but are not
> > necessarily limited to:
> > - OSPF
> > - ISIS
> > - RIP
> > - BGP (Internal and External)
> >
> >
> > We intend to publish the first methodology document before the
> > deadline for
> > the 51st IETF meeting in London and present this, along 
> with the existing
> > terminology draft at that meeting.
> >
> > We believe that this work will subsume and extend the 
> existing drafts and
> > will reduce the amount of work required to define test
> > methodology for such
> > protocols.
> >
> > Nick Ambrose
> > Principal Software Engineer, Ixia.
> > <mailto:[email-masked]>, http://www.ixiacom.com
> > (818) 444-2303
> >
> > Ixia...
> >      When the test really counts.
> >
> >
> > Nick Ambrose
> > Principal Software Engineer, Ixia.
> > <mailto:[email-masked]>, http://www.ixiacom.com
> > (818) 444-2303
> >
> > Ixia...
> >      When the test really counts.
> >
> >
> >
> > _______________________________________________
> > bmwg mailing list
> > [email-masked]
> > http://www.ietf.org/mailman/listinfo/bmwg
> >
> 

_______________________________________________
bmwg mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/bmwg

---

From: "David Newman" <[email-masked]>
Date: Tue, 19 Jun 2001 14:06:07 -0700

> elements

It will be interesting to see how you handle protocols that are
fundamentally different, such as RIP and OSPF. I believe you will find some
significant differences in the size of the table updates. :-)

(snip)

> without seeing a draft of the method doc which we intend to publish soon)

Well, yes, we do need to see a draft.

I can say from personal experience that the specific trumps the general.
When I first brought a firewall performance draft to the bmwg in 1997, the
intent was to cover not only firewalls but also other security devices like
IPSec VPN gateways. WG feedback suggested that it would be much better to
tackle one type of device at a time.

That was really good advice. The mechanics of firewalls are so different
than VPN gateways, intrusion detection systems, biometric devices, etc.,
that in retrospect I feel silly for even suggesting that multiple classes of
devices could be measured in the same way. The same is true with routing
protocols; they are *very* different.

> is all I see in your previous comments.

Eh? First of all, we have yet to see a draft and so, second, I couldn't and
didn't say it will never work.

Finally, I'm not asking anyone to trust anyone. Rather, I observed that
because we don't yet have tests for specific and very different routing
protocols, any attempt at a Grand Unified Theory of Routing Protocol Testing
strikes me as problematic.

> groups involved in the other drafts but have not yet met with any positive

Perhaps this is because those authors feel that defining specific tests for
specific protocols is appropriate.

I look forward to reading your draft.

dn