Subject: Re: [ietf-outcomes] [OPS-AREA] IETF Outcomes wiki

From: Dave CROCKER <[email-masked]>
Date: Sun, 07 Feb 2010 10:00:17 -0800

Dave,

Thoughtful posting.  Thanks!

Responding only to the larger issues you raise (but hoping others will also 
discuss the detail)...

On 2/7/2010 8:08 AM, David Harrington wrote:
> Hi,
>
> I think this results table an interesting idea, but I have some
> concerns.
>
> 1) My first concern is about the clarity of adoption ratings.
> SNMPv1 was obviously widely adopted.
> But a recent survey shared by Bert showed 14% using SNMPv1, 65% using
> SNMPv2, and 12% using SNMPv3. Yet SNMPv1 is listed as massive
> adoption, SNMPv2 is listed as some adoption, and SNMPv3 as complete
> failure.

The rating is like a 'best achieved' meter.  It pegs at the best value /ever/ 
achieved.  It's not designed to show reduced use over time.

Perhaps another column should be added, to distinguish between "best" and 
"current"?  (Although that creates a serious challenge, for keeping the 
'current' value, ummmmm... current.)

SNMPv1 was certainly massively successful... in its day.

> ipfix did not start out as an IETF design; I think an analysis of IPR
> disclosures would certainly reflect that; the ipfix WG just developed
> existing work into a standard.
>
> What criteria should be used for this binary decision?

Kernigan or Ritche defined "portable code" as needing less than 10% re-writing. 
  Perhaps some similarly pragmatic definition should be developed for 
determining whether something started "within" the IETF?

Since everything we define incorporates some details from outside, the right 
choice here is probably something along the lines of "the ietf assembled or 
invented all the pieces" vs. "the ietf made some changes to a pre-existing 
spec".  But I'm just guessing.

> 3) I have concerns about whether "adoption" is historical adoption or
> current adoption.
> At what point in time is adoption measured? Is netconf a complete
> failure because it has not been widely adopted yet? Was SNMPv1
> massively adopted in 1990? Maybe the chart should have adoption
> ratings at 2 years, 5 years, 10 years, and 15+ to get a better idea of
> adoption trends.

You are raising a question that is clearly reasonable.  I don't know what my own 
opinion about the particulars of the answer is.

However the one caution I will suggest is against trying to make this wiki 
assert much "precision".  The method for developing entries is highly 
subjective, and that means precision is extremely coarse.  It will be misleading 
to represent more precision about community rough consensus than is warranted.

There is a natural tendency to want finer-grained resolution to information, but 
we typically are not going to have assessment techniques that justify them, nor 
will we find a compelling community /need/ for the effort to develop them.

To be pedantic about the precisions limitations when reporting assessment:  for 
empirical efforts like this, the answer to 1.5 * 1.7 is not 2.55.  It's 2.6.

>       We certainly do not want to discourage
> people from adopting Netconf because so far it has not achieved
> massive adoption. Yet this chart could do that if the timeline of
> availability versus adoption isn't represented.

This is an interesting point:  To what extent can entries in this table 
influence the use of the things being listed?

I had assumed that an assessment should only be listed after the value to use 
(degree of success or failure) was quite clear to the community and that there 
therefore was/is consensus about it.  Other than the "0" value (still pending), 
of course.  I guess I could also imagine going from + to ++, or ++ to ++>, over 
time.

> 4) I have concerns about "adoption" when it relates to extensions
> versus the base technology. SNMPv2 and SNMPv3 build upon the base SNMP
> design. What has been adopted widely in SNMPv2? Is it the operations
> (getBulk), or is it SMIv2 extensions (Counter64)?
...
> SNMPv2 and SNMPv3 still use the base protocol features, such as GETs
> and SETs, traps and request/responses, and so on. That doesn't really
> get reflected in this table.

Not sure what sort of "reflected" you are looking for.

Please note that the table is attempting to report on the utility of particular 
a standards effort, not necessarily the functional relationship among different 
effort.

That is, the bald form of the question is:  Was the incremental effort to 
produce SNMPv2 worth the considerable effort?  Every standards effort is 
enormously expensive.  Try calculating the aggregate cost of participant time 
for even the smallest effort; it's intimidating.

(However Jon Callas commented to me that some efforts that are overtly a failure 
do provide valuable learning for later efforts that succeed; and indeed, 
capturing this is part of the goal in the "phases" idea, for efforts that have 
to change direction.)

> I notice Diameter is listed as a separate protocol from RADIUS, but
> isn't Diameter really RADIUS v2?

This gets back to the notation question I raised separately, for such things as 
phases.  It would be great if we could resolve this to cover such cases.

And this particular example can't reasonably be supported by the choice I had 
stated a preference for, which is a version number in column 1, since the basic 
name is different.

>  Diameter has been adopted by the
> telecom/service provider segment, but it has been largely rejected by
> the enterprise segment who prefer the simplicity of RADIUS. But
> adoption by some market segements and not others certainly doesn't get
> reflected in this table.

That's where the Target/Segment column helps.

> I would be interested in seeing the adoption trends for other
> protocols and later versions or their extensions, like DHCP and TLS
> and RADIUS and ipfix and IPv4/IPv6.

+1.

> 5) I am concerned about incomplete information, especially concerning
> area production.

The good news, is that you can directly remedy that deficiency....

> 6) I am concerned about statistics without analysis. The info
> presented has no analysis with it.

The right-most column is conveniently available for pointing to in-depth 
information...  Discussion of history, core issues, and the like, is not the 
goal of this wiki, but yes, it could be extremely helpful to develop that 
information.

> I am especially concerned because I see many efforts that try to build
> the pieces separately. For example, netconf was published two years
> ago, but the data modeling language is still in development, and there
> are no manageable data models available. Sip-clf is trying to build
> just a data model without considering how protocols will need to work
> to acheive the targeted use cases (and what effect that will have on
> the information and data models).

Is there a way to depict this sort of issue in a table?  Can the wiki be 
modified to support it?

> --Summary--
>
> I think this results chart is way too simplistic, and it can be very
> misleading, and it could lead to bad decisions about how we should
> develop protocols.

Your premise is that greater complexity will produce greater insight.

The cognitive reality is that there needs to be a balance between simplicity and 
complexity.  Go too far in either direction and utility is lost.

Where the balance needs to be largely depends on the details of the target 
population for consuming the wiki data.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "David Harrington" <[email-masked]>
Date: Tue, 9 Feb 2010 10:53:11 -0500

Hi,

I think it would be good to set success and failure to match rfc 5218,
and have the legends page provide brief descriptions of the
categories, and reference RFC 5218 for more details.

dbh

> -----Original Message-----
> From: Bernard Aboba [mailto:[email-masked]] 
> Sent: Tuesday, February 09, 2010 1:47 AM
> To: 'Joel Jaeggli'; 'Bert Wijnen (IETF)'
> Cc: 'David Harrington'; 'Romascanu, Dan (Dan)'; 'ops-area 
> (IETF)'; [email-masked]
> Subject: RE: [OPS-AREA] IETF Outcomes wiki
> 
> I'd argue that the wiki is not using the same definition of 
> "success" that we used in RFC 5218. 
> 
> If the data shows that 85 percent of the original 
> implementers of SNMPv1 went on to implement
> SNMPv2 at some point, then if SNMPv1 was a success under the 
> RFC 5218 criteria, then SNMPv2
> would also have to be judged a success. 
> 
> In fact, using the RFC 5218 criteria, SNMPv1/v2 might also be 
> judged as a "wild success"
> in that it was used more widely and for things beyond what 
> the creators envisaged. 
> 
> Note that the RFC 5218 criteria explicitly include the 
> original scope in the definition of
> "success".  For example, if a protocol was deployed for its 
> original purpose in the originally
> envisaged use, that is success. 
> 
> So a protocol such as Kerberos can be judged a "success" even 
> though it is not used in
> consumer scenarios in the same manner as say, SSL/TLS.  
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Joel Jaeggli [mailto:[email-masked]] 
> Sent: Monday, February 08, 2010 9:48 PM
> To: Bert Wijnen (IETF)
> Cc: David Harrington; 'Romascanu, Dan (Dan)'; ops-area 
> (IETF); [email-masked]
> Subject: Re: [OPS-AREA] IETF Outcomes wiki
> 
> Bert Wijnen (IETF) wrote:
> > 
> > I hope (assume/assumed) that you understand that I 
> certainly also do not
> > agree with
> > what is currently listed on the OPS outcomes page.
> >  
> > It is just that if you read that page.... one would have to 
> come to that
> > conclusion.
> 
> One might also conclude that appart from some rather obvious 
> successes,
> that attempting to determine if something is a success or failure as
> part of the outcomes exercise is a fools errand. Although as with
many
> IETF exercises if enough fools work hard enough you might luck out
and
> end up with something useful.
> 
> > Bert
> 
>

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 09 Feb 2010 08:21:15 -0800

Dave,

> categories, and reference RFC 5218 for more details.

The easy question is:  why?

The definition in RFC 5218 is:

      "we consider a successful protocol to be one that both meets its original
      goals and is widely deployed."

This means that a protocol is a failure if it is widely used, but for different 
purposes than it was intended.

"Deployed" is also a problem, since there is a long track record of industry's 
having deployed something but never actually using it very much.  I submit all 
of OSI as a prime example.

      These are the reasons the wiki was premised on the simple measure of use.

However I note that the column that lists degree of success only uses the word 
'adoption'. However Target Segment uses 'use'.

d/

ps.  I also note you've been busy updating net management-related entries.  Thanks!

-- 

   Dave Crocker

---

From: "David Harrington" <[email-masked]>
Date: Tue, 9 Feb 2010 13:34:24 -0500

Hi,

Because it would be good to "standardize" our measurements of success
and failure of IETF protocols.
RFC5218 was written, presumably, to help developers better understand
what makes a protocol successful.
The results wiki is, presumably, for a similar purpose.
It would be nice if the IETF reached consensus on some definitions for
success and failure before we start advertising our successes and
failures in a public wiki.
To represent consensus, maybe both the wiki and the definitions in
RFC5218 need to be modified.

I support the wiki, but would like the methodology to be documented so
different people can edit it in a consistent manner, so readers can
interpret it with some consistency. It has too much opportunity for
unintended consequences to do it wrong. 

One problem with a wiki is that it can represent the personal opinion
of the last person to edit it, rather than to represent the consensus
of the IETF. But it is posted on the official IETF web site, and can
easily be mistaken to represent an official statement of consensus.

dbh

> From: Dave CROCKER [mailto:[email-masked]] 
> Sent: Tuesday, February 09, 2010 11:21 AM
> To: David Harrington
> Cc: 'ops-area (IETF)'; [email-masked]
> Subject: Re: [ietf-outcomes] [OPS-AREA] IETF Outcomes wiki
> 
> Dave,
>

---

From: "Bernard Aboba" <[email-masked]>
Date: Tue, 9 Feb 2010 12:28:04 -0800

Dave Crocker said:

"This means that a protocol is a failure if it is widely used, but for
different 
purposes than it was intended."

[BA] If a protocol has been implemented, deployed and used, I don't think
that 
can qualify as a failure under the RFC 5218 definition. From 
Section 1.4:

"   Failure, or the lack of success, cannot be determined before allowing
   sufficient time to pass (e.g., 5-10 years for an average protocol).
   Failure criteria include:

   o  No mainstream implementations.  There is little or no support in
      hosts, routers, or other classes of relevant devices.

   o  No deployment.  Devices that support the protocol are not
      deployed, or if they are, then the protocol is not enabled.

   o  No use.  While the protocol may be deployed, there are no
      applications or scenarios that actually use the protocol."

Dave Crocker also said:

"Deployed" is also a problem, since there is a long track record of
industry's 
having deployed something but never actually using it very much.  I submit
all 
of OSI as a prime example.

However I note that the column that lists degree of success only uses the
word 
'adoption'. However Target Segment uses 'use'."

[BA] I'd suggest that implementation, deployment and use are separate
stages,
worth documenting separately.  If that is done, we may learn 
something about the characteristics that affect success at each stage. 

One aspect that bothers me about use is the definition of each category. 

Surely if almost nobody uses something, it is a "--" failure (e.g. IPsec
AH). 
But what if the protocol is used by the people it is designed for, but not
the vast majority of humanity? 

I use Kerberos all the time, but most people probably don't.  What category
does
this fall in?  Probably not "++" or "++>", but not "--" either.  So we're
left
with "-" and "+".  How are these categories defined?

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 10 Feb 2010 11:45:25 +0100

The wiki page includes a line that says: 

'The wiki is only as complete and accurate as community effort makes
it.' 

It would probably be good to add a phrase that makes clear that the
information contained here does not represent the consensus of the IETF.

Dan

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of David Harrington
> Sent: Tuesday, February 09, 2010 8:34 PM
> To: [email-masked]
> Cc: [email-masked]
> Because it would be good to "standardize" our measurements of 
> success and failure of IETF protocols.
> RFC5218 was written, presumably, to help developers better 
> understand what makes a protocol successful.
> It would be nice if the IETF reached consensus on some 
> definitions for success and failure before we start 
> advertising our successes and failures in a public wiki.
> I support the wiki, but would like the methodology to be 
> documented so different people can edit it in a consistent 
> manner, so readers can interpret it with some consistency. It 
> has too much opportunity for unintended consequences to do it wrong. 
> 
> One problem with a wiki is that it can represent the personal 
> opinion of the last person to edit it, rather than to 
> represent the consensus of the IETF. But it is posted on the 
> official IETF web site, and can easily be mistaken to 
> represent an official statement of consensus.
> 
> dbh
> 
> >       "we consider a successful protocol to be one that 
> both meets its 
> > original
> > This means that a protocol is a failure if it is widely 
> used, but for 
> > different purposes than it was intended.
> > 
> > industry's having deployed something but never actually 
> using it very 
> > much.  I submit all of OSI as a prime example.
> > 
> > However I note that the column that lists degree of success 
> only uses 
> > the word 'adoption'. However Target Segment uses 'use'.
> > 
> > d/
> > 
> _______________________________________________
> ietf-outcomes mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-outcomes
>

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 10 Feb 2010 12:01:52 -0800

On 2/10/2010 2:45 AM, Romascanu, Dan (Dan) wrote:
> 'The wiki is only as complete and accurate as community effort makes it.'
>

I'll offer a friendly amendment, in the hopes of achieving some... ummm... 
consensus...

Outcomes home page
------------------

Currently:
> The IETF has operated in its current form since 1989. Some of its standards
> have succeeded on a very large and visible scale. Others are successful, but
> visible only to a more limited community. Others have fared poorly. This
> wiki lists technologies and services that were developed in the IETF and
> represent notable successes or failures. It is a collaborative effort of
> IETF participants, to provide feedback to the community about the utility of
> IETF efforts, as well as to facilitate public understanding of IETF work and
> impact.

Proposed revision:

The IETF has operated in its current form since 1989. Some of its standards have
succeeded on a very large and visible scale. Others are successful, but visible
only to a more limited community. Others have fared poorly. This wiki lists
technologies and services that were developed in the IETF and represent notable
successes or failures. It is an informal, collaborative effort of IETF
participants, to provide feedback to the community about the utility of IETF
efforts, as well as to facilitate public understanding of IETF work and impact.
The contents of the are revised according to the views of individual
contributors and have not been subject to IETF Rough Consensus assessment.

Charter page
------------

Currently:

> The IETF produces technical specifications, with the goal of their being used
> on the Internet. Once a specification is published, there is no formal
> mechanism for assessing the eventual utility of the work. However feedback
> about successes or problems can be useful in developing a shared community
> sense of the way work is -- or should be -- done. The IETF Outcomes Wiki is a
> collaborative effort to produce subjective assessments about successes and
> failures of notable efforts, based on rough consensus.

Proposed revision:

The IETF produces technical specifications, with the goal of their being used on 
the Internet. Once a specification is published, there is no formal mechanism 
for assessing the eventual utility of the work. However feedback about successes 
or problems can be useful in developing a shared community sense of the way work 
is -- or should be -- done. The IETF Outcomes Wiki is
an informal, collaborative, incremental effort among individual IETF 
participants, to produce subjective assessments about successes and failures of 
notable efforts.  Although tending to reflect a degree of community agreement, 
the contents of the wiki are not subject to formal IETF Rough Consensus assessment.

-- 

   Dave Crocker

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 10 Feb 2010 21:07:43 +0100

In line. 

Regards,

Dan

> [mailto:[email-masked]] On Behalf Of Dave CROCKER
> Sent: Wednesday, February 10, 2010 10:02 PM
> To: [email-masked]
> > The IETF has operated in its current form since 1989. Some of its 
> > standards have succeeded on a very large and visible scale. 
> Others are 
> > successful, but visible only to a more limited community. 
> Others have 
> > fared poorly. This wiki lists technologies and services that were 
> > developed in the IETF and represent notable successes or 
> failures. It 
> > is a collaborative effort of IETF participants, to provide 
> feedback to 
> > the community about the utility of IETF efforts, as well as to 
> > facilitate public understanding of IETF work and impact.
> 
> 
> Proposed revision:
> 
> The IETF has operated in its current form since 1989. Some of 
> its standards have succeeded on a very large and visible 
> scale. Others are successful, but visible only to a more 
> limited community. Others have fared poorly. This wiki lists 
> technologies and services that were developed in the IETF and 
> represent notable successes or failures. It is an informal, 
> collaborative effort of IETF participants, to provide 
> feedback to the community about the utility of IETF efforts, 
> as well as to facilitate public understanding of IETF work and impact.
> The contents of the are revised according to the views of 
> individual contributors and have not been subject to IETF 
> Rough Consensus assessment.

Looks OK but the phrase is missing something, maybe the word 'wiki'
between 'the' and 'are'. 

> 
> 
> 
> Charter page
> ------------
> 
> Currently:
> 
> > The IETF produces technical specifications, with the goal of their 
> > being used on the Internet. Once a specification is 
> published, there 
> > is no formal mechanism for assessing the eventual utility 
> of the work. 
> > However feedback about successes or problems can be useful in 
> > developing a shared community sense of the way work is -- 
> or should be 
> > -- done. The IETF Outcomes Wiki is a collaborative effort 
> to produce 
> > subjective assessments about successes and failures of 
> notable efforts, based on rough consensus.
> 
> 
> Proposed revision:
> 
> The IETF produces technical specifications, with the goal of 
> their being used on the Internet. Once a specification is 
> published, there is no formal mechanism for assessing the 
> eventual utility of the work. However feedback about 
> successes or problems can be useful in developing a shared 
> community sense of the way work is -- or should be -- done. 
> The IETF Outcomes Wiki is an informal, collaborative, 
> incremental effort among individual IETF participants, to 
> produce subjective assessments about successes and failures 
> of notable efforts.  Although tending to reflect a degree of 
> community agreement, the contents of the wiki are not subject 
> to formal IETF Rough Consensus assessment.

This looks OK to me.

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 11 Feb 2010 08:34:08 -0800

I've made these two changes to the wiki.

(Thanks for catching the typo.)

d/

-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 11 Feb 2010 09:31:25 -0800

(opsarea mailing list dropped.)

David and Bernard, et al,

On 2/9/2010 12:28 PM, Bernard Aboba wrote:
> different purposes than it was intended."
>
> that can qualify as a failure under the RFC 5218 definition. From Section
> 1.4:

You quote Section 1.4 and I quoted Section 1.2.  They appear to be conflicting 
definitions, and the conflict doesn't appear to be subtle.  Both bits of text 
are clearly written and easily understood.

So while it well might be good for us to have a single definition of 'success' 
that is universally applied, simply referring to the RFC does not seem to get us 
there.

> industry's having deployed something but never actually using it very much.
> I submit all of OSI as a prime example.

I'll repeat that I think we should have a trivial rule, and that I think it 
should simply be a measure of 'benefit'.  If people are using something, it has 
benefit.  If they have deployed it but are not using it, there is no benefit.

We need to be careful that we do not define an array of subtle distinctions that 
wind up adding to our effort but don't make major impact on the utility of the 
assessment exercise.

So, for example, the difference between deployment and use is not subtle, but 
just how much do we learn by adding the distinction to the core assessment 
effort?  Adding pieces to an assessment effort increases the complexity and the 
potential for disagreement.

Contrast this with:  The community agrees that a protocol is not used or is not 
used very much.  (A - or -- assessment.)  After we agree on that, folk ask what 
happened and discover that it was deployed but difficult to use.  That's worth 
knowing, but only /after agreeing that it wasn't used/.

In other words, the distinction between Used and Deployed is interesting as a 
derivative measure in the case of non-use, and is not interesting in the case of 
use.  That means it should not be part of the core assessment effort, which 
makes that effort simpler.

 > easily be mistaken to represent an official statement of consensus.

There are a few layers to dealing with this concern.  I fully expect that we 
will, indeed, eventually find conflict about an entry to be a serious problem. 
When we do, we'll have to proceed thoughtfully.

Stepping through the layers, here's my own, starting view:

The wiki software used by the IETF does not provide the wiki equivalent of email 
list 'moderation', where an authority can mandate certain controls on the text. 
  So the underlying mechanics of enforcing resolution of a conflict about an 
entry is going to have to rely on social conventions, rather than software 
mechanics.  If that becomes insufficient, we'll have to figure out some way to 
deal with it.

Although I did not write anything into the initial charter -- because I think 
there should be discussion and agreement before adding a statement about it -- I 
have assumed that the process for resolving conflicts about an entry should be 
debate and rough consensus on this mailing list.  I'm tasked with facilitating 
and assessing.

I'm a believer in overt steps to resolve debate, including on-list efforts to 
summarize and suggest compromises, and finally to make explicit, on-list 
statements about assessment of rough consensus.  Sometimes that can be reduced 
to a one-step "default yes", such as the change I just installed onto the wiki 
homepage and charter:  Bernard and I had a public exchange and quickly reached 
an agreement.  I waited a day and no one else chimed in.  Absent objections, I 
declared that I'd make the change, and then did.  To the extent that anyone 
comes in afterward and cares about this, they can see the sequence.  If they 
object and the group comes to consensus about wanting to reverse or modify the 
change, wiki's make this easy.

If there is a disagreement about my handling of any wiki or mailing list issues, 
I'm assuming we can invoke the usual appeals process, through the IESG. 
(Authorization for establishing the wiki on the IETF site came from the IETF 
Chair, so I assume that's the reporting structure for on-going issues.)

It is certain that there are issues about changing the wiki's format or contents 
that will run into a problem that requires new or different procedures.  As 
these come up, we will figure them out, as long as the community stays engaged 
in the effort.

d/
-- 

   Dave Crocker