Subject: [ietf-outcomes] "Instances" column

From: Dave CROCKER <[email-masked]>
Date: Sat, 13 Feb 2010 14:04:04 -0800

Folks,

Ray is a busy guy today.  He looked at the Outcomes pages some more and went the 
next step in thinking of a way to make it easier for average, non-geek/non-IETF 
folk to relate to particular pieces of work, by asking whether it would be 
possible to indicate products implement the work.

I did a version of this for DKIM

    <http://dkim.org/deploy/index.html>

but it is manually maintained and was for the purpose of /promoting/ DKIM 
adoption, rather than /demonstrating/ it.  At the least, it has more detail than 
I would suggest for the Outcomes page.

I do think that some amount of pointing to implementations could be helpful, 
since it provide concrete demonstration of availability (if not actual use.) 
However, it needs to be self-administered, carry no implications of quality or 
conformance, and no marketing.

I've created an example of this on the Security page

    <http://trac.tools.ietf.org/misc/outcomes/wiki/IetfSecurity>

and would appreciate comments.

It has an example of pointing to a list that is maintained outside (the DKIM 
page) and one that is part of the wiki (DNSSec).  Since I don't know anything 
about DNSSec, I've only created the skeleton for the sub-page; but I am guessing 
that there are folk who are highly motivated to tout DNSSec support...)

OK.  Start shooting...

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "David Harrington" <[email-masked]>
Date: Sun, 14 Feb 2010 13:21:40 -0500

Hi Dave,

I would like to see more consistency with existing IETF terminology.

RFC2026 defines three levels of standard.
Proposed requires an agreed upon document.
Draft requires implementations.
Standard requires operational experience.

RFC2026 doesn't talk about instances and usage, it talks about
implementations and operational experience.
Your "instances" column is about implementations, but uses different
terminology than is already in use in the IETF.
Your "adoption" column is about operational experience, but uses
different terminology than IETF process.
I would like to see the results wiki use terminology consistent with
our standards process.

I think using terminology consistent with our process would make these
adoption categories much more intuitive to IETF participants, AND to
those who follow the IETF, such as journalists. I think even to an
IETF newbie, the engineering distinction between implementation and
operational experience would be clear; I do not think the distinction
between instances and adoption is very clear.

It would also be nice if somewhere in the "implementation/instances"
page hierarchy, we had pointers to BCP 9 (RFC 5657) "Guidance on
Interoperation and Implementation Reports for Advancement to Draft
Standard", and to specific implementation reports found at
http://www.ietf.org/iesg/implementation-report.html. This might help
encourage implementers to file actual implementation reports.

dbh

> -----Original Message-----
> From: Dave CROCKER [mailto:[email-masked]] 
> Sent: Sunday, February 14, 2010 10:37 AM
> To: David Harrington
> Subject: Re: [ietf-outcomes] "Instances" column
> 
> David,
> 
> On 2/13/2010 10:16 PM, David Harrington wrote:
> > Hi,
> >
> > I think there should be one column for implementations and 
> another for
> > adoption (i.e. usage).
> 
> Hmmm.  Isn't that essentially what is accomplished by having 
> the current 
> Adoption (use) column and the new Instances column?
> 
> What would you like to see that is different (but still 
> simple)... and why?
> 
> 
> > You already set up the ability to have a link to an additional
page
> > via the comments column.
> 
> Yeah.  I thought about that when Ray raised the issue, but 
> decided that a 
> Comments column pointer was generic; it could be for 
> anything.  While the 
> Instances column would be entirely specific. The premise 
> justifying the new 
> column  is that that specific information is worth implicitly 
> asking for.
> 
> 
> >   I recommend making it possible to have an
> > implementation column with a legend (++ through --) and an
optional
> > link to a page with a list of implementations (and possibly 
> links from
> > that page to implementation reports).
> 
> There is a particular benefit in being able to read through a 
> list of names and 
> pointers to their implementations. It gives serious substance 
> to a claim of success.
> 
> What is the benefit in 'rating' the degree of implementation 
> success, as 
> separate from Use?  What question is it answering that 
> affects IETF work, and 
> for whom?
> 
> 
> > The same could be done for the usage column - allow an iptional
link
> > to a page listing known usages, with possible deployemnt 
> reports (and
> > if usage is what we are measuring in the adoption column, then the
> > column should probably be called usage).
> 
> "Reports" for such things as deployment moves this entire 
> activity into a world 
> of formal accounting that is common among some standards 
> efforts, but relatively 
> foreign to the IETF.  It's the sort of thing entailing 
> significant effort, for 
> little apparent benefit.  But feel free to enlighten me...
> 
> As for changing adoption to usage, i suspect you are right.  
> Want to make that 
> suggestion to the list?  (I'm posting too much there, as it 
> is, and really want 
> others to take initiative, as you have been.)
> 
> d/
> -- 
> 
>    Dave Crocker

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 14 Feb 2010 13:46:10 -0500

David,
     The purpose of this is something simple, and easily readable.
     Factors like the maturity of the documents on the standards track 
are basically outside the scope of what we are trying to do here.  If we 
can not figure out a simple way to look at things and saying clear 
things about whether the work does or has mattered to a significant 
community, or whether this work is still developing, then I think we are 
just confusing ourselves.
     Note that the point is not to argue about whether one work effort 
is better than another, or even whether it is more used than another.
     To try to capture subtle distinctions in this would complicate 
things without adding significant value.  This is, as I read things, 
intended as a broad brush, subjective effort to capture things where we 
can say whether it is a success, a failure, or still still growing.  I 
would not be surprised if relatively new efforts don't even fit in the 
chart. (There is no point in listing the ForCES effort yet.)

Yours,
Joel

David Harrington wrote:
> _______________________________________________
> ietf-outcomes mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-outcomes
>

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 14 Feb 2010 10:55:07 -0800

David,

> Standard requires operational experience.

Well, that's certainly a worthy discussion and it would be great to get the list 
engaged on it.

And indeed, I'd guess that ++ and Full Standard ought to correlate highly.

There is an underlying question that I'd suggest you post an answer to:

       Why should the two be linked?

       What benefit derives?

       On what basis would it make sense for them /not/ to be linked?

For example, the standards label is a formal, political process.  It's rare and 
difficult to get to Full.  (The YAM wg has embarked on a test of this, and the 
experience so far is not pretty.)

The wiki is intended as an extremely informal and flexible process.

The difference between these is fundamental.

> implementations and operational experience.

For Draft it is only required to talk about /two/ implementations.

Full Standard pretty much /is/ in terms of usage.

> terminology than is already in use in the IETF.

If your concern is 'instance' vs. 'implementation', that's to keep the column 
narrower.  And no, I'm not kidding.  If there is a real indication of community 
misunderstanding, then the disparity is important.  However the idea that the 
community cannot tolerate synonyms -- especially given how loose people tend to 
be with their vocabulary -- should be documented.

If your concern is something else, please elucidate.

> different terminology than IETF process.

What do you believe is established terminology that achieves the same purpose as 
the Adoption column is trying to serve?

> I would like to see the results wiki use terminology consistent with

That's an inclination that has intuitive appeal.

However we need to be careful that it does not produce slavish adherence that 
undermines actual utility.  Since the standards process and the wiki are not 
identical instruments, it's entirely possible they need disparate terminology.

I'm not saying I'm sure they do, but merely that the decision needs to be made 
carefully?

> those who follow the IETF, such as journalists.

Again, that has intuitive appeal.  Note, however, that essentially everyone but 
serious, longer-term IETF geeks have no feel for the standards levels at all. 
So I am not sure I see how the link makes anything intuitive for the larger 
community.  (FWIW, I'm not happy about that observation, but I think it's 
factually correct.)

> I think even to an

You still have not answered the underlying question I've asked a couple of times:

    Implementation is interesting only for standards that have not been very 
successfull.  As such it is a secondary measure.

    Why elevate it to primary?

> encourage implementers to file actual implementation reports.

David, you keep wanting to turn the wiki into an extension of formal processes, 
when it is carefully designed to be quite different from it.

We have extensive community assessment that the formal process is problematic in 
effort and vagaries.  The wiki is designed to be informal, simple and direct. 
It asks only one question:  Did an expensive community effort succeed?

For each change you suggest, please think hard about the value proposition of 
the change and look for ways that an otherwise-reasonable value proposition 
might actually /not/ be true.

d/
-- 

   Dave Crocker

---

From: "David Harrington" <[email-masked]>
Date: Sun, 14 Feb 2010 20:45:49 -0500

Hi, 

> From: Dave CROCKER [mailto:[email-masked]] 
> Sent: Sunday, February 14, 2010 1:55 PM
> To: David Harrington
> Cc: [email-masked]

Maybe ought to, but often do not.
SNMPv3 is a full standard, but doesn't achieve ++ usage.
The IETF decided years ago not to bother taking MIB modules to draft
and full standards because most vendors implement the Proposed
Standard, if it is applicable to their products. A number of MIB
modules might achieve ++
usage but remain at PS. Other MIB modules also are at PS, but achieve
no usage.

So I recognize that the formal and informal adoption levels can be
very different.
I am not suggesting that we want the wiki to reflect the formal
status.
I think it can be important to understand that formal status and
informal status can differ markedly.
Maybe we should understand where and why, so we can change the formal
process to better match reality.
(That work, of course, has been underway I think since 2026 was
published ;-)

>        Why should the two be linked?

Most comments to this list have been about the confusing terminology
related to adoption.
If we were only informally measuring adoption, then I would not argue
that we should use existing IETF terminology.
But when you add instances, you are now talking informally about both
implementation and operational experience.
If we are going to have both of these metrics in the wiki, which
happen to be roughly the same metrics we use for standards, then I
think we should use consistent terminology. I think the adoption
levels are confusing enough, but if you are now going to discuss
instances as well as adoption levels, you seem to be adding
implementation information. And I think that confuses what adoption
means - SNMPv3 was (initially) widely implemented in devices, but not
in managers, so it didn't get used. So how do you measure adoption? Do
many instances imply ++ adoption?

That's the big issue I have. I think the terminology being used is
confusing to interpret because there are mixed metrics, and adding
'instances' seems to confuse it even more.

If you think of the formal process as "two interoperable
implementations" and "scalability" rather than just "implementations"
and "usage", then I agree the two are not the same and should not be
linked. 

>        What benefit derives?

The main benefit I see is that engineers understand the distinction
between implementation and usage, and the overwhelming majority of the
readers of this wiki will have an engineering bent. 
I think instances and adoption lacks clarity.

> the Adoption column is trying to serve?

operational experience.
I'm fine with usage.

> community.  (FWIW, I'm not happy about that observation, but 

I did not say we should use terms like Proposed, Draft and Full
Standard which most people who haven't been in the IETF for a long
time would not necessarily understand. But I think engineers would
understand the distinction between implementation and operational
experience (or usage) better then between instances and adoption.

> successfull.  As such it is a secondary measure.

But isn't instances a pointer to implementations?
If not, then I guess I really don't know what 'instances' means or
what having it here gives me.
I think instances and implementations are synonymous, although
implementations is more specific (and thus narrower).
Please explain how instances is narrower than implementations.

As you observed, a listing of 'instances' tends to promote the
relevant standard, presumably to improve usage.
Does that mean that listing instances is interesting only for
standards that have not been very successful?
Or are we listing instances to help market products?
I'm not sure of the purpose of 'instances'.

>     Why elevate it to primary?

I am suggesting elevating it to an even level with usage to help
better understand the curve of adoption. Implementation necessarily
precedes usage.

There might well be interesting correlation of timelines of
implementation versus usage.
BEEP doesn't get used, largely because there are very few
implementations. It is reputedly expensive to implement.
It has implementation issues. It's not used because it's not
available.
SNMPv3 hasn't gotten ++ usage, even though there are ++
implementations (on the agent side).
As somebody mentioned in this wiki discussion, SNMPv3 didn't get used
because configuring it is too hard. It's not that the implementations
aren't there; it has usage issues.
There are multiple successful standards that have both properties -
available implementations and easy to use.

I think it important to understand both implementation adoption (i.e
availability from vendors, which might include open source developers)
and usage adoption (by end-users) of a protocol. If the purpose of the
wiki is to allow us to mine the data for what makes a protocol a
notable success or failure, I think the availability and usage metrics
are far more relevant to success/failure than the number of years it
took to produce, or (in most cases) the year in which it was first
published.

> might actually /not/ be true.

Well, part of the problem is that I do not really understand the
purpose of the wiki if there is an instances column.
Is it still about general rough perceptions of adoption, and instances
imply some level of adoption?
or is it about marketing, which the IETF typically stays away from?
I know Ray asked for instances. Why does he want that? Is he going to
buy an instance of the standard? 
Is it to prove that adoption is somehow reflected in the number of
products available? 
Is it to reflect the informal "maturity" of the standard in the
industry?

I think the otherwise-reasonable value proposition of the instances
column might actually not be very clear.
I think 'instances' can add to the confusion of the metrics used to
informally measure success and failure.
I think columns to represent implementations (vendor adoption) and
usage (user adoption) would be more useful than instances and
adoption.

---

From: "David Harrington" <[email-masked]>
Date: Sun, 14 Feb 2010 20:55:43 -0500

Hi, 

> From: Joel M. Halpern [mailto:[email-masked]] 
> Sent: Sunday, February 14, 2010 1:46 PM
> To: David Harrington
> Cc: 'Dave CROCKER'; [email-masked]

I never said anything about including information about the documents
on the standards track.

> is better than another, or even whether it is more used than
another.

And I never thought it was, and still don't.

> can say whether it is a success, a failure, or still still 
> growing.  

totally understood.

> I 

that's why we have a pending status.

See my other email about why I think implementation (availability or
vendor adoption) and usage (end-user adoption) should both be
reflected in this wiki.

dbh

---

From: Ray Pelletier <[email-masked]>
Date: Sun, 14 Feb 2010 22:01:59 -0500

On Feb 14, 2010, at 8:45 PM, David Harrington wrote:

<snip>
> Is it to reflect the informal "maturity" of the standard in the

When I approached Dave about adding a 'Utilization' column I said:
So Adoption might say ++, but it naturally follows to ask - where? What
products?  Like XMPP for Jabber. SIP for ... People can relate to  
that. It's a
demonstration of the relevancy of the IETF in people's everyday lives.

I thought it would be easier for regular people, the media and  
potential sponsors
to understand the enormous contribution the IETF has made towards the  
success
of the Internet, and the importance of the continuation of that effort.

You know better than they how this work has been translated into  
products and services
that people and companies use every day.

Ray