Subject: [Mtgvenue] Exploration of a "posting" metric - draft-elkins-mtgvenue-participation-metrics

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 20 Jul 2016 09:37:08 +0000

In your draft and in your talk, you mention the possibility of measuring a person's participation or contribution in terms of email postings. You also mention a number of other possible models, such as whether their name winds up be "acknowledged" or listed as a "contributor" in a draft or RFC.

I took a crack at extracting information about email postings from the mail archive. You can find a very preliminary result at https://www.dropbox.com/sh/2t1d3m96fi4brbf/AACqZ8-fQ08bbtFwJA9ZQczQa?dl=0

The point of that was to identify low-hanging fruit. We can and do measure a person's registration for or attendance at face-to-face meetings. We can also determine whether their email address shows up in an internet draft (I find the email address easier to think about than their name, as names can have several formats, wrap around line breaks, and so on). In this prototype, I rsync'd the mailman archive, and for each list and each month's box capture, parsed the file to find "From:" lines from emails.

The output is in the CSV file in that dropbox link. There's nothing magic in there - we could sort things in a different way or etc easily enough. What I tracked in this version was a sender's email address, the lists they posted to, for each such list the total number of messages they sent, and then per month how many messages they sent in that month.

My thought in that is that it's pretty easy to associate an email address with a mailing list, and determine an individual's transmission rate. If one knew that a given person used a given email address, one could determine a person's transmission rate on a monthly basis. What isn't readily found is anything about topic, whether the contribution is positive or negative, useful or otherwise, etc. Semantic analysis, and perhaps the distinction between "participation" and "contribution", is not, from my perspective, "low-hanging fruit".

I'm wondering here what you want in terms of a participation metric.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 20 Jul 2016 11:48:30 +0200

On 7/20/2016 11:37 AM, Fred Baker (fred) wrote:
> I took a crack at extracting information about email postings from the mail archive.
...
> The point of that was to identify low-hanging fruit.

The problem is that quite a lot of that fruit is underdeveloped, or rotten.

The S/N ration for postings is sufficiently poor as to render measuring 
posting possibly only slightly better than the 'just showing up' metric 
of registering for a meeting.  Simply put, just counting whether a 
person posts to the lists tells us nothing about whether they've 
actually "contributed" to the substance of the work.

"Contribution" needs to be measured meaningfully, as well as 
objectively, even if the measurement is coarse and doesn't count everyone.

This is why I've suggested reaching no deeper than the Acknowledgements 
and Contributors sections of I-D's and RFCs (as well as author lists, of 
course.)  If a name shows up there, the person made it through a basic 
filter for having been seen to actually contribute.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 20 Jul 2016 18:08:48 +0000

> On Jul 20, 2016, at 11:48 AM, Dave Crocker <[email-masked]> wrote:
> 
> "Contribution" needs to be measured meaningfully, as well as objectively, even if the measurement is coarse and doesn't count everyone.
> 
> This is why I've suggested reaching no deeper than the Acknowledgements and Contributors sections of I-D's and RFCs (as well as author lists, of course.)  If a name shows up there, the person made it through a basic filter for having been seen to actually contribute.

I'm thinking through the mechanisms by which one might extract the acknowledgements, contributors, and authors sections of an internet draft or RFC, and extract from it human names. If you have some good heuristics, I'd be interested. I don't think "hire an intern" is a long term solution.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 20 Jul 2016 20:11:49 +0200

On 7/20/2016 8:08 PM, Fred Baker (fred) wrote:
> If you have some good heuristics, I'd be interested. I don't think "hire an intern" is a long term solution.

Well, it gets easier when the input is an xml source, but for classic 
ascii source, other than obvious header-searching algorithms, no I 
haven't a clue.

A simple algorithm would likely be acceptable at finding the right 
document sections, but would probably parse the contents into many 
possible names that aren't really names,  I suspect that correlating 
these candidate with various signup lists (ietf mailing lists, ietf 
registrations, whatever) could get pretty close.  mumble.

d/

-- 

   Dave Crocker

---

From: <[email-masked]>
Date: Thu, 21 Jul 2016 04:44:43 +0000 (UTC)

On 7/20/2016 11:37 AM, Fred Baker (fred) wrote:
>> In your draft and in your talk, you mention the possibility of measuring a person's participation or contribution in terms of email postings. >>You also mention a number of other possible models, such as whether their name winds up be "acknowledged" or listed as a "contributor" >>in a draft or RFC.
>>
...
>> The point of that was to identify low-hanging fruit.

Fred, sorry too much going on.  I have not had a chance to look at the metrics yet.  

>actually "contributed" to the substance of the work.

Dave, respectfully, I could not disagree more.  I think that email lists and the way we participate on them works.  By and large, if someone contributes and continues to contribute regularly on email lists (esp. the WG lists) and generates discussion, then that is definitely contribution.  
IMHO, people need to start by reviewing drafts - that is what I tell newcomers in the mentoring program, and reviews by other people posted on the email lists for everyone to see are one of the things that actually make the IETF drafts much better.   An insightful comment leads to discussion, etc.
We may need to have some nuance in how we measure contribution on email lists.  But, I stand by my opinion that participation on email lists is a metric that is a valuable measure of contribution because it is in fact, a valuable contribution.

>filter for having been seen to actually contribute.

Definitely agree that Acknowledgements and Contributors ALSO be counted.
Nalini

---

From: Dave Crocker <[email-masked]>
Date: Thu, 21 Jul 2016 06:53:06 +0200

On 7/21/2016 6:44 AM, [email-masked] wrote:
> Dave, respectfully, I could not disagree more.  I think that email lists
> and the way we participate on them works.

I didn't say they didn't.

The question is how to distinguish folks who post and their postings 
constitute real contributions, versus folk who post and don't contribute 
much or anything to the substance of the work.

That's a subjective call and there is no way to make the distinction by 
just scanning mailing list text, short of an AI engine that will have a 
very poor validity performance.  The acknowledgements/contributors 
sections of documents constitute the document author's performance of 
exactly that filtering function.  So let's use it.

>          By and large, if someone
> contributes and continues to contribute regularly on email lists (esp.
> the WG lists) and generates discussion, then that is definitely
> contribution.

Sorry, no it isn't.  Not in theory and not in practice.  And this is 
exactly the difference between activity vs. contribution.

It is quite common to have posters who contribute little or nothing, or 
even detract.

d/

-- 

   Dave Crocker

---

From: <[email-masked]>
Date: Thu, 21 Jul 2016 05:00:25 +0000 (UTC)

>It is quite common to have posters who contribute little or nothing, or 

I would be very interested to hear what others have to say on this topic.  My experience is that in the WGs that I follow, that most of the contributions have a fair amount of merit.  Of course, some more than others.  And sure,every once in a while, someone makes a comment that is not so great and it is, by and large, disregarded or censured by the community.  
But, that is only my experience and only one data point.   What do others think?   
 Nalini

---

From: Melinda Shore <[email-masked]>
Date: Thu, 21 Jul 2016 07:08:09 +0200

> But, that is only my experience and only one data point.   What do

Well, it really depends.  "+1" posts really don't contribute
that much, while "+1 because < ... >" or "+1 except for < ... >"
might.  Or, they might not.

But stepping back a bit I'm a bit concerned about the overall
approach (leaving aside questions of scope).  As a matter of
experimental design it probably makes more sense to collect
the data, run at least some summary numbers, and identify some
testable questions from them than it does to start with a set of
definitions and see who falls into the categories you've identified.

But, we do need to address the higher-level questions of whether
or not this is worth doing and if so where the work should take
place before digging into experimental design, I think.

Melinda

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 21 Jul 2016 06:58:02 +0000

> On Jul 20, 2016, at 8:11 PM, Dave Crocker <[email-masked]> wrote:
> 
> Well, it gets easier when the input is an xml source, but for classic ascii source, other than obvious header-searching algorithms, no I haven't a clue.

One thing I found myself thinking about last night was the Note Well. Legally, https://www.ietf.org/about/note-well.html says that a variety of fairly unmeasurable events, such as speaking at a mike, are contributions. I understand the desire to find the substantive contributors, which include those acknowledged in, contributing to, or writing drafts; in my working group I have a number of people whose contributions I value quite a bit that AFAIK never attend a meeting, but comment on the email list. From my perspective, there are some forms of contribution that are more readily measured and some that are less so. It seems to me that the way to approach it is to figure out what can be measured and decide to what degree we think it's relevant or gives us the information we need.

Yes some of that is being rewarded for showing up, for some definition of "showing up". Our numbers are all over the map; we have 1346 in physical attendance, and 264 in remote attendance this week, Ray said last night IIRC. We have a total of 2135 posted internet drafts as of this morning, with 2746 distinct email addresses in their respective "author" sections, which is almost twice as many. 2812 people (including [email-masked], whoever that is) posted email to some [email-masked] in 2016, which might have been pages and pages or as simple as "+1", which is important when I'm measuring consensus. I have no idea how many people are acknowledged or made it into "contributor" sections, because last night I slept rather than reading 2135 internet drafts. I did think about a tool, and there may be some analytic heuristics out there. If Google is to be believed, someone once posted a corpus of tens of thousands of names...

I think we need a measurable, repeatable, way to identify contributors or participants, and to say where they are located, if we are going to try to position meetings near our contributors. Our definitions in the past have been based on internet draft authorship and physical attendance. Happy to extend that, but I need a way to do it that I can implement in a tool, that anyone can use to repeat the experiment. And absent that, I'm not sure how much pontification on what it means to be a participant or a contributor provides value.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 21 Jul 2016 09:01:50 +0200

On 7/21/2016 8:58 AM, Fred Baker (fred) wrote:
> It seems to me that the way to approach it is to figure out what can be measured and decide to what degree we think it's relevant or gives us the information we need.

+1

The difficult part is, of course, the latter half of your sentence.

-- 

   Dave Crocker

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 21 Jul 2016 19:18:59 +1200

> I'm thinking through the mechanisms by which one might extract the acknowledgements, contributors, and authors sections of an internet draft or RFC, and extract from it human names. If you have som…

It isn't. But "talk to Henrik" might point to a solution. He has a bunch of code
that extracts stuff from drafts and RFCs using a bunch of heuristics. I think
he could tell us pretty quickly what is realistic and what is science fiction.

   Brian

P.S. Re postings on mailing lists: I think that's binary. Either we count
all postings, or none. The fact that someone's posting is trivial is hard
to decide (even for humans, and definitely for algorithms). The only exception
might be to discard all messages that top-post the string "+1". (Personally
I'd also discard all messages that only make sense when viewed as HTML, but
that's me being old-fashioned.)

---

From: S Moonesamy <[email-masked]>
Date: Thu, 21 Jul 2016 09:33:58 -0700

Hi Nalini,
At 22:00 20-07-2016, [email-masked] wrote:
>I would be very interested to hear what others have to say on this 
>topic.  My experience is that in the WGs that I follow, that most of 
>the contributions have a fair amount of merit.  Of course, some more 
>than others.  And sure,every once in a while, someone makes a 
>comment that is not so great and it is, by and large, disregarded or 
>censured by the community.
>

The difference between a person attending a face-to-face meeting and 
a person only posting to a Working Group mailing list is USD 
875.  There hasn't been a need to rate the contribution of the person 
in either of those two categories for meeting purposes.

Regards,
S. Moonesamy

---

From: Niels ten Oever <[email-masked]>
Date: Sat, 23 Jul 2016 23:49:35 +0200

Hi Fred,

This tool we've been working on for mailinglist analysis might proof
useful to you: https://github.com/nllz/bigbang

Also relatively easy to add new notebooks for analysis (or use parts of
the existing ones).

If you have any issues, please let me know via the issue tracker or per
e-mail.

If you find something nice, I would be very happy with a pull request.

Cheers,

Niels

On 07/20/2016 08:08 PM, Fred Baker (fred) wrote:
>> This is why I've suggested reaching no deeper than the
>> Acknowledgements and Contributors sections of I-D's and RFCs (as
>> well as author lists, of course.)  If a name shows up there, the
>> person made it through a basic filter for having been seen to
>> actually contribute.
> 
> I'm thinking through the mechanisms by which one might extract the
> acknowledgements, contributors, and authors sections of an internet
> draft or RFC, and extract from it human names. If you have some good
> heuristics, I'd be interested. I don't think "hire an intern" is a
> long term solution.
> 
> 
> 
> _______________________________________________ Mtgvenue mailing
> list [email-masked] 
> https://www.ietf.org/mailman/listinfo/mtgvenue
> 

-- 
Niels ten Oever
Head of Digital

Article 19
www.article19.org

PGP fingerprint    8D9F C567 BEE4 A431 56C4
                   678B 08B5 A0F2 636D 68E9

---

From: <[email-masked]>
Date: Sun, 24 Jul 2016 06:41:01 +0000 (UTC)

>If you find something nice, I would be very happy with a pull request.

Niels, thanks for the pointer to this.   I will look at this also.
One thing, we had a meeting with the WG chairs and have decided to slow down the discussion on this draft and have everyone focus on the other drafts so they can get finished up.   Then, we will take up the issues posed in this draft to see how to proceed.  
I will continue to work in the background on thinking and exploring these issues.
Thanks so much for your help & interest.
Nalini

-- 
Niels ten Oever
Head of Digital

Article 19
www.article19.org

_______________________________________________
Mtgvenue mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mtgvenue