Subject: Re: [Eligibility-discuss] Measuring contribution or participation? [was "Additional Criteria"]

From: Ted Lemon <[email-masked]>
Date: Sat, 1 Aug 2020 08:25:59 -0400

I don’t think you are wrong in what you want. I hope that we can come up with a way to approximate it. I think the only measures that we have of actual contribution that are machine countable are for leadership positions, article editorship and directorate reviews. Everything else is measuring whether people are showing up, in one form or another. 

The distinction between physically showing up and showing up on the mailing is really a measure of work, but heavily confounded by opportunity: you have to want to come to the meeting, or your employer has to want representation at the meeting and what comes with it, /and/ you have to live in a country where there is money for travel and ability to travel, /and/ you have to be able to get a job where there is a possibility they will send you. This has always excluded a significant number of genuine contributors. 

The mailing list approach removes these barriers, and that’s worrisome because now we are opening the floodgates, and obviously it has the potential for abuse. But mailing list participation could actually be a better proxy for contribution. It doesn’t exclude people who lack opportunity /as much/. (You still need an Internet connection that can carry a live video stream if we intersect it with “register and showed up online.”)

I think if we really try to measure contribution we are going to miss enough people to strongly bias the results because of the machine counting problem. I think counting mailing list participation is a decent proxy for participation, although not perfect. We’ve discussed some ways to make it harder to game. 

So I think we actually agree on our goals at least, if not our methods. 

> On Aug 1, 2020, at 00:08, Joel M. Halpern <[email-masked]> wrote:
> 
> What I would like to measure is meaningful participation.  However, as far as I can tell directly measuring that is impossible.  So we are discussing various measures of contribution.  And some measures of participation that are not calibrated to "meaningful".  I personally tend to see the measures of contribution as better proxies for what I want.  But I may be wrong on that.
> 
> Yours,
> Joel
> 
>> On 7/31/2020 11:31 PM, Brian E Carpenter wrote:
>>> On 01-Aug-20 15:19, Ted Lemon wrote:
>>> Sorry, I meant intersection, not union. I think we are measuring participation. I don’t know how we could measure contribution. Maybe that’s our disconnect?  I might have misspoken on that point—it’s helpful that you made the distinction explicit. There is not way to do that that’s not subjective and I don’t blame you for questioning my sanity if that was what I seemed to you to be saying.
>> My sanity is at least as much in question ;-). Indeed, I think the objective has been a bit unclear in a lot of the discussion on this list. I'd be interested in other people's opinion on this point.
>>     Brian
>>>> On Jul 31, 2020, at 23:04, Brian E Carpenter <[email-masked]> wrote:
>>>> 
>>>> Ted,
>>>> 
>>>> I have a specific question at the end:
>>>> 
>>>>> On 01-Aug-20 09:20, Ted Lemon wrote:
>>>>>> On Jul 31, 2020, at 4:59 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:
>>>>>> The general objection that people raised earlier** was that if sending email and/or writing reviews was counted, it would be easy for people to cheat the system by sending trivial email or reviews. And that would be possible well below the threshold for triggering PR-Actions. So either you need a subjective judgment or an AI solution to prevent gaming. And either of those would be very much subject to appeal.
>>>>> 
>>>>> Brian, I think I’ve said three or four times, please actually argue with me if you disagree with this:
>>>> 
>>>> (I have found myself reluctant to express personal preferences as long as I'm editing the draft, since I'd prefer to edit it in the direction of possible consensus.)
>>>> 
>>>>> right now all you have to do to be counted for nomcom is show up at a meeting. That’s a _really_ low bar if we start counting online meetings.
>>>> 
>>>> Exactly right. And at the moment the draft ducks that question. If we want to add on-line attendance, we have to be careful what it means. And we hit the same problem as for email and reviews: what level of participation counts? Shows up and listens? Shows up and hums? Shows up and speaks?
>>>> 
>>>>> If you union that with other participation measures, I think it raises the bar while still meaningfully including the right people. Mailing list participation is one such measure.
>>>> 
>>>> Wait, doesn't union *lower* the bar by including a wider set of people? Intersection raises the bar.
>>>> 
>>>> I suppose that what we're trying to measure is really contribution, not participation. That's why it's tricky. If we gave up on that and only measured participation, life gets much easier. What is our goal?
>>>> 
>>>> Regards
>>>>   Brian
>>>> 
>>>> 
>>> 
> 
> -- 
> Eligibility-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/eligibility-discuss