Subject: [Eligibility-discuss] Thinking outside the box

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 6 Jul 2020 19:06:22 +0000

I've been reading these posts. Given there seems to be consensus that what the people on this list want to achieve (an optimized means of identifying people qualified to be NomCom members, where "qualified" means they have experience with and knowledge of IETF processes and people)...

Why not create a test people can take to prove their knowledge? It could even have a recorded or on-line class to teach the materials that will be on the test.

This would be something that could be experimented with (create and offer a test; see how many people take and pass the test) before changing any NomCom selection criteria.

If the test is well-designed, a significant number of people who pass it should have the desired knowledge.

Of course there also needs to be an "experience" criterion (loosely put: needs to have been active in IETF for several years). But since experience would no longer the only criterion (currently, there exist no knowledge-based criteria) it becomes less important that it be "perfectly" optimized.

Just a thought. The thought popped into my head and wouldn't go away till I sent this email.
Barbara

---

From: Ted Lemon <[email-masked]>
Date: Mon, 6 Jul 2020 15:27:17 -0400

I think this is a really useful thought experiment. My reaction to it is that it doesn’t capture one thing that we need, which is for people to actually want the IETF to succeed. I think this _is_ measured by participation. If you want to see why this matters, look at the Wikipedia rules, and look at how they’ve been gamed by people who care about pushing their viewpoint, not about making an awesome encyclopedia.

> On Jul 6, 2020, at 3:06 PM, STARK, BARBARA H <[email-masked]> wrote:
> 
> -- 
> Eligibility-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/eligibility-discuss

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 6 Jul 2020 19:57:56 +0000

> From: Ted Lemon <[email-masked]>
> 
> I think this is a really useful thought experiment. My reaction to it is that it
> doesn’t capture one thing that we need, which is for people to actually want
> the IETF to succeed. I think this _is_ measured by participation. If you want to
> see why this matters, look at the Wikipedia rules, and look at how they’ve
> been gamed by people who care about pushing their viewpoint, not about
> making an awesome encyclopedia.

Right. People need experience/interest AND knowledge. Trying to gauge knowledge by only judging experience/commitedness doesn't work well; similarly, trying to judge experience/commitedness by only judging knowledge wouldn't work well. If these are both desirable, then they should be separately judged. What I'm seeing is discussion of "what experience/commitedness criteria can we use that will optimize likelihood the people are also knowledgeable of IETF processes?". If the criteria are separated, then criteria for experience/commitedness can perhaps be simplified. Use a knowledge-based test to judge the knowledge criterion.
Barbara

> > I've been reading these posts. Given there seems to be consensus that
> what the people on this list want to achieve (an optimized means of
> identifying people qualified to be NomCom members, where "qualified"
> means they have experience with and knowledge of IETF processes and
> people)...
> >
> > Why not create a test people can take to prove their knowledge? It could
> even have a recorded or on-line class to teach the materials that will be on
> the test.
> >
> > This would be something that could be experimented with (create and
> offer a test; see how many people take and pass the test) before changing
> any NomCom selection criteria.
> >
> > Of course there also needs to be an "experience" criterion (loosely put:
> needs to have been active in IETF for several years). But since experience
> would no longer the only criterion (currently, there exist no knowledge-
> based criteria) it becomes less important that it be "perfectly" optimized.
> >

---

From: Victor Kuarsingh <[email-masked]>
Date: Mon, 6 Jul 2020 16:04:21 -0400

Ted / Barbara,

I am willing to look at this as a thought experiment as well.  In addition
to Ted's comments on finding folks willing to want the IETF to succeed,
there is one more element which may be hard to capture.

In my experience, it has been highly valuable to have members of the Nomom
who know how the flow of work is progressing and know the individuals whom
are running.  This applies to all roles, but definitely applies to the IESG
and IAB roles.

The point here is that "what is relevant" changes over time, so what is
needed does as well.  Knowing not just the technical components (i.e.
follow drafts, emails, lists, etc), but also the dynamics of the work, long
with the behaviors/methods of the folks being considered, is highly
valuable and needed context.  If this changes, how do we update the "test"
to be most relevant to the team? (I suspect the relevancy may change year
to year).

Having folks whom (at least traditionally) attended the meetings, provides
a coarse way of ensuring there was at least some of this
relevant context within the DNA of the sitting Nomcom.

I realize this all changes with more remote meetings, but thought bringing
the up as a datapoint would be helpful.

regards,

Victor K

On Mon, Jul 6, 2020 at 3:27 PM Ted Lemon <[email-masked]> wrote:

> I think this is a really useful thought experiment. My reaction to it is
> that it doesn’t capture one thing that we need, which is for people to
> actually want the IETF to succeed. I think this _is_ measured by
> participation. If you want to see why this matters, look at the Wikipedia
> rules, and look at how they’ve been gamed by people who care about pushing
> their viewpoint, not about making an awesome encyclopedia.
>
> identifying people qualified to be NomCom members, where "qualified" means
> they have experience with and knowledge of IETF processes and people)...
> >
> even have a recorded or on-line class to teach the materials that will be
> on the test.
> >
> would no longer the only criterion (currently, there exist no
> knowledge-based criteria) it becomes less important that it be "perfectly"
> optimized.
> >

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 7 Jul 2020 16:10:38 +1200

While we're outside the box, it did occur to me the other day that a Mechanical Turk approach (you have to do some work) could be applied too. At random times when using the data tracker, a little captcha pops up and says "click here if you're interested in the IETF nominating process". At NomCom time, the top 500 clickers form the eligible pool.

The reason for making it happen at random is to prevent it being too easy; you have to be actually doing something with the tracker to be offered the captcha.

Getting back to Barbara's idea, could somebody mock it up on... no wait:
https://www.surveymonkey.com/r/3L8PX5P
Designing the right questions is not so easy.

Regards
   Brian

On 07-Jul-20 08:04, Victor Kuarsingh wrote:
> Ted / Barbara,
> 
> I am willing to look at this as a thought experiment as well.  In addition to Ted's comments on finding folks willing to want the IETF to succeed, there is one more element which may be hard to capture. 
> 
> In my experience, it has been highly valuable to have members of the Nomom who know how the flow of work is progressing and know the individuals whom are running.  This applies to all roles, but definitely applies to the IESG and IAB roles.  
> 
> The point here is that "what is relevant" changes over time, so what is needed does as well.  Knowing not just the technical components (i.e. follow drafts, emails, lists, etc), but also the dynamics of the work, long with the behaviors/methods of the folks being considered, is highly valuable and needed context.  If this changes, how do we update the "test" to be most relevant to the team? (I suspect the relevancy may change year to year).
> 
> Having folks whom (at least traditionally) attended the meetings, provides a coarse way of ensuring there was at least some of this relevant context within the DNA of the sitting Nomcom. 
> 
> I realize this all changes with more remote meetings, but thought bringing the up as a datapoint would be helpful.  
> 
> regards,
> 
> Victor K
> 
> 
> On Mon, Jul 6, 2020 at 3:27 PM Ted Lemon <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     > On Jul 6, 2020, at 3:06 PM, STARK, BARBARA H <[email-masked] <mailto:[email-masked]>> wrote:
>     >
>     > [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Ted Lemon <[email-masked]>
Date: Tue, 7 Jul 2020 00:14:37 -0400

> While we're outside the box, it did occur to me the other day that a Mechanical Turk approach (you have to do some work) could be applied too. At random times when using the data tracker, a little…

Good god no.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 7 Jul 2020 17:05:58 +1200

On 07-Jul-20 16:14, Ted Lemon wrote:
> On Jul 7, 2020, at 12:10 AM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:

Well, frankly I don't see it as any more ridiculous than simply being subscribed to a few mailing lists. It wouldn't actually be easy to game.

   Brian

---

From: Ted Lemon <[email-masked]>
Date: Tue, 7 Jul 2020 01:14:19 -0400

A bit of JavaScript and we’d all be in the top five hundred. But if it worked the way you intend, it would be interrupting us to do useless work. Pretty much turning the ietf into adware. No thanks. 

> On Jul 7, 2020, at 01:06, Brian E Carpenter <[email-masked]> wrote:
>

---

From: Victor Kuarsingh <[email-masked]>
Date: Tue, 7 Jul 2020 08:52:26 -0400

Ted,

What about having the option when filling in the blue sheets?  Seems like
an appropriate time to check a box.  If we move to virtual blue sheets,
that still works.  Just thinking

regards,

Victor K

On Tue, Jul 7, 2020 at 1:14 AM Ted Lemon <[email-masked]> wrote:

> A bit of JavaScript and we’d all be in the top five hundred. But if it
> worked the way you intend, it would be interrupting us to do useless work.
> Pretty much turning the ietf into adware. No thanks.
>
> >>> On Jul 7, 2020, at 12:10 AM, Brian E Carpenter <
> [email-masked] <mailto:[email-masked]>> wrote:
> >>> While we're outside the box, it did occur to me the other day that a
> Mechanical Turk approach (you have to do some work) could be applied too.
> At random times when using the data tracker, a little captcha pops up and
> says "click here if you're interested in the IETF nominating process". At
> NomCom time, the top 500 clickers form the eligible pool.
> >>
> >> Good god no.
> >
> > Well, frankly I don't see it as any more ridiculous than simply being
> subscribed to a few mailing lists. It wouldn't actually be easy to game.
> >
> >   Brian
>

---

From: Ted Lemon <[email-masked]>
Date: Tue, 7 Jul 2020 10:13:39 -0400

That’s an easy way to replicate the in-person approach online. It’s not too bad, but definitely very low effort for someone who wants to game the process. The advantage of using reviews or some other measure of active participation is that the effort involved even to phone it in is higher. 

> On Jul 7, 2020, at 08:52, Victor Kuarsingh <[email-masked]> wrote:
> 
> 
> Ted,
> 
> What about having the option when filling in the blue sheets?  Seems like an appropriate time to check a box.  If we move to virtual blue sheets, that still works.  Just thinking 
> 
> regards,
> 
> Victor K
>