Subject: [Eligibility-discuss] some different conclusions from the additional data Robert and Stephen dug up

From: Michael Richardson <[email-masked]>
Date: Sun, 05 Jul 2020 14:49:42 -0400

Hi, based upon some comments from Ted, I've realized that the conclusions I
was making were misguided!  Not wrong, but perhaps unhelpful.

I was looking at how we can enfranchise more people.
This was the problem of 2019: how can we be more inclusive.

The problem of 2020 is how can we continue to enfranchise people when people
who were attending in person can not.

These are *not* mutually exclusive goals!!
Some of my comments about path 2 (WG chairs) not being useful, because
3-of-5, are incorrect.  Path 2 is probably not useful to enfranchise *more*
Path 2 is probably very useful to preserve franchise.

Given that view, the diagrams/numbers need to re-interpreted as follows:
{I attach my original text at the bottom so you don't need to dig for it.
Please note the difference in tone.
I do not advocate dis-enfranchising anyone, but as non-volunteers, they
should not be a priority}

(1-2018-5yr) of the 779 previously eligible,
    379 of those who were previously eligibile via 3-of-5, would continue to
        be eligible.
    357 of the 3-of-5 eligible non-volunteering people would be dis-enfranchised.
    43  volunteers would be lost from the pool without 3-of-5

(2-2018-3yr) of the 779 previously eligible,
    379 of those who were previously eligibile via 3-of-5, would continue to
        be eligible.
    353 of the 3-of-5 eligible non-volunteering people would be dis-enfranchised.
    47  volunteers would be lost from the pool without 3-of-5

(3-2019-5yr) of the 797 previously eligible,
    361 of those who were previously eligibile via 3-of-5, would continue to
        be eligible.
    380 of the 3-of-5 eligible non-volunteering people would be dis-enfranchised.
    56  volunteers would be lost from the pool without 3-of-5

(3-2019-3yr) of the 797 previously eligible,
    334 of those who were previously eligibile via 3-of-5, would continue to
        be eligible.
    405 of the 3-of-5 eligible non-volunteering people would be dis-enfranchised.
    58  volunteers would be lost from the pool without 3-of-5

Given that that the volunteer pool is typically of order 200, 43 to 58
represents about 25% of the pool.

1) It would be good to understand why so few of the doubly-eligible people
   volunteer.  If we can get 15% of those to step forward, that would
   preserve the size of the volunteer pool.

2) We have between ~320 (3yr) and ~500 (5yr) potential volunteers,
   who have never been eligible before.  We have no idea how many would
   volunteer.  We should probably ask them!
   Getting 10 to 15% of them would also preserve the size of the pool.

3) *preserving* the size of our pool is probably not an amphitious enough
   goal. Growing the pool is important.
   *disenfranchising* 25% of our pool as we move towards fewer in-person
   meetings is probably a really really bad thing.  These are people
   who have volunteered in the past, and probably a few have served.

   ** Determining what other criteria would include them is probably very
   ** important.


My attempt to summarize the diagram data.

(1-2018-5yr) of the 779 previously eligible,
    379 were also eligible according to either criteria, of which 142 volunteered.
    An additional 43 volunteers were eligible only by 3-of-5.
    357 3-5 eligible people did not volunteer.
    Conclusion 2018/5yr brings 542 potential volunteers to the pool.

(2-2018-3yr) of the 779 previously eligible,
    379 were also eligible according to either criteria, of which 138 volunteered.
    An additional 47 volunteers were eligible only by 3-of-5.
    353 3-5 eligible people did not volunteer.
    Conclusion 2018/3yr brings 364 potential volunteers to the pool.

(3-2019-5yr) of the 797 previously eligible,
    361 were also eligible according to either criteria, of which 115 volunteered.
    An additional 56 volunteers were eligible only by 3-of-5.
    380 3-5 eligible people did not volunteer.
    Conclusion 2018/5yr brings 484 potential volunteers to the pool.

(3-2019-3yr) of the 797 previously eligible,
    334 were also eligible according to either criteria, of which 113 volunteered.
    An additional 58 volunteers were eligible only by 3-of-5.
    405 3-5 eligible people did not volunteer.
    Conclusion 2018/3yr brings 303 potential volunteers to the pool.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: "Pete Resnick" <[email-masked]>
Date: Sun, 05 Jul 2020 14:06:42 -0500

> who were attending in person can not.

Something has been bugging me about the whole direction of this 
discussion, and I think this gets to the nub of it.

A few folks have been talking in terms of "enfranchising" more people 
(normally defined as "granting people the *right* to vote"), or that 
more people "should be eligible" to be on the NomCom, or that somehow we 
want NomCom eligibility to be more "fair" to contributors, or remote 
participants, or others. This seems completely backwards.

As far as I can tell, this is not about the rights of any individual or 
the fairness to any individual or anything of the sort. This is about 
how, to some best approximation, using some objective and easily 
measurable criteria, we can find people who are familiar with IETF 
processes and procedures (and as John said, even though unstated, 
familiar with enough people in the IETF, including potential candidates 
and community members who could give feedback) to be able to 
productively participate in the NomCom discussions and contribute to 
making reasonable selections.

I would argue that this is not deciding on rules to enfranchise voters; 
this is deciding on rules for jury selection. We're looking to maximize 
qualified jurists and minimize unqualified jurists.

No objective and easily measurable criteria we come up with are going to 
eliminate all unqualified jurists (that is, NomCom volunteers), nor 
include all qualified ones. Our aim should be to come up with criteria 
that doesn't exclude lots of qualified volunteers (for example, many of 
the remote participants that have been quite involved and can contribute 
well to a NomCom) and still minimize the number of unqualified 
volunteers.

We used to think that attendance at meetings was, as John said, a good 
approximation, maximizing the qualified and minimizing the unqualified. 
We (a) don't think meeting attendance is maximizing the qualified 
anymore; (b) know that with the past two meetings remote and more people 
using remote attendance, we're likely missing a larger and larger number 
of qualified folks; and (c) think we can come up with criteria that can 
do better. We don't have the resources to evaluate the criteria 
beforehand and get reliable answers; we're going to do this process 
experiment and see what we think of the volunteer pools we do end up 
with.

I think we can come up with those criteria, but only if we ask the 
question, "Is this criteria likely to help maximize qualified volunteers 
without too many unqualified volunteers?" We shouldn't (IMO) be looking 
to simply maximize the pool, and we especially should be asking about 
the fairness to or rights of the potential volunteers.

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: "Pete Resnick" <[email-masked]>
Date: Sun, 05 Jul 2020 14:09:29 -0500

On 5 Jul 2020, at 14:06, Pete Resnick wrote:

> and we especially should be asking about the fairness to or rights of 
> the potential volunteers.

Darn. s/should/shouldn't

pr

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 5 Jul 2020 20:17:51 +0100

Hi Pete,

>> the potential volunteers.

I agree with you as to our primary goal, but disagree with
the above. I do think we ought prefer changes that are more
fair to IETF participants who don't attend meetings over
changes that are less fair to that cohort.

Cheers,
S.

---

From: Michael Richardson <[email-masked]>
Date: Sun, 05 Jul 2020 15:46:26 -0400

Pete Resnick <[email-masked]> wrote:
    > I think we can come up with those criteria, but only if we ask the question,
    > "Is this criteria likely to help maximize qualified volunteers without too
    > many unqualified volunteers?" We shouldn't (IMO) be looking to simply
    > maximize the pool, and we especially shouldn't be asking about the fairness to
    > or rights of the potential volunteers.

I agree that maximizing the size of the volunteer pool is not a terribly good
metric! I note that we get less than 1/4 of the currently enfranchised volunteering.
So many having more eligible people won't help improve the size of the
volunteer anyway.
Maybe we are getting the 1/4 highest qualified people, and self-selection is
actually the best criteria.  This seems unlikely, but it could be.

I am not overly concerned about gaming, nor am I concerned that we will get
too many unqualified people.

I have been mostly concerned that we systematically dis-enfranchise people
who can not travel, often due to family reasons.
This hits people in their child-bearing years, and while fathers are often
unwilling to travel, the gender bias here is pretty strong.
It also hits people with who have to care for elderly, or who develop health
issues, whether personal or pandemnic.

I believe that we could write out the RFC3933 criteria in this document
as-is, and ship it.
While I have said that we should have an idea on step two, it's a sword
I'm less willing to fall on now than three days ago.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 6 Jul 2020 09:02:03 +1200

> I'm less willing to fall on now than three days ago.

Do you mean as-is with the review team activity (path 3) added back in?

Indeed, this is suggested as a 2-year experiment, which retains the
current attendance criterion, so it would disenfranchise nobody compared
to the current rules. It would give us more data to make a long term
change, including dropping the meeting attendance criterion as John hinted.

(I'd also be in favour of adding tooling to support WG-internal reviews,
for its own sake, but with the side benefit of making it a measurable
criterion. However, that really does deserve its own separate draft and
discussion.)

    Brian

---

From: John C Klensin <[email-masked]>
Date: Sun, 05 Jul 2020 17:47:11 -0400

Pete,

With a few quibbles, I think this is exactly right.  Quibbles
inline...

--On Sunday, July 5, 2020 14:06 -0500 Pete Resnick
<[email-masked]> wrote:

>...
 
> As far as I can tell, this is not about the rights of any
> individual or the fairness to any individual or anything of
> the sort. This is about how, to some best approximation, using
> some objective and easily measurable criteria, we can find
> people who are familiar with IETF processes and procedures
> (and as John said, even though unstated, familiar with enough
> people in the IETF, including potential candidates and
> community members who could give feedback) to be able to
> productively participate in the NomCom discussions and
> contribute to making reasonable selections.
> 
> I would argue that this is not deciding on rules to
> enfranchise voters; this is deciding on rules for jury
> selection. We're looking to maximize qualified jurists and
> minimize unqualified jurists.

Yes, exactly.

> No objective and easily measurable criteria we come up with
> are going to eliminate all unqualified jurists (that is,
> NomCom volunteers), nor include all qualified ones. Our aim
> should be to come up with criteria that doesn't exclude lots
> of qualified volunteers (for example, many of the remote
> participants that have been quite involved and can contribute

Right, and that is where I agree with Michael's comments about 1
in 10, while repeating my concern about stochastic accidents.
However, let me push the jury analogy a bit further. If we have
some set of criteria for Nomcom members that are legitimately
tied to the needs of the IETF (even if some of them are a bit
subjective or hard to measure for other reasons) and the jury
(or the jury pool) systematically excludes one
easily-identifiable group of people who meet those criteria
while including other groups who also meet them, that is, in
jury-speak, a biased jury (and A Bad Thing).  Second, unlike a
typical jury setup (at least in the US, UK, and countries using
similar systems, we have no real challenge mechanism (other than
the "no more than two from one company" rule) that could be used
to preserve balance and absence of inappropriate influences if
random selection from a potentially-biased pool doesn't do that
for us.

> We used to think that attendance at meetings was, as John
> said, a good approximation, maximizing the qualified and
> minimizing the unqualified. We (a) don't think meeting
> attendance is maximizing the qualified anymore; (b) know that
> with the past two meetings remote and more people using remote
> attendance, we're likely missing a larger and larger number of
> qualified folks;

I would have said that we have been missing larger and larger
numbers of qualified folks for some time.  The change to
all-remote meetings certainly creates a steep inflection point
in the curve, but probably does not change the fact of a rising
trend.   What the new, steeper, part of the curve does is to
make the problem harder to ignore.

>  and (c) think we can come up with criteria
> that can do better. We don't have the resources to evaluate
> the criteria beforehand and get reliable answers; we're going
> to do this process experiment and see what we think of the
> volunteer pools we do end up with.

yep.

> I think we can come up with those criteria, but only if we ask
> the question, "Is this criteria likely to help maximize
> qualified volunteers without too many unqualified volunteers?"
> We shouldn't (IMO) be looking to simply maximize the pool, and
> we especially should [not] be asking about the fairness to or
rights
> of the potential volunteers.

But that introduces two other problems.  First, while I agree
about fairness and rights of volunteers, it is not hard to
imagine a systematically unfair Nomcom, e.g., one with a bias
toward selecting candidates for IETF positions from a particular
organization or cluster of companies.  If we are not going to
reduce the chances of that by how we control admission criteria
for the pool or how the Nomcom is selected from that pool, we
have no other mechanisms for doing so.  

Second, as long as we are going to use the same criteria for
volunteering for the pool that we use for people who can protect
themselves against abusive or inappropriate behavior, especially
behavior that selectively impacts them and people like them,
there really is a question of fairness (and maybe rights) of
people who are Nomcom-eligible.  So, if we don't want to ask
about fairness to or rights of potential volunteers, we need to
separate the ability to initiate a recall petition off and do so
more or less right now.

best,
  john

---

From: Michael Richardson <[email-masked]>
Date: Sun, 05 Jul 2020 18:21:56 -0400

> Do you mean as-is with the review team activity (path 3) added back in?

I'm mostly agnostic actually.

I would be happiest if we had some agreement about how/when to put some
path-3 like activity back in.

That would be an example of a "phase 2" type thing.
By announcing that the criteria will include something, and providing a
strawman criteria, subject to another IETF LC probably, in two years,
then we would be able to see how this changed review team activity.
  -> nobody would be surprised.

    > change, including dropping the meeting attendance criterion as John hinted.

Well, it depends upon how the 2021 nomcom chair/IESG decide to count IETF107
and IETF108.  And while it is conceivable that 109 could go ahead in-person,
if it went ahead while USA located people could not attend, then that would
be even more difficult.

I have to assume that this document will be the basis for NomCom 2021,
thus my focus on the ~50 volunteers who would be dis-enfranchised.

    > criterion. However, that really does deserve its own separate draft and

Agreed.

---

From: "Pete Resnick" <[email-masked]>
Date: Mon, 06 Jul 2020 12:27:12 -0500

On 5 Jul 2020, at 14:17, Stephen Farrell wrote:

>>> and we especially should[n't] be asking about the fairness to or 
>>> rights of

Absolutely, insofar as "fair" means "unbiased", in that we don't exclude 
volunteers simply because they don't attend meetings, I agree; if there 
are qualified (i.e., understand IETF policies, procedures, etc) people 
who are remote, we want them. But people seem to be using "fair" to mean 
"equitably rewarded", where being eligible for the NomCom is the reward. 
That's what I disagree with. Nobody "deserves" to be on the NomCom 
except those who have the requisite knowledge to be on the NomCom.

pr

---

From: Ted Lemon <[email-masked]>
Date: Mon, 6 Jul 2020 13:36:54 -0400

On Jul 6, 2020, at 1:27 PM, Pete Resnick <[email-masked]> wrote:
> Absolutely, insofar as "fair" means "unbiased", in that we don't exclude volunteers simply because they don't attend meetings, I agree; if there are qualified (i.e., understand IETF policies, procedures, etc) people who are remote, we want them. But people seem to be using "fair" to mean "equitably rewarded", where being eligible for the NomCom is the reward. That's what I disagree with. Nobody "deserves" to be on the NomCom except those who have the requisite knowledge to be on the NomCom.

I think this is a useful distinction to make, but I think it would be a more accurate characterization to say that we failed to make it rather than that our actual focus was on “fairness.”  Although admittedly there is the “can file appeals” aspect which _is_ about fairness.

It might be better to frame this as “if we only draw our pool from people who are able to show up for many meetings, that introduces a likely significant bias which we are not competent to measure.”

But another question is, does how we frame this actually matter? It feels a bit like you’re saying that only one frame can be right, but in fact I think both frames are right, both frames are important, and the conclusions they lead us to are not incompatible.

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 6 Jul 2020 18:49:53 +0100

Hiya,

> except those who have the requisite knowledge to be on the NomCom.

Thanks for the clarification. I'm fully with ya now:-)

Cheers,
S.

---

From: "Pete Resnick" <[email-masked]>
Date: Mon, 06 Jul 2020 12:54:42 -0500

> With a few quibbles, I think this is exactly right.  Quibbles

Skipping down to quibbles...

> If we have
> jury-speak, a biased jury (and A Bad Thing).

Absolutely. See my reply to Stephen. The criteria should not introduce 
bias into the process.

> Second, unlike a

Right. We're trying to set up the criteria to maximize getting it right 
because we don't have a voir dire step after we get the list of 
volunteers; it's random selection. Again, it can't be perfect, so we 
just try to maximize qualified folks and minimize the unqualified ones.

> make the problem harder to ignore.

No disagreement. I was giving the history short shrift.

> have no other mechanisms for doing so.

Without the voir dire step, the only way we have to control for this is 
to set up reasonable, objectively measurable, criteria (as you point 
out) or have additional post-selection criteria like the "no more than 2 
from one org" rule. The additional rules that I can imagine aren't 
terribly palatable (e.g., no more that 2 from any given area, at least 
one person who has filed a successful appeal, ...).

> separate the ability to initiate a recall petition off and do so

I'm claiming that if we've chosen the criteria correctly, the above 
shouldn't have a disparate impact. For instance, it is plausible that 
remote attendees might be selectively subject to misbehavior worthy of 
the ability to appeal or recall. If they are disqualified simply because 
of their remoteness, that's a problem, so we are inclined to have NomCom 
eligibility criteria that do not eliminate remote participants. Now, if 
we can come up with instances certain classes of folks that we agree 
should have the ability to appeal or recall that are nonetheless 
eliminated by the NomCom criteria, and we can't come up with a 
reasonable NomCom criteria that includes those folks, then perhaps we do 
need a different set of criteria for NomComs as against appeals and 
recalls. But I haven't seen one of those yet. And still, I don't think 
that's about fairness to those folks (in the "equitable reward" sense), 
but rather that we don't have an unbiased objectively measurable 
criterium that captures them.

pr

---

From: "Pete Resnick" <[email-masked]>
Date: Mon, 06 Jul 2020 13:08:39 -0500

On 6 Jul 2020, at 12:36, Ted Lemon wrote:

>> Absolutely, insofar as "fair" means "unbiased", in that we don't 
>> exclude volunteers simply because they don't attend meetings, I 
>> agree; if there are qualified (i.e., understand IETF policies, 
>> procedures, etc) people who are remote, we want them. But people seem 
>> to be using "fair" to mean "equitably rewarded", where being eligible 
>> for the NomCom is the reward. That's what I disagree with. Nobody 
>> "deserves" to be on the NomCom except those who have the requisite 
>> knowledge to be on the NomCom.
>
> I think this is a useful distinction to make, but I think it would be 
> a more accurate characterization to say that we failed to make it 
> rather than that our actual focus was on “fairness.”

Fair enough (no pun intended), but some of the language that you and 
Michael use made me think that you were not simply conflating the two, 
but were slipping toward framing the argument as about a "right" or 
"equity". So long as we're on the same page, I'm fine.

> Although admittedly there is the “can file appeals” aspect which 
> _is_ about fairness.

See the very end of my recent reply to John. I *think* we can come up 
with objectively measurable NomCom criteria that include all of the "can 
file appeals" (and "can file a recall") people, but if we can't, then we 
should probably have a different set of objectively measurable criteria 
for them.

> It might be better to frame this as “if we only draw our pool from 
> people who are able to show up for many meetings, that introduces a 
> likely significant bias which we are not competent to measure.”

I think we are all in total agreement about that point.

> But another question is, does how we frame this actually matter? It 
> feels a bit like you’re saying that only one frame can be right, but 
> in fact I think both frames are right, both frames are important, and 
> the conclusions they lead us to are not incompatible.

I think the "rights" and "equitable reward" frames could lead us to 
include criteria that don't satisfy the "familiar with IETF policies, 
procedures, and people" qualification, and I *think* we don't want those 
folks on a NomCom, even it feels like the "deserve" to be. I've got no 
problem using that frame to give us an intuitive poke by saying, "My gut 
tells me that this set of people shouldn't be excluded, and the criteria 
we have proposed exclude them", but you still have to answer the 
question, "So, are there folks in that group who are qualified but still 
excluded?" and then explain why they are qualified yet don't fit in the 
existing criteria.

pr

---

From: Ted Lemon <[email-masked]>
Date: Mon, 6 Jul 2020 14:19:20 -0400

On Jul 6, 2020, at 2:08 PM, Pete Resnick <[email-masked]> wrote:
> I think the "rights" and "equitable reward" frames could lead us to include criteria that don't satisfy the "familiar with IETF policies, procedures, and people" qualification, and I *think* we don't want those folks on a NomCom, even it feels like the "deserve" to be.

Okay. Thanks for digging into this. This makes your concern a lot clearer to me, and I agree with you.

---

From: S Moonesamy <[email-masked]>
Date: Tue, 07 Jul 2020 07:38:10 -0700

Hi Pete,
At 12:06 PM 05-07-2020, Pete Resnick wrote:
>A few folks have been talking in terms of "enfranchising" more 
>people (normally defined as "granting people the right to vote"), or 
>that more people "should be eligible" to be on the NomCom, or that 
>somehow we want NomCom eligibility to be more "fair" to 
>contributors, or remote participants, or others. This seems 
>completely backwards.
>
>As far as I can tell, this is not about the rights of any individual 
>or the fairness to any individual or anything of the sort. This is 
>about how, to some best approximation, using some objective and 
>easily measurable criteria, we can find people who are familiar with 
>IETF processes and procedures (and as John said, even though 
>unstated, familiar with enough people in the IETF, including 
>potential candidates and community members who could give feedback) 
>to be able to productively participate in the NomCom discussions and 
>I would argue that this is not deciding on rules to enfranchise 
>voters; this is deciding on rules for jury selection. We're looking 
>to maximize qualified jurists and minimize unqualified jurists.

I'll comment on the first part of the message.  Here are some points 
which the Internet Architecture Board made a few years ago:

   (a) The people who judge whether consensus has been reached -- the
       Area Directors -- are selected at regular intervals by a nominations
       committee.

   (b) The members of the NomCom are volunteers who are regular participants
       in the IETF.

   (c) [The volunteers] are seated by random lot.

I'll assume that Point (a) is not being disputed.  I am not sure 
whether there is agreement on Point (b) as there isn't anyone 
assessing the mailing list discussions.  The interpretation of 
"regular participants" in combination with Point (c) raises the 
question of who has the right to vote.

Regards,
S. Moonesamy