Subject: [Eligibility-discuss] Gaming eligibility

From: Brian E Carpenter <[email-masked]>
Date: Fri, 27 Mar 2020 11:48:38 +1300

On jabber yesterday in gendispatch:

> Barry Leiba: Anything can be gamed, but let's not make it too EASY.
> Jonathan Lennox: Anything can be gamed, but attending in person is *expensive* to game.

It occurrs to me that making gaming the system cheaper would fix one of the
known problems of today: rich companies obtaining undue influence.
 
Regards
   Brian

---

From: Warren Kumari <[email-masked]>
Date: Thu, 26 Mar 2020 19:23:03 -0400

[ Top post ]

I'm personally not that worried about the gaming possibilities -- this
is for becoming eligible to *serve* on the NomCom, not win a car, or a
large cash prize. We already have the "No more than two volunteers
with the same primary affiliation may be selected for the nominating
committee.", and (IMO) the organizations which might want to exert
undue influence are all large enough that they can just ship people to
meeting already.

What I *am* concerned about is the side effect of gaming - tying the
eligibility to something which it either trivial to gain ("send an
email, get at point!"), which causes work for others / adds noise
("comment on 7 drafts!"),  or which require a judgment call ("submit a
review which results in 'substantive' change") will all add noise
without signal.  We really don't want people writing drafts just to
qualify, or getting listed as titular authors for this.

Yes, I did just argue that this isn't worth people gaming, while also
arguing that people will game -- I don't actually see this as a
tension - I don't think that we need to protect against adversaries
expending significant effort (other than for the lolz), but we do need
to worry about the slot drip of the side effect...

I'm also concerned that we might end up in a bikeshedding effort with
the "points" idea -- is writing a draft worth 17.3 points? why not 18
or 9.643? Is 4 directorate reviews worth more than a comment at a mic?
Do we rejigger the points system over time?
For some reason this also feels like it turns this into something like
"Collect 24 box tops, send them in a stamped self-addressed envelope,
and in 4 - 6 weeks we will send you a shiny sticker!!!!"

W

On Thu, Mar 26, 2020 at 6:49 PM Brian E Carpenter
<[email-masked]> wrote:
> --
> Eligibility-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/eligibility-discuss

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Michael StJohns <[email-masked]>
Date: Thu, 26 Mar 2020 21:17:33 -0400

On 3/26/2020 7:23 PM, Warren Kumari wrote:
> is for becoming eligible to*serve*  on the NomCom, not win a car, or a
> large cash prize.

Hi Warren -

It would be naive to ignore the fact that there's money involved here. 
The large cash prize might just be selecting an AD who selects a WG 
chair who ends up blessing a particular protocol mode that results in 
more sales or a bigger share of the market for the volunteer's

> We already have the "No more than two volunteers

Which turns out not to be the best protection against gaming. The math 
is pretty clear - with a large enough contribution (eligible volunteers) 
to the pool, a large organization can pretty much guarantee capture of 2 
slots.    Over the years we've generally had 3-4 "large" organizations 
that have most years collectively provided the majority of voting 
members and sometimes a few more.

Let's make it a bit harder to game:   If an org has 1-5 volunteers they 
get 1 token in the first pull, 6-20 - tokens, and 20+ 3 tokens (numbers 
subject to change).   For each token pulled, there's a second random 
selection from the org-related volunteers to pick the slot associated 
with the token.  Third token pulled from an org gets discarded.  This 
still gives larger organizations an edge in the number of selected 
volunteers, but will generally result in a more organizationally diverse 
set of volunteers.

The idea here is to prevent packing the Nomcom volunteer pool with seat 
fillers with the goal of getting *someone* selected and rather have an 
org pick the most qualified from its group as possible voting members.

As far as I can tell the criteria for an acceptable volunteer has to be 
something like:

1) The criteria used to accept volunteers into the pool must be clear 
and objective and not require massive amounts of work by the secretariat 
to adjudicate.

2) It should be possible to match any electronic volunteer statement 
with a unique real world person (e.g. the no sock puppet criteria)

3) It should be possible to identify any relevant associations for a 
given volunteer (e.g. the no organizational packing criteria).

4) - the hard one - The volunteer needs to have some sort of skin in the 
game with respect to the IETF (with a dual of "It should be hard to game 
this").

Perhaps one or more of:

    a) 3 of 5 in-person meetings (current criteria)
    b) Active WG chair after 1 year as chair (measured by meetings) (ADs 
should be getting rid of inactive WG chairs so?) Plus 1 year for each 
year of service as a WG chair to max of 3 years after last service.
    c) Former IAB/IESG for the 3 years after their last service
    d) Former NOMCOM members for 2 years after their last service.
    e) With my organization limit changes suggest above - 3 of 5 paid 
meetings regardless of in-person or virtual (assuming we can come to 
agreements on how to identify sockpuppets and associations).    Place 
the price high enough to be expensive to attempt to game, but low enough 
to broaden the community.

Later, Mike

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 27 Mar 2020 13:05:39 +0000

Yes, exactly.

I have been trying to make this point for awhile, this says it better than I have.

---

From: Victor Kuarsingh <[email-masked]>
Date: Fri, 27 Mar 2020 12:42:50 -0400

> known problems of today: rich companies obtaining undue influence.

Where it is true that "rich" companies can send more people to the
meetings, given we limit affiliation to no more than 2 for voting members,
businesses quickly reach their limit (seeding of attendees).  I would
actually be more worried about people gaming the system in general.

I think we need to keep some sort of bar which makes sense.  I find having
a nomcom body which is familiar with the work, the people and the
dynamics of the organization which is both what happens on lists and at the
venue to be quite valuable (IMO).

regards,

Victor K

---

From: Ted Lemon <[email-masked]>
Date: Fri, 27 Mar 2020 12:52:47 -0400

On Mar 27, 2020, at 12:42 PM, Victor Kuarsingh <[email-masked]> wrote:
> Where it is true that "rich" companies can send more people to the meetings, given we limit affiliation to no more than 2 for voting members, businesses quickly reach their limit (seeding of attendees).  I would actually be more worried about people gaming the system in general.

As someone recently pointed out, the more people you push into the pool, the more likely you are to get both of those nomcom seats.  I’ve volunteered for nomcom pretty much every time it’s come up in the past decade at least, and never been selected.  The chances for individuals are fairly sparse; for companies with many attendees it’s hard _not_ to get at least one person on nomcom.

Personally I think we should have a limit of one voting member per company.

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 27 Mar 2020 16:53:15 +0000

Hiya,

On 27/03/2020 16:42, Victor Kuarsingh wrote:
> businesses quickly reach their limit (seeding of attendees).  

I guess perceptions on that differ. IMO if it is the case
that many nomcoms have at least one company who reaches
the 2-person limit, then that indicates a problem with the
panel of volunteers and hence with our eligibility
criterion. (I've not checked yet if that's the case
though.) Other people, also reasonably, likely consider
the at-most-2 rule as being a sufficient mitigation and
then don't find the make up of the typical volunteer panel
problematic.

> I would

I'm willing to go try figure out numbers, but my own
expectation before doing that is that we won't be enabling
any significant gaming. People's worries on this do need to
be addressed though, which I guess will require analysis
of the data and proposed new criteria. (I do plan to do
that but it won't be this week.)

> venue to be quite valuable (IMO).

I think everyone I've heard so far has agreed with your
last para.

Cheers,
S.

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 27 Mar 2020 17:16:14 +0000

>Where it is true that "rich" companies can send more people to the meetings, given we limit affiliation to no more than 2 for voting members, businesses quickly reach their limit (seeding of attendees).  I would actually be more worried about people gaming the system in general

I don’t think that this is right.  If you send 20 people whose only purpose is to get into the nomcom pool, then you are more likely to get one of them selected by the random process.

---

From: Michael StJohns <[email-masked]>
Date: Fri, 27 Mar 2020 13:41:56 -0400

On 3/27/2020 12:53 PM, Stephen Farrell wrote:
> though.)

Last nomcom - Ericsson had 2 and Futurewei had 2.

Previous Nomcom - Huawei had 2, Ericsson has 1, Cisco had 1.

Previous - Huawei 1, Cisco 1, Ericsson 1

Previous Huawei 2, Cisco 1,

Previous Huawei 2, Cisco 2, Ericsson 2

> Other people, also reasonably, likely consider

The table below is the one I came up with using Excel.   Each of the 
calculated cells is =BINOMDIST(B$18,10,$A19,FALSE)  where B$18 is the 
number of "wins" or nomcom members, 10 is the total number of nomcom 
members, $A19 is the percentage (expressed as a decimal value .01 == 1%) 
which gives you the percentage of having exactly N wins given M percent 
of the pool.  The value in the >=2 column is the =SUM() of the 2-10 
columns to give you the percentage of at least 2 members (or in our case 
exactly 2 members).

The last row was based on one of the nomcoms where a company had 15 of 
202 members of the volunteer pool.

I didn't show it, but if you have 20% of the pool, you have ~90% chance 
of having at least one member.

> 	Number of Nomcom Members
> Percent of Volunteer Pool 	0 	1 	2 	3 	4 	5 	6 	7 	8 	9 	10 	>= 2
> 1.0% 	90.4% 	9.1% 	0.4% 	0.0% 	0.0% 	0.0% 	0.0% 	0.0% 	0.0% 	0.0% 
> 0.0% 	0.4%
> 5.0% 	59.9% 	31.5% 	7.5% 	1.0% 	0.1% 	0.0% 	0.0% 	0.0% 	0.0% 	0.0% 
> 0.0% 	8.6%
> 10.0% 	34.9% 	38.7% 	19.4% 	5.7% 	1.1% 	0.1% 	0.0% 	0.0% 	0.0% 	0.0% 
> 0.0% 	26.4%
> 15.0% 	19.7% 	34.7% 	27.6% 	13.0% 	4.0% 	0.8% 	0.1% 	0.0% 	0.0% 	0.0% 
> 0.0% 	45.6%
> 20.0% 	10.7% 	26.8% 	30.2% 	20.1% 	8.8% 	2.6% 	0.6% 	0.1% 	0.0% 	0.0% 
> 0.0% 	62.4%
> 25.0% 	5.6% 	18.8% 	28.2% 	25.0% 	14.6% 	5.8% 	1.6% 	0.3% 	0.0% 	0.0% 
> 0.0% 	75.6%
> 30.0% 	2.8% 	12.1% 	23.3% 	26.7% 	20.0% 	10.3% 	3.7% 	0.9% 	0.1% 	0.0% 
> 	0.0% 	85.1%
> 35.0% 	1.3% 	7.2% 	17.6% 	25.2% 	23.8% 	15.4% 	6.9% 	2.1% 	0.4% 	0.1% 
> 0.0% 	91.4%
> 7.4% 	12.5% 	27.0% 	28.1% 	18.8% 	9.0% 	3.3% 	1.0% 	0.2% 	0.0% 	0.0% 
> 0.0% 	60.6%
>

---

From: Victor Kuarsingh <[email-masked]>
Date: Fri, 27 Mar 2020 13:52:45 -0400

Rich,

I think you missed my point.  You are correct, if you (rich company) seed
folks, you are likely to get one  or two into the selection, but no more
than 2 is what I am saying.    The Nomcom has 10 voting members as well as
a number of liaisons and the chair which are there to guide and manage the
process to be fair and equitable.

I am also not denying we should address the issue of rich companies seeding
attendees, but with respect to Nomcom challenges we face, I am not sure
that's the most important and impactful item to address as we already
have checks and balances in place to mitigate that component.

regards,

Victor K

On Fri, Mar 27, 2020 at 1:16 PM Salz, Rich <[email-masked]> wrote:

> *>*Where it is true that "rich" companies can send more people to the
> actually be more worried about people gaming the system in general
>
>
>
> I don’t think that this is right.  If you send 20 people whose only
> purpose is to get into the nomcom pool, then you are more likely to get one
> of them selected by the random process.
>
>
>

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 27 Mar 2020 17:54:44 +0000

Okay, you think the two-max is sufficient protection. I disagree (and Mike’s spreadsheet numbers are concerning), but I also acknowledge that you have much experience in nomcom, and I have none.  Tnx.

---

From: Michael StJohns <[email-masked]>
Date: Fri, 27 Mar 2020 14:16:29 -0400

On 3/27/2020 1:54 PM, Salz, Rich wrote:
>
> Okay, you think the two-max is sufficient protection. I disagree (and 
> Mike’s spreadsheet numbers are concerning), but I also acknowledge 
> that you have much experience in nomcom, and I have none.  Tnx.
>
>
What Victor discounts is the impact of a company or organization having 
an ability to influence most Nomcoms even with different people each and 
every year.   In some cases, there are actually 3 nomcom members from a 
company - e.g. also a liaison.  While liaisons don't vote on 
nominations, they can speak and influence other members, they have the 
inside knowledge, and they CAN vote on everything else (there are 8 
"non-voting" members on the current Nomcom, and there is actually no 
limit on the number of observers that the chair can appoint).

And I've had experience with 4 Nomcoms - one as chair.   I believe this 
has been an issue for a while, and will be even worse if we go to a 
model that allows eligibility for remote participants without making 
changes to this one specific item.

Later, Mike

---

From: Victor Kuarsingh <[email-masked]>
Date: Fri, 27 Mar 2020 16:02:41 -0400

Rich,

Well, for clarity, I think the two-person max is adequate for now
considering we also have a full body which includes other members that help
manage the process.  That said, I would be fine with one person per
affiliation as well if that's what the community thought reasonable to set
and enforce.

regards,

Victor K

On Fri, Mar 27, 2020 at 1:54 PM Salz, Rich <[email-masked]> wrote:

> Mike’s spreadsheet numbers are concerning), but I also acknowledge that you
> have much experience in nomcom, and I have none.  Tnx.
>
>
>

---

From: "Bron Gondwana" <[email-masked]>
Date: Sat, 28 Mar 2020 17:02:20 +1100

On Sat, Mar 28, 2020, at 04:16, Salz, Rich wrote:
> *>*Where it is true that "rich" companies can send more people to the meetings, given we limit affiliation to no more than 2 for voting members, businesses quickly reach their limit (seeding of attendees). I would actually be more worried about people gaming the system in general

> I don’t think that this is right. If you send 20 people whose only purpose is to get into the nomcom pool, then you are more likely to get one of them selected by the random process. 

The easiest way to solve this (if it was considered a serious risk) would be to limit the number of candidates from a particular company as well - either first-come-first-served or allow people to withdraw from the nomcom candidate list until the company gets down to the limit. If we said "no more than 10 people per company" or "no more than 10% of the candidate pool per company" (along with the existing 2 selected per company rule) that would counter the balance.

On the other hand if a company sends 20 people and they do useful work, then we're getting a lot of valuable contribution.

Bron ( unlikely to send 10 people from Fastmail any time soon... so I guess I'm suggesting rules which increase MY chance of being selected for the nomcom, yo )

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 [email-masked]

---

From: S Moonesamy <[email-masked]>
Date: Sat, 28 Mar 2020 02:07:38 -0700

Hi Victor,
At 10:52 AM 27-03-2020, Victor Kuarsingh wrote:
>I think you missed my point.  You are correct, if you (rich company) 
>seed folks, you are likely to get one  or two into the selection, 
>but no more than 2 is what I am saying.    The Nomcom has 10 voting 
>members as well as a number of liaisons and the chair which are 
>there to guide and manage the process to be fair and equitable.
>
>I am also not denying we should address the issue of rich companies 
>seeding attendees, but with respect to Nomcom challenges we face, I 
>am not sure that's the most important and impactful item to address 
>as we already have checks and balances in place to mitigate that component.

What are the checks and balances which prevent a few companies from 
exercising undue influence in the process?

According to the process rules, "a method is unbiased if no one can 
influence its outcome in favor of a specific outcome".  Is it 
possible to bias the selection method specified in RFC 3797?

Regards,
S. Moonesamy

---

From: "Salz, Rich" <[email-masked]>
Date: Sat, 28 Mar 2020 14:21:27 +0000

> Is it possible to bias the selection method specified in RFC 3797?
  
Absolutely:  swamp the pool with candidates.

The belief that a verifiably-random selection of candidates from a self-selected pool gets a random set of candidates is wrong.

---

From: Victor Kuarsingh <[email-masked]>
Date: Sun, 29 Mar 2020 12:54:52 -0400

SM,

> exercising undue influence in the process?

I do realize that companies whom can afford and to send many
representatives to the IETF meetings, can have those members entered into
the Nomcom pool. And I do realize across any set of folks, there can be
bias.  However, it should be noted that company affiliation is not the only
vector by which we need to watch for bias with any pool of individuals.

As for the "checks".   The ISOC President selects a chair whom they feel
will exercise the duties of running the process in a fair and unbiased
way.  That chair (or his chair advisor) can raise a red flag should bias be
detected early in the process or anytime through the process.  This is our
first line of defense.

Second, there are liaisons whom participate within the full Nomcom body
which have been selected by those bodies to watch, provide input (as
directed by the Nomcom) and report back to the various bodies by which they
represent.  That, for example in this past nomcom, would have included the
IESG, IAB, ISOC board, LLC and IETF Trust.   Each of those bodies have an
opportunity to challenge the process should a flag be raised.

Further, the Nomcom only selects a candidate which then is reviewed and
approved by the confirming bodies (another opportunity to catch issues if
they exist).  The ISOC BoT approves the IAB slate, the IAB approves the
IESG selections and the IESG approves the Trust and LLC selections.  After
all this, the community also has a recall process which can be exercised.

There is no perfect system and no matter what we do , we need to watch for
any bias.  Corporate bias is only one type.  I am not saying we should
ignore it, but we also need to be balanced and focus on any vector which
can drive bias into the process.   However, there is quite a bit of
oversight and multiple opportunities to call things out if that should be
needed.

> possible to bias the selection method specified in RFC 3797?

Well I think this is accurate with respect to the fact that we do not know
the outcome of the random numbers which are used to run the seed (with
sufficient entropy based on selected random number sources).  For any
number list, we do not know in advance what the outcome would be (which
numbered positions in the list will be selected).  The process is built
such that we all participate as individuals.

The list of potential nomcom members is also announced in advance, so the
community can review and challenge the list.

regards,

Victor K

---

From: Victor Kuarsingh <[email-masked]>
Date: Sun, 29 Mar 2020 13:13:22 -0400

On Fri, Mar 27, 2020 at 2:16 PM Michael StJohns <[email-masked]>
wrote:

> What Victor discounts is the impact of a company or organization having an
> ability to influence most Nomcoms even with different people each and every
> year.
>

I have not discounted it, but as noted in other emails, it's not the only
vector that can drive influence.  We need to be mindful and watch out for
all ways in which bias can enter into the system.  I think having a process
that assumes this can occur, and manages for that is the way to deal with
it (versus assuming we can remove all vectors that can create influence or
bias).

As I said before, if the community feels that one person from each
affiliation is appropriate, I am ok with that too (but I don't think single
that will really fix our issue).

 In some cases, there are actually 3 nomcom members from a company - e.g.
also a liaison.  While liaisons don't vote on nominations, they can speak
and influence other members, they have the inside knowledge, and they CAN
vote on everything else (there are 8 "non-voting" members on the current
Nomcom, and there is actually no limit on the number of observers that the
chair can appoint).

Its possible we had different experiences.

> has been an issue for a while, and will be even worse if we go to a model
> that allows eligibility for remote participants without making changes to
> this one specific item.
>

I do agree that allowance of remote attendees can create a problem we need
to watch out for (low. bar to seed more people into the pool).

regards,

Victor K

---

From: Michael Richardson <[email-masked]>
Date: Sun, 29 Mar 2020 17:16:14 -0400

> Which turns out not to be the best protection against gaming. The math is
    > pretty clear - with a large enough contribution (eligible volunteers) to the
    > pool, a large organization can pretty much guarantee capture of 2 slots.   
    > Over the years we've generally had 3-4 "large" organizations that have most
    > years collectively provided the majority of voting members and sometimes a
    > few more.

So, you are trying to make the rules harder, and we are trying to expand the
pool of volunteers.  While I generally agree with your goals and even your
proposal, I don't think we can do this now.

The proposed rules are a superset of the existing rules.
By expanding the pool of people, and if that can include more people from not
so large organization, then we will also have an effect.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Sun, 29 Mar 2020 17:19:54 -0400

> affiliation as well if that's what the community thought reasonable to set

I would also welcome this change, but I don't want to have the community
rathole on this topic rather than on the extended eligibility changes.

---

From: Michael StJohns <[email-masked]>
Date: Sun, 29 Mar 2020 17:29:50 -0400

> proposal, I don't think we can do this now.

Um.  No.   And apples (rules) are not the same as oranges (pool of 
volunteers) which are not the same as bananas (the actual composition of 
the Nomcom).

As it currently stands, we have a system where the volunteers for the 
pool are a) real people, and b) have paid some cost to be put in the 
pool.   We also have a system where c) Any company that comes up with 
sufficient money can have have a very high probability of having 1 or 2 
members of the Nomcom each and every time.

I'd be happy if we basically said that a) any real person can volunteer, 
b) that they can do it without paying a cost, and c) that a company 
can't buy Nomcom slots simply by having all of their employees volunteer.

I think its probably going to be more along the lines of  a) any real 
person with some continuing relationship with the IETF > 2 years can 
volunteer, b) that we'll have some objective criteria for a), and c) if 
we don't pay attention to fixing the pool packing problem we will 
continue to have large organizations dominate the membership of the IETF 
regardless of the expansion of the pool.

> so large organization, then we will also have an effect.

Please, please, please look at the math.   Expanding the pool does 
nothing to change the outcome.  The people might vary - the 
organizations - statistically speaking - won't.  Or do you think that by 
expanding the pool, the percentage of people in the pool from large 
organizations will decrease?

Later, Mike

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 30 Mar 2020 10:49:17 +1300

Mike,

On 30-Mar-20 10:29, Michael StJohns wrote:
...
> regardless of the expansion of the pool.

True. But can we please separate the issues?

If you want to draft an RFC3933 experiment to change the cap from
2 to 1 per employer, that would be fine of course. But running two
experiments in parallel would not be good science.

Stay well,
    Brian

---

From: S Moonesamy <[email-masked]>
Date: Sun, 29 Mar 2020 23:15:52 -0700

Hi Victor,
At 09:54 AM 29-03-2020, Victor Kuarsingh wrote:
>Well I think this is accurate with respect to the fact that we do 
>not know the outcome of the random numbers which are used to run the 
>seed (with sufficient entropy based on selected random number 
>sources).  For any number list, we do not know in advance what the 
>outcome would be (which numbered positions in the list will be 
>selected).  The process is built such that we all participate as individuals.
>
>The list of potential nomcom members is also announced in advance, 
>so the community can review and challenge the list.

I'll comment on the second part of your reply.

There is an assumption that everyone participates as an 
individual.  That shouldn't be a problem and there shouldn't be an 
affiliation limit if people participate as individuals.  The 
historical data [1] shows that two companies can get 40% of the 
NomCom seats.  Once a company get to that point, it has the ability 
to defeat what was described as checks and balances in the first part 
of your reply.

There was a message from Rich at 
https://mailarchive.ietf.org/arch/msg/ietf/gDQMyE3fVfZ-0KGcEqVWlw_oSrU 
about whether a self-selected pool will give a random set of members.

Regards,
S. Moonesamy

1. 
https://mailarchive.ietf.org/arch/msg/eligibility-discuss/brVGJ6c8C0McxeCX_rlI7FkecJA/