Subject: [Eligibility-discuss] "Additional Criteria"

From: Ted Hardie <[email-masked]>
Date: Thu, 30 Jul 2020 07:47:12 -0700

First, let me apologize by saying that I was apparently not subscribed to
this list and have not yet read the archives.  I wanted to post today in
advance of that in part to explain my objections at the gendispatch meeting
today.

The primary one is structural.  This document proposes that it be treated
as a process experiment (rather than some other type of fix), but its
design is not a very good experiment, as it is experimenting along multiple
paths at once.

Path 1:  experiments by eliminating meeting attendance as a path to
eligibility for the duration of the period of online-only meetings.

Path 2: experiments by introducing eligibility for WG Chair and Secretary
roles.  These probably have a high overlap with currently eligible
participants, though "BoF Chair" could introduce eligibility early to some
participants (when a proponent is selected as one of the BoF Chairs.

Path 4: Experiments by retaining eligibility for IESG or IAB members for 5
years, no matter what more recent level of participation they have had.  It
also offers eligibility for those who have "been appointed to a formal role
by the IESG or IAB, within the last 5 years."  Formal role is not defined,
but if this includes IANA experts and IAB-named liaisons, there are
individuals whose specialized knowledge is very useful but does not reflect
IETF participation in the same way attendees have it.  Some of the
individuals who serve in these roles have not attended an IETF or met its
current pool of participants for many years.

Path 5: Experiments by adding eligibility to those who author WG-adopted
drafts.  This both speeds eligibility for some individuals (who may have a
WG adopt their draft at their first meeting) and may retain or restore it
for those who are not currently involved.  This is because it is common for
-bis drafts to list as authors the authors of the original RFC.

Assessing which ones of these worked well at the end of the experiment will
be difficult, as will be the assessment of the elimination of the current
participation metric. They each represent changes to the tie between
participation and the opportunity to select leaders and it is not clear
that some of these roles actually involve familiarity with the pool of
volunteers for leadership positions, which is a critical aspect of the
nomcom's role.

At the meeting today, I suggested a dispatch to a working group for this
document, as I believe the long term issues need more discussion to forge
community consensus on this.

Several people pointed out that we will need to handle this much more
quickly, given the current pandemic.  I agree.  I think for that we could
run with a much more simple policy, possibly even within the process
experiment mechanism.  The heart of that would be something like this:

Beginning with IETF 109, for any plenary meeting in which the IETF had no
in-person participation, any registrant for the meeting will be treated as
an in-person registrant for the purposes of the NomCom rules. The
experiment for this will terminate when two in-person meetings have been
held after IETF 109 or at the third meeting in 2021, whichever is earlier.

Like for the "additional criteria" proposal, an assessment after the fact
would be useful, but this is both a much smaller change from our current
rules and seems likely to result in a larger pool.  Given the number of
meetings in the longest possible period, there is some risk of gaming this,
but the other nomcom limits and the limited time of the experiment mitigate
this to some extent.

Again, my apologies for not participating up to this point and for
following up on today's discussion prior to reviewing the archives.  I hope
to review them soon.

regards,

Ted Hardie

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 30 Jul 2020 16:04:20 +0100

Hi Ted,

Thanks for bringing this up. I don't think anyone had made
that specific concrete proposal before so you're not
retreading well-trodden ground.

> design is not a very good experiment, as it is experimenting along multiple

FWIW, I don't agree. I think the complexity here is not in
the mechanism(s) set out in the draft, nor in the one you
propose, should we go that way, but rather in evaluating
the effects of the change. I think that is as complex in
either case, and in one sense easy ("how many nomcom voting
members were picked during the experiment that [would|would
not] have been eligible before?") but in another sense is
very hard ("Was there a [good|bad] effect of the change?").

> by the IESG or IAB, within the last 5 years."  Formal role is not defined,

That's fair and we should fix it.

> -bis drafts to list as authors the authors of the original RFC.

Just to be clear: I think more discussion of points related to
the specific paths is still needed. Assuming we don't plump for
Ted's alternative below, maybe a thread on each might help make
progress on those?

> held after IETF 109 or at the third meeting in 2021, whichever is earlier.

Like I said in the meeting jabber, I could live with that but
much prefer what's in the draft as that goes at least a little
down the path of eligibility for people who are habitually
remote for whatever reason, and I do think we very much need
to take that step in any case.

Cheers,
S.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 11:17:14 -0400

On Jul 30, 2020, at 10:47 AM, Ted Hardie <[email-masked]> wrote:
> Beginning with IETF 109, for any plenary meeting in which the IETF had no in-person participation, any registrant for the meeting will be treated as an in-person registrant for the purposes of the NomCom rules. The experiment for this will terminate when two in-person meetings have been held after IETF 109 or at the third meeting in 2021, whichever is earlier.
> 

We’ve had long and far-ranging discussions on this topic, Ted. Right now, we use in-person attendance as an proxy for “meaningful participation.” Going forward, even after COVID, we have other crises in the world, and requiring everyone to fly halfway around the globe to attend IETF is incompatible with other goals, and disenfranchises anyone who can’t afford to do that and can’t get sponsored to do that. This significantly impacts IETF diversity. This isn’t theoretical—just look at who shows up every year.

The problem with your proposal is that it just kicks the can down the road. We aren’t actually making progress on the very real problem I just described. There is a great deal of pressure to solve this problem right now. Once the current crisis is over, that pressure will diminish. I think it’s a poor choice to deliberately remove that pressure, so I am not in favor of continuing to use attendance as a proxy for meaningful participation.

What we should be doing instead is figuring out what we think “meaningful participation” means, and then coming up with ways to measure that.

---

From: Ted Hardie <[email-masked]>
Date: Thu, 30 Jul 2020 12:28:38 -0700

Hi Ted,

On Thu, Jul 30, 2020 at 8:17 AM Ted Lemon <[email-masked]> wrote:

> We’ve had long and far-ranging discussions on this topic, Ted. Right now,
> we use in-person attendance as an proxy for “meaningful participation.”
> Going forward, even after COVID, we have other crises in the world, and
> requiring everyone to fly halfway around the globe to attend IETF is
> incompatible with other goals, and disenfranchises anyone who can’t afford
> to do that and can’t get sponsored to do that. This significantly impacts
> IETF diversity. This isn’t theoretical—just look at who shows up every year.
>

I apologize for not being part of the wide-ranging discussions up until
now.  But I think there's a real risk in evaluating eligibility by using
these particular measures of "meaningful" for meaningful participation.
Looking at the current draft, I see what look to be a number changes that
bias for providing eligibility to those who are already known to the
community:  paths 2,4, and 5 all contain elements of that type.  While
there are some outliers in both BoF chairs and WG authors that may have new
participants, the methods we use to select I* members, WG chairs, and
document editors mostly run in the other direction.  Combined with the
elimination of the path via meeting participation by the circumstances of
the pandemic, I think that is more likely to reduce diversity over the
course of the experiment than to increase it.

Put differently, I think the Nomcom criteria now look to create a simple
metric that maps to  "volunteers know the community" rather than
"volunteers are known to the community".  This draft switches that up, and
I think that changing that is a significant difference.  As it stands now,
folks can put their names in the hat to serve on the Nomcom even if they
have significant differences with the current powers-that-be.  I'm not sure
that would be the case with these new criteria; it is much more certain
that the well-networked stay in than the unknown or unhappy have a voice.
With the number of changes in this doc, I'm also not sure we'd be able to
measure that effect over this period of time, especially given the language
in path one.

The problem with your proposal is that it just kicks the can down the road.

I made that proposal only because I was told creating a working group and
discussing it post-charter would take too long.  I also believe that that
simple proposal would include a broader variety of potential NomCom
participants than the current draft would.   There are some risks to it,
among them:

* If the off-line meeting streak goes on a long time, the extent to which
volunteers from all-remote meetings are familiar with the community will
not be the same as those who attended 3 of 5 physical meetings.

* There is some increased risk of gaming the NomCom, because the costs to
do so go down significantly.

but I think these are manageable enough for us to have the broader
discussion in a WG while that policy is in place. The safety valve of a
date by which it ends provides enough pressure not just to talk endlessly.

I agree with the goal of increasing diversity, but I ask you to consider
carefully the extent to which additional methods which require someone be
already acceptable to the incumbents provides that.

regards,

Ted Hardie

We aren’t actually making progress on the very real problem I just
> described. There is a great deal of pressure to solve this problem right
> now. Once the current crisis is over, that pressure will diminish. I think
> it’s a poor choice to deliberately remove that pressure, so I am not in
> favor of continuing to use attendance as a proxy for meaningful
> participation.
>
> What we should be doing instead is figuring out what we think “meaningful
> participation” means, and then coming up with ways to measure that.
>
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 15:54:58 -0400

On Jul 30, 2020, at 3:28 PM, Ted Hardie <[email-masked]> wrote:
> I agree with the goal of increasing diversity, but I ask you to consider carefully the extent to which additional methods which require someone be already acceptable to the incumbents provides that.

Fortunately that wasn’t what I suggested at all. What I suggested was measuring meaningful participation. I am really uneasy about using things like I* membership and draft authorship that way.

What I would like to see measured are things like reviews, meaningful engagement on the mailing list (this is obviously hard), note-taking, etc. Stuff that shows that you actually took part in the IETF, and didn’t just pay an entry fee and show up physically.

Ideally, these measures would only rely on judgment in the sense that attempts to abuse them could result in a judgment call being required; absent abuse, the presumption is that whether anyone else thought what you had to say was interesting, you participated, and are eligible. Trying to get this perfect is an np-complete problem, but I think we can get a good enough approximation that it would be better than the relatively crappy approximation we have been using.

It’s also a lot more work.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 30 Jul 2020 15:51:48 -0500

At the risk of chiming in and causing trouble ...

(Let me say that this particular thread seems helpful - Ted and Ted, thanks
for that)

On Thu, Jul 30, 2020 at 2:55 PM Ted Lemon <[email-masked]> wrote:

> Fortunately that wasn’t what I suggested at all. What I suggested was
> measuring meaningful participation. I am really uneasy about using things
> like I* membership and draft authorship that way.
>
> What I would like to see measured are things like reviews, meaningful
> engagement on the mailing list (this is obviously hard), note-taking, etc.
> Stuff that shows that you actually took part in the IETF, and didn’t just
> pay an entry fee and show up physically.
>

What would the experimental criteria look like, if "active participation",
not measured through a proxy, was the only criteria, and the goal was NOT
to find new ways of declaring the same people qualified that were qualified
under 3-of-5, whether in-person or virtual?

ISTM that draft authors, draft editors, draft reviewers, text contributors,
document shepherds, and BOF proponents would all qualify. Who else?

> Ideally, these measures would only rely on judgment in the sense that
> attempts to abuse them could result in a judgment call being required;
> absent abuse, the presumption is that whether anyone else thought what you
> had to say was interesting, you participated, and are eligible. Trying to
> get this perfect is an np-complete problem, but I think we can get a good
> enough approximation that it would be better than the relatively crappy
> approximation we have been using.
>

The goal (in the soccer/football sense) is the quality of the
Nomcom-reviewed bodies, viewed over time, and I'm not sure how you would
know the experimental criteria were objectively better or worse for a while
(given that you could have had a better pool of volunteers and picked the
least qualified at random).

I'd suggest that we try to do the right thing, and engineer the process if
it needs to be better.

True.

Best,

Spencer

-
> Eligibility-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/eligibility-discuss
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 17:04:24 -0400

On Jul 30, 2020, at 4:51 PM, Spencer Dawkins at IETF <[email-masked]> wrote:
> ISTM that draft authors, draft editors, draft reviewers, text contributors, document shepherds, and BOF proponents would all qualify. Who else?

Anybody who made an on-topic comment on a working group mailing list discussion (this one is a bit hard, but if the presumption is that a comment is on-topic, I think you get pretty close). Anybody who offered feedback on a working group draft or working group candidate draft. Anybody who scribes for a meeting (jabber or notes). I have suggested adding a formal review tracker for working groups; anybody who did a review there would count. These are examples I can think of; I’m sure there are more.

> The goal (in the soccer/football sense) is the quality of the Nomcom-reviewed bodies, viewed over time, and I'm not sure how you would know the experimental criteria were objectively better or worse for a while (given that you could have had a better pool of volunteers and picked the least qualified at random). 

Why not just evaluate the qualifications of the inputs to the pool, rather than the outputs of the selection process?

> I'd suggest that we try to do the right thing, and engineer the process if it needs to be better. 

I don’t disagree. But are you saying that counting attendees is good enough? I think several people (me included) have argued that it is not—that it excludes people.

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Jul 2020 17:13:10 -0400

--On Thursday, July 30, 2020 15:54 -0400 Ted Lemon
<[email-masked]> wrote:

>> I agree with the goal of increasing diversity, but I ask you
>> to consider carefully the extent to which additional methods
>> which require someone be already acceptable to the incumbents
>> provides that.
> 
> Fortunately that wasn't what I suggested at all. What I
> suggested was measuring meaningful participation. I am really
> uneasy about using things like I* membership and draft
> authorship that way.
> 
> What I would like to see measured are things like reviews,
> meaningful engagement on the mailing list (this is obviously
> hard), note-taking, etc. Stuff that shows that you actually
> took part in the IETF, and didn't just pay an entry fee and
> show up physically.

Agreed.  But I also agree with Ted Hardie that criteria that
reinforce ending up with a Nomcom dominated by people who are
acceptable to -- and, from a diversity standpoint, similar to--
the incumbents are undesirable.

> Ideally, these measures would only rely on judgment in the
> sense that attempts to abuse them could result in a judgment
> call being required; absent abuse, the presumption is that
> whether anyone else thought what you had to say was
> interesting, you participated, and are eligible. Trying to get
> this perfect is an np-complete problem, but I think we can get
> a good enough approximation that it would be better than the
> relatively crappy approximation we have been using.
> 
> It's also a lot more work.

Indeed.  It also means dropping --or vastly lowering the
importance of-- something I think Brian et al. considered their
key criterion, which was some objective measure that could
easily be checked by the secretariat.

And...

--On Thursday, July 30, 2020 15:51 -0500 Spencer Dawkins at IETF
<[email-masked]> wrote:

>...
> What would the experimental criteria look like, if "active
> participation", not measured through a proxy, was the only
> criteria, and the goal was NOT to find new ways of declaring
> the same people qualified that were qualified under 3-of-5,
> whether in-person or virtual?
> 
> ISTM that draft authors, draft editors, draft reviewers, text
> contributors, document shepherds, and BOF proponents would all
> qualify. Who else?
>...

Well, I think that more or less has to be it.  However, it means
we have to accept the risk of sock puppets and the risk of
people submitting meaningless (or worse) drafts because their
companies thing that submitting drafts (not matter their content
or impact) and getting onto the Nomcom are Good Things that
should be rewarded.  It may also include people whose idea of a
review of a technical document amounts to "I read this.  I
didn't understand it.  I did, however, find a comma out of place
on page six and a word I believe some people would find
demeaning on page 9".  

The more I think about the conversation on this list and the one
I indirectly alluded to above, the more I think that risk may be
better than the risk of a homogeneous Nomcom who are very much
like the incumbents and perhaps unlikely to select candidates
who are different from those incumbents.

I'd like criteria like that even more if there were some
effective challenge procedure, as suggested above, but, with
subject criteria, I fear that could easily degenerate into very
unpleasant name-calling and character assassination.

YMMD.
   john

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 17:15:00 -0400

We already have people submitting drafts that are essentially DoS attacks. :]

> On Jul 30, 2020, at 5:13 PM, John C Klensin <[email-masked]> wrote:
> 
> 
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 30 Jul 2020 16:15:06 -0500

Hi, Ted (Lemon),

On Thu, Jul 30, 2020 at 4:04 PM Ted Lemon <[email-masked]> wrote:

> contributors, document shepherds, and BOF proponents would all qualify. Who
> else?
>
>
> Anybody who made an on-topic comment on a working group mailing list
> discussion (this one is a bit hard, but if the presumption is that a
> comment is on-topic, I think you get pretty close). Anybody who offered
> feedback on a working group draft or working group candidate draft. Anybody
> who scribes for a meeting (jabber or notes). I have suggested adding a
> formal review tracker for working groups; anybody who did a review there
> would count. These are examples I can think of; I’m sure there are more.
>
> Why not just evaluate the qualifications of the inputs to the pool, rather
> than the outputs of the selection process?
>

Mostly because we're guessing at what qualifications of the inputs to the
pool is easier than measuring the outputs of the selection process, so
we're (once again) using a proxy that we can measure to stand in for what
we want to measure. But that's a different conversation, and we can punt it
down the road.

We need to do SOMETHING really soon, and we're talking about what SOMETHING
might be. At some point, we need to figure out if we did the RIGHT thing,
but figuring out how to evaluate the experiment needn't gate us from
starting what is a time-critical process really soon.

> I don’t disagree. But are you saying that counting attendees is good
> enough? I think several people (me included) have argued that it is
> not—that it excludes people.
>

I agree with the theory that we're counting attendees because they're easy
to count, and hoping for the best, isn't a great plan.

What would have been clearer, would be if I said, "I'd suggest that we try
to do the right thing IN THIS EXPERIMENT, and engineer the process if it
needs to be better."

But sometimes I am unclear :-)

Best,

Spencer

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Jul 2020 17:31:40 -0400

> We already have people submitting drafts that are essentially

I know.  And we have people doing reviews that are a waste of
good bits as well as the time of authors who need to respond.
And I'm fairly sure we have both coming from people who meet the
3/5 f2f criterion.   I think we may be saying the same thing,
which is that it would be better to accept some risk of attacks
by people/entities we can't authenticate (as human) --especially
when those attacks are already possible from people who show up
and pay money-- than to continue with Nomcoms that may be too
homogeneous now and likely to get more so over time.

   john

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 30 Jul 2020 17:38:00 -0400

It seems very strange to me that in trying to get the people who are 
active in the IETF, and have visibility to the IETF and an understanding 
of what may be likely to be good for the organization, the proposal 
below does not include WG chairs.

I realize that WG chairs are appointed by earlier ADs.  But they are 
clearly active, willing to work, and have visiblity and perspective.

Yours,
Joel

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 30 Jul 2020 21:48:22 +0000

Very strong +1 to this.

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 Jul 2020 17:54:32 -0400

--On Thursday, July 30, 2020 17:38 -0400 "Joel M. Halpern"
<[email-masked]> wrote:

> It seems very strange to me that in trying to get the people
> who are active in the IETF, and have visibility to the IETF
> and an understanding of what may be likely to be good for the
> organization, the proposal below does not include WG chairs.
> 
> I realize that WG chairs are appointed by earlier ADs.  But
> they are clearly active, willing to work, and have visiblity
> and perspective.

Ted and Ted can (and probably will) speak for themselves, but I
have no objection to including WG Chairs... and even I'm not
paranoid enough to worry about ADs selecting only those WG
Chairs who will volunteer for the Nomcom and they support
incumbents or people just like them.    

That said, the criterion may be largely redundant.  The number
of WG Chairs we have who have not written or edited drafts, done
reviews, etc., has got to be rather small (as is the number of
WG Chairs who do not qualify under 3/5 criteria although that
might change).

    john

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 19:03:09 -0400

On Jul 30, 2020, at 5:38 PM, Joel M. Halpern <[email-masked]> wrote:
> I realize that WG chairs are appointed by earlier ADs.  But they are clearly active, willing to work, and have visiblity and perspective.

They most likely also do other things that would qualify them…

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 19:04:46 -0400

> They most likely also do other things that would qualify them…

E.g., if a WG chair doesn’t participate on the mailing list to the point where that would qualify them, then they probably aren’t actually being a chair, and shouldn’t get credit because they happen to have that label either.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 31 Jul 2020 11:17:30 +1200

This is going to be a portmanteau response since I was asleep during most of this thread:
On 31-Jul-20 02:47, Ted Hardie wrote:
> First, let me apologize by saying that I was apparently not subscribed to this list and have not yet read the archives.  I wanted to post today in advance of that in part to explain my objections at the gendispatch meeting today.

Ted, nobody expects you to be omniscient and omnipresent! No need to apologise.
... 
> Path 1:  experiments by eliminating meeting attendance as a path to eligibility for the duration of the period of online-only meetings.

Yes, I suppose that is an unintended side effect. Now that we have two meetings with formalised on-line participation, we could extend Path 1 to include them (if we believe that is the right thing to do).
...

> Assessing which ones of these worked well at the end of the experiment will be difficult, as will be the assessment of the elimination of the current participation metric. They each represent changes to the tie between participation and the opportunity to select leaders and it is not clear that some of these roles actually involve familiarity with the pool of volunteers for leadership positions, which is a critical aspect of the nomcom's role.

True (and it certainly applies to the old criterion). Also, there may be feedback loops affecting people's behaviour. As was discussed earlier, if we included review team membership, there might be a lot of new volunteers to join review teams. And so on. It's very hard to eliminate that problem, or to predict the unintended side effects.
...

> Beginning with IETF 109, for any plenary meeting in which the IETF had no in-person participation, any registrant for the meeting will be treated as an in-person registrant for the purposes of the…

I don't really see how that addresses the wider problem, which is that meeting participation alone does not demonstrate that a person is deeply familiar with IETF practice and has a record as a contributor (i.e. is a doer, not just a goer). It seems more like another stop-gap. If we end up with an IETF that decides it only needs one face-to-face meeting a year, which I regard as a very plausible future following the COVID-19 natural experiment, the whole discussion would be set back to zero. The criteria in the draft are ones that would work in *any* future scenario for plenary meetings, and that's a feature, not a bug.

So IMHO there is a very good reason to experiment with new criteria during this period when the old criterion is ineffective.

On 31-Jul-20 03:04, Stephen Farrell wrote:
...
> I think the complexity here is not in
> the effects of the change.

Exactly. The mechanisms were all chosen because thay can be applied automatically by the Secretariat; but the evaluation is non-trivial (more than just glancing at some Venn diagrams) and that part needs more thought, whichever approach we adopt for an experiment.

...
>>   Formal role is not defined,
> 

Yes.

On 31-Jul-20 03:17, Ted Lemon wrote:
...
> What we should be doing instead is figuring out what we think “meaningful participation” means, and then coming up with ways to measure that.

Yes.

On 31-Jul-20 08:51, Spencer Dawkins at IETF wrote:
...
> What would the experimental criteria look like, if "active participation", not measured through a proxy, was the only criteria...

I don't understand. If the Secretariat can't measure it automatically from data in the tracker or meeting attendance records, whatever "it" is simply isn't available to us.

On 31-Jul-20 09:04, Ted Lemon wrote:
...
> Anybody who made an on-topic comment on a working group mailing list discussion (this one is a bit hard, but if the presumption is that a comment is on-topic, I think you get pretty close). Anybody…

Just to repeat what was said a while ago, several of these seem very easy to game, which is why we were persuaded to remove review team activity from an earlier draft. The problem arises if you do indeed presume that a comment is "on-topic" or that a review is useful as opposed to useless. To caricature the problem, is a comment consisting of "+1" acceptable? How many "+1"s per month would qualify me? Or reviews that simply says "Ready"?

On 31-Jul-20 09:13, John C Klensin wrote:
...
> subject[ive] criteria, I fear that could easily degenerate into very

+1

On 31-Jul-20 09:38, Joel M. Halpern wrote:
...
> below does not include WG chairs.

FWIW the draft currently includes "Has been a WG Chair or Secretary, or a BOF Chair, within the last 3 years."

    Brian

---

From: Joel Halpern Direct <[email-masked]>
Date: Thu, 30 Jul 2020 19:43:53 -0400

It seems to me that the argument "that category will qualify in other 
ways" is at best misleading.  Either people qualify on a particular 
basis or they don't.  One of my co-chairs does not write many drafts and 
is not on any of teh review teams.  Does that mean that if he is 
attending meetings remotely he should not qualify for nomcom?  That does 
not seem a sensible conclusion to me.

Put differently, of course the criteria overlap.  If we have criteria 
related to active and useful participation, people will do so in 
multiple ways.  That does not mean we should leave one out because it 
will frequently overlap with others.

Yours,
Joel

On 7/30/2020 7:03 PM, Ted Lemon wrote:
> On Jul 30, 2020, at 5:38 PM, Joel M. Halpern <[email-masked] 
> <mailto:[email-masked]>> wrote:

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 30 Jul 2020 19:45:50 -0400

Part of my problem is likely that things like "meaningful participation 
on email list" does not seem to be subject to mechanistic determination. 
  And as I understand it, we do not wnt some human being having to 
decide if each person qualifies for nomcom.

Yours,
Joel

On 7/30/2020 7:04 PM, Ted Lemon wrote:
> On Jul 30, 2020, at 7:03 PM, Ted Lemon <[email-masked] 
> <mailto:[email-masked]>> wrote:
> E.g., if a WG chair doesn’t participate on the mailing list to the point 
> where that would qualify them, then they probably aren’t actually being 
> a chair, and shouldn’t get credit because they happen to have that label 
> either.
> 
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 20:08:07 -0400

On Jul 30, 2020, at 7:43 PM, Joel Halpern Direct <[email-masked]> wrote:
> It seems to me that the argument "that category will qualify in other ways" is at best misleading.  Either people qualify on a particular basis or they don't.  One of my co-chairs does not write many drafts and is not on any of teh review teams.  Does that mean that if he is attending meetings remotely he should not qualify for nomcom?  That does not seem a sensible conclusion to me.

Does that co-chair never do any work on the mailing list?  If so, and if that co-chair is still adding value, then I agree that you’ve found a flaw in my clever plan.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 20:12:12 -0400

On Jul 30, 2020, at 7:45 PM, Joel M. Halpern <[email-masked]> wrote:
> Part of my problem is likely that things like "meaningful participation on email list" does not seem to be subject to mechanistic determination.  And as I understand it, we do not wnt some human being having to decide if each person qualifies for nomcom.

What I’m suggesting is that _any_ participation on the mailing list that doesn’t stand out as “obviously bad” would be automatically counted as “meaningful participation.”  So the work would be in excluding particularly egregious abuses, not in trying to figure out on a message-by-message basis what was good enough to meet the standard. Remember, we’re comparing this to “shows up at meetings and sits in a chair.” That’s a pretty low bar.

We might want to have some mechanism whereby people can say “this email address is mine, please credit me when you see a message go by on the list from this address.”

But the point is, this could be counted automatically, since only in exceptional cases (think “sergeant-at-arms”) would a message be excluded.

I am not suggesting that one message to any IETF mailing list would be enough to qualify, but I’m pretty sure we could come up with a machine-countable method for this.

Of course, it’s possible that we would suddenly get a lot of useless chaff on the mailing list; the question is, would we notice?

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 30 Jul 2020 20:21:13 -0400

We have enough trouble with folks sending +1 messages for "the WG should 
adopt this" with no other content.  (Or LC, or...)  I would hate to 
reward such messages in other ways.

Yours,
Joel

PS: In terms of chairs participating in email lists, it probably depends 
upon how many emails you have to send over what period of time.  A lot 
of the work is behind the scenes.   Yes, shepherd reviews and the like 
go to the list.  But as chairs we actually encourage other folks to be 
shepherds.

On 7/30/2020 8:12 PM, Ted Lemon wrote:
> On Jul 30, 2020, at 7:45 PM, Joel M. Halpern <[email-masked] 
>> Part of my problem is likely that things like "meaningful 
>> participation on email list" does not seem to be subject to 
>> mechanistic determination.  And as I understand it, we do not wnt some 
>> human being having to decide if each person qualifies for nomcom.
> 
> What I’m suggesting is that _any_ participation on the mailing list that 
> doesn’t stand out as “obviously bad” would be automatically counted as 
> “meaningful participation.”  So the work would be in excluding 
> particularly egregious abuses, not in trying to figure out on a 
> message-by-message basis what was good enough to meet the standard. 
> Remember, we’re comparing this to “shows up at meetings and sits in a 
> chair.” That’s a pretty low bar.
> 
> We might want to have some mechanism whereby people can say “this email 
> address is mine, please credit me when you see a message go by on the 
> list from this address.”
> 
> But the point is, this could be counted automatically, since only in 
> exceptional cases (think “sergeant-at-arms”) would a message be excluded.
> 
> I am not suggesting that one message to any IETF mailing list would be 
> enough to qualify, but I’m pretty sure we could come up with a 
> machine-countable method for this.
> 
> Of course, it’s possible that we would suddenly get a lot of useless 
> chaff on the mailing list; the question is, would we notice?
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 20:34:25 -0400

My experience is that chairs don’t send enough mail to their mailing lists, eg to facilitate discussion. And chairs should be actively discouraging +1 messages. It’s a problem that they do not.  But if they continue not to, those messages are still more participation than sitting on a chair. 

> On Jul 30, 2020, at 20:21, Joel M. Halpern <[email-masked]> wrote:
> 
> We have enough trouble with folks sending +1 messages for "the WG should adopt this" with no other content.  (Or LC, or...)  I would hate to reward such messages in other ways.
> 
> Yours,
> Joel
> 
> PS: In terms of chairs participating in email lists, it probably depends upon how many emails you have to send over what period of time.  A lot of the work is behind the scenes.   Yes, shepherd reviews and the like go to the list.  But as chairs we actually encourage other folks to be shepherds.
> 
>>> On Jul 30, 2020, at 7:45 PM, Joel M. Halpern <[email-masked] <mailto:[email-masked]>> wrote:

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 30 Jul 2020 20:42:31 -0400

I have never found a good way to discourage +1 messages.
I frequently write calls with "Please provide motivation and 
explanation..." or similar text.
And still, most of the responses are "+1".

The question of whether technical emails from chairs encourage or 
discourage discussion probably depends on a myriad factors.  I suspect 
many folks consider that I engage too much.

Yours,
Joel

---

From: Michael StJohns <[email-masked]>
Date: Thu, 30 Jul 2020 21:17:40 -0400

On 7/30/2020 8:12 PM, Ted Lemon wrote:
>> mechanistic determination.  And as I understand it, we do not wnt 
>> some human being having to decide if each person qualifies for nomcom.
>
> What I’m suggesting is that _any_ participation on the mailing list 
> that doesn’t stand out as “obviously bad” would be automatically 
> counted as “meaningful participation.”  So the work would be in 
> excluding particularly egregious abuses, not in trying to figure out 
> on a message-by-message basis what was good enough to meet the 
> standard. Remember, we’re comparing this to “shows up at meetings and 
> sits in a chair.” That’s a pretty low bar.
>
> We might want to have some mechanism whereby people can say “this 
> email address is mine, please credit me when you see a message go by 
> on the list from this address.”
>
Ted -

How do you propose to handle challenges?  E.g.,  "Why am I not on the 
list and what about the three comments I made on that obscure mailing 
list a year ago that obviously mattered?" or "Why is X on the list - 
they keep making stupid comments.  If I knew all I had to do was make 
stupid comments I would have made more."

Any criteria that anyone proposes that cannot be evaluated in an 
objective manner (and ideally from public records automatically by 
software) is IMNSHO a non-starter.   If you propose subjective 
requirements e.g. "participate on the mailing list to the  point where 
that would qualify them" you're going to need someone to decide :the 
point where that would qualify them" and lots of someones to handle the 
appeals and that should be part of the proposal.  So far I haven't seen 
that.

With all of these discussions, I've pretty much settled back into 
"registered (and attended) for 3 of the last 5 meetings in the mode for 
which the meeting is held excluding 107".   It's simple, it shows at 
least a little effort to participate, and we've got 109 and 110 to watch 
for possible gaming of the system since the next Nomcom won't be 
selected until after those two meetings.    I might add in "prove you're 
a unique non-virtual human" and I might track the rate of change for all 
the organizations in qualified people and report those statistics with 
an eye to revisiting the selection process statistical bias if necessary.

Later, Mike

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 22:34:58 -0400

On Jul 30, 2020, at 9:17 PM, Michael StJohns <[email-masked]> wrote:
> How do you propose to handle challenges?  E.g.,  "Why am I not on the list and what about the three comments I made on that obscure mailing list a year ago that obviously mattered?"

“That wasn’t an IETF working group mailing list.”

> or "Why is X on the list - they keep making stupid comments.  If I knew all I had to do was make stupid comments I would have made more."

"X is on the mailing list because the IETF is an open organization and anybody can join our working group mailing lists and attempt to contribute."
> Any criteria that anyone proposes that cannot be evaluated in an objective manner (and ideally from public records automatically by software) is IMNSHO a non-starter. 
> 
“A message that resulted in the intervention of the sergeant-at-arms or resulted in the Ombudsteam intervening is excluded from consdieration."
> If you propose subjective requirements e.g. "participate on the mailing list to the  point where that would qualify them" you're going to need someone to decide :the point where that would qualify them" and lots of       someones to handle the appeals and that should be part of the proposal.  So far I haven't seen that.
> 
N messages (order of 10?) to a working group mailing list no more than M seconds apart (where M is probably on the order of 10368000) during the past L seconds (where L is probably on the order of 63072000) are required to qualify. The script we ran showed that you had a gap that was too long: you disappeared for 10400000 seconds between April and July of 2020. This isn’t hugely difficult to automate.
> With all of these discussions, I've pretty much settled back into "registered (and attended) for 3 of the last 5 meetings in the mode for which the meeting is held excluding 107".   It's simple, it shows at least a little effort to participate, and we've got 109 and 110 to watch for possible gaming of the system since the next Nomcom won't be selected until after those two meetings.    I might add in "prove you're a unique non-virtual human" and I might track the rate of change for all the organizations in qualified people and report those statistics with an eye to revisiting the selection process statistical bias if necessary.
> 
And when we go back to in-person meetings, what then?

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 31 Jul 2020 14:43:09 +1200

Ted,

> “A message that resulted in the intervention of the sergeant-at-arms or resulted in the Ombudsteam intervening is excluded from consdieration."

Can you supply the Python code for that?

Regards
   Brian

---

From: Ted Lemon <[email-masked]>
Date: Thu, 30 Jul 2020 23:06:48 -0400

I could certainly write code to do that, assuming we decided that this was a good idea. We would have to track PR actions formally, if we don’t already. I assume we already do this for ombudsteam interventions. This is not exactly rocket science. 

> On Jul 30, 2020, at 22:43, Brian E Carpenter <[email-masked]> wrote:
> 
> Ted,
>

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 31 Jul 2020 11:23:50 +0000

>   I could certainly write code to do that, assuming we decided that this was a good idea. We would have to track PR actions formally, if we don’t already. I assume we already do this for ombudsteam…

"I could make a ham sandwich if I had ham."  We DON’T have the meta-data you require, which I believe is the point Brian was trying (subtly) to make.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 08:08:53 -0400

> "I could make a ham sandwich if I had ham."  We DON’T have the meta-data you require, which I believe is the point Brian was trying (subtly) to make.

Then maybe he could just say that.

I agree, in order for this to work, we’d have to collect the data. In the absence of that data, we wouldn’t be able to exclude messages that were clearly not “wanted participation.” Right now, people who get PR-actions and who get intervention from the Ombudsteam can still show up for in-person IETFs and still be eligible for nomcom. Excluding their abusive participation is a new feature, and would require new work. Doing a PR-action or an Ombudsteam intervention is already running code; having the person who takes that action note the message ID in a private database that can be used when running the nomcom algorithm is not a heavy lift, nor would it be a significant imposition on those individuals.

I think it might be worth asking why the resistance to this idea is so strong? Is it because you actually think the code is hard to write? Is it because you think this is a bad idea, but aren’t quite sure why? If it’s the latter, it would help us to get to make progress if you could sit and think for a bit and figure out what your real objection is (directed to Brian, mostly, I think).

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 31 Jul 2020 14:47:10 +0100

Hiya,

On 31/07/2020 00:17, Brian E Carpenter wrote:
> Now that we have two meetings with formalised on-line participation,
> we could extend Path 1 to include them (if we believe that is the
> right thing to do).
Assuming there's no mega-shift those vs. previous
f2f meetings, (and it didn't seem like it to me
from this weeks mail and meetings), that seems like
a good plan.

As a minor note: I think we want to include anyone
who asked for a waiver for two reasons: to keep
with the spirit of the waiver programme and also
because we don't want someone being challenged if
selected on the basis that they didn't pay. That
should also be simpler.

Cheers,
S.

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 31 Jul 2020 14:14:23 +0000

+1

---

From: "Pete Resnick" <[email-masked]>
Date: Fri, 31 Jul 2020 09:16:13 -0500

On 30 Jul 2020, at 21:34, Ted Lemon wrote:

> “A message that resulted in the intervention of the sergeant-at-arms 
> or resulted in the Ombudsteam intervening is excluded from 
> consdieration."

According to the BCP, you will not get that information from the 
Ombudsteam.

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 31 Jul 2020 14:19:26 +0000

*   Then maybe he could just say that.

This was my guess; Brian’s in NZ so wait for him to reply. In my view, it’s also not unreasonable to assume good familiarity with existing operations for those proposing large-scale changes.

  *   I think it might be worth asking why the resistance to this idea is so strong? Is it because you actually think the code is hard to write? Is it because you think this is a bad idea, but aren’t quite sure why? If it’s the latter, it would help us to get to make progress if you could sit and think for a bit and figure out what your real objection is (directed to Brian, mostly, I think).

I have a concern about maintaining a record of “personnel issues” and making what would normally be private discussions part of a public record.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 10:25:09 -0400

On Jul 31, 2020, at 10:16 AM, Pete Resnick <[email-masked]> wrote:
> According to the BCP, you will not get that information from the Ombudsteam.

I don’t know if there’s a way to do it that would be considered okay. Obviously the Ombudsteam would have to be in the flow, and I suppose one could compare inputs and outputs, so that might not work.  If not, c’est la vie.  I think this isn’t a general problem, but could be wrong since I don’t have access to the data. The reason I suspect it’s not a general problem is that I suspect that there aren’t any IETF participants whose _only_ countable participation results in Ombudsteam intervention. It may be that there just isn’t any point in carving out these exceptions.

---

From: S Moonesamy <[email-masked]>
Date: Fri, 31 Jul 2020 11:12:35 -0700

Hi Stephen,
>selected on the basis that they didn't pay. That

I assume that you meant: I think that we do not want ... in the above note.

I commented about the so-called "waiver programme" at 
https://mailarchive.ietf.org/arch/msg/ietf/0D7EonXw_0SihZFrodqgRL8GcYw/

Which section of the relevant document specifies that an attendee has 
to pay to qualify as having attended an IETF meeting?

Regards,
S. Moonesamy

---

From: Alissa Cooper <[email-masked]>
Date: Fri, 31 Jul 2020 15:11:59 -0400

Hi Ted,

I have some questions and comments for you and the group below.

> Several people pointed out that we will need to handle this much more quickly, given the current pandemic.  I agree.  I think for that we could run with a much more simple policy, possibly even within the process experiment mechanism.  The heart of that would be something like this:
> 

The third meeting in 2021 is scheduled to be IETF 112. Under your proposal, if IETF 112 is online, would it count as an in-person meeting or not?

I’m also wondering, regardless of which approach is taken, if it would make sense to scope one experiment just for the nomcom to be seated in 2021, rather than for the next two nomcoms. That way there is perhaps less danger of just codifying whatever gets specified in the experiment since it will have become the norm after two cycles, and it may make it easier to get something specified in time for the 2021 cycle while we spin up a parallel track where people can take more time to debate a longer term solution.

For your proposal specifically, scoping it to just the 2021 nomcom substantially reduces the potential for people to game the system. If your proposal extends to the 2022 nomcom and IETF 109, 110, 111, 112, and 113 are all online meetings, your scheme could be easily gamed by someone registering sock puppets at any three of the next four meetings.

I understand why draft-carpenter-eligibility-expand frames its proposal as a process experiment, but it strikes me that no matter what “experiment” is run we won’t be able to do a rigorous evaluation of the results because we don’t have agreed criteria that we’re optimizing for. That is, we don’t have agreement about what metrics characterize a “good” pool of nomcom volunteers — bigger or smaller, more or less “diverse,” better or worse familiarity with the community, etc. — other than perhaps by comparing to the size of the pool in recent years. With just months to get this done I don’t see a path to agreeing on other metrics. So despite the use of the “experiment” label, I think it will help us conclude this discussion in a timely fashion if we put the idea of experimental evaluation aside.

Alissa

> 
> Like for the "additional criteria" proposal, an assessment after the fact would be useful, but this is both a much smaller change from our current rules and seems likely to result in a larger pool.  Given the number of meetings in the longest possible period, there is some risk of gaming this, but the other nomcom limits and the limited time of the experiment mitigate this to some extent.
> 
> Again, my apologies for not participating up to this point and for following up on today's discussion prior to reviewing the archives.  I hope to review them soon.
> 
> regards,
> 
> Ted Hardie
> --

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 08:59:35 +1200

On 01-Aug-20 00:08, Ted Lemon wrote:
> On Jul 31, 2020, at 7:23 AM, Salz, Rich <[email-masked] <mailto:[email-masked]>> wrote:

I didn't intend to be subtle. I would prefer to hear directly from people familiar with the internals of the tracker and registration databases, but I don't believe we have such metadata. I don't know whether data from the formally established review teams is readily accessible.

> I think it might be worth asking why the resistance to this idea is so strong? Is it because you actually think the code is hard to write? Is it because you think this is a bad idea, but aren’t qui…

The general objection that people raised earlier** was that if sending email and/or writing reviews was counted, it would be easy for people to cheat the system by sending trivial email or reviews. And that would be possible well below the threshold for triggering PR-Actions. So either you need a subjective judgment or an AI solution to prevent gaming. And either of those would be very much subject to appeal.

But really my rule of thumb has been: if we can't measure it by running a tool on *today's existing databases* then it's a non-starter for next year's NomCom eligibility.

** Without archive-diving, I can't identify who the "people" were. Before the -00 draft, I thought about measuring email activity and concluded that it was a non-starter since it is so trivial to game. The same goes for submitting individual I-Ds. I did include review team activity but we took it out after the first round of discussion.

I hope that's clearer.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 09:02:05 +1200

> selected on the basis that they didn't pay. That

I agree. I wouldn't want to undercut the waiver system. However,
I'd prefer it if we could verify that people actually attended rather
than simply registering.

    Brian

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 17:03:00 -0400

> I'd prefer it if we could verify that people actually attended rather

Blue sheets.

---

From: Michael StJohns <[email-masked]>
Date: Fri, 31 Jul 2020 17:04:35 -0400

On 7/30/2020 10:34 PM, Ted Lemon wrote:
> On Jul 30, 2020, at 9:17 PM, Michael StJohns <[email-masked] 
> <mailto:[email-masked]>> wrote:
>> list a year ago that obviously mattered?"
>

Yes, but for example [email-masked] is an IETF community working 
group mailing list, and [email-masked] is an IETF community mailing list 
- need I go on?

>
>> or "Why is X on the list - they keep making stupid comments.  If I 
>> knew all I had to do was make stupid comments I would have made more."
>
> "X is on the mailing list because the IETF is an open organization and 
> anybody can join our working group mailing lists and attempt to 
> contribute."

Why is X on the list of qualified people - not why is X on the mailing 
list....    in any event, is "+1" a contribution?   Is "this idea is 
stupid for [list of reasons that have been debunked many many many 
times]" a contribution?   If so, I can automate a random +1 or -1 to be 
sent to the list every say 10 topics.

>> software) is IMNSHO a non-starter.
>>
> or resulted in the Ombudsteam intervening is excluded from consdieration."

The only list that has an SAA is the IETF list - other lists are 
generally managed by the WG chair or some other means. And as Pete 
noted, its going to be rare that anyone from the OT is going to be 
providing public information.

>> If you propose subjective requirements e.g. "participate on the 
>> mailing list to the  point where that would qualify them" you're 
>> going to need someone to decide :the point where that would qualify 
>> them" and lots of someones to handle the appeals and that should be 
>> part of the proposal.  So far I haven't seen that.
>>
> N messages (order of 10?) to a working group mailing list no more than 
> M seconds apart (where M is probably on the order of 10368000) during 
> the past L seconds (where L is probably on the order of 63072000) are 
> required to qualify. The script we ran showed that you had a gap that 
> was too long: you disappeared for 10400000 seconds between April and 
> July of 2020. This isn’t hugely difficult to automate.

And again - I can set up a counter-script to mail to the list to meet 
any published criteria like this.

But regardless, what you miss with your criteria are actual 
contributions: private notes to the WG chair, or back and forths with 
the authors on a document, or, for the GITHUB based WGs, you might miss 
pull requests or comments in the issues trackers.  Or the document that 
the contributor was working on is awaiting an action by the WG chair, by 
the AD, or by the RPC.  I don't find your criteria either useful or all 
that related to the concept of "contributing".

>> "registered (and attended) for 3 of the last 5 meetings in the mode 
>> for which the meeting is held excluding 107".   It's simple, it shows 
>> at least a little effort to participate, and we've got 109 and 110 to 
>> watch for possible gaming of the system since the next Nomcom won't 
>> be selected until after those two meetings.    I might add in "prove 
>> you're a unique non-virtual human" and I might track the rate of 
>> change for all the organizations in qualified people and report those 
>> statistics with an eye to revisiting the selection process 
>> statistical bias if necessary.
>>
The phrase was "in the mode for which the meeting is held" - e.g. if its 
an in person meeting, then in-person counts.   Else remote counts.  Note 
that this is not a long term proposal, just one to get us through the 
next Nomcom.

I'm coming to the belief that maybe we need to concentrate more on how 
the Nomcom is selected, and less on who is eligible to be selected.  As 
others have noted - "3 of 5 in person" was rough shorthand for "knows 
the IETF, and is up to date on its needs". Randomly picking people from 
a hat to then having them select from a set of volunteers for each 
position has mostly worked, but the IETF of today is not the IETF of 
even 10 years ago, and it may be time to rethink the process by which we 
select the ADs, Chair and IAB members.   That is obviously a much longer 
discussion.

Assuming we institute protections against sock puppets and virtual 
identities, I would be surprised if taking 3 of 5 from the registered 
attendees (who show up for at least one on-line session) for as long as 
we're virtual is any better or any worse than the results we were 
getting from the in person meetings. Let's pick something simple, and 
revisit it after the next Nomcom is seated.

Later, Mike

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 09:05:57 +1200

> to pay to qualify as having attended an IETF meeting?

None, as far as I know. Of course, for traditional meetings you will
only be on the attendee list if you have paid. (Well, perhaps that
statement is over-simplified, because media attendees and student
helpers don't have to pay, if I recall correctly.)

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 09:10:00 +1200

Hi Alissa, 

On your last point:
On 01-Aug-20 07:11, Alissa Cooper wrote:
...
> I understand why draft-carpenter-eligibility-expand frames its proposal as a process experiment, but it strikes me that no matter what “experiment” is run we won’t be able to do a rigorous evaluati…

Yes, good point. We should probably add a few words about how the experiment will be judged, but it can certainly not be a complete scientific evaluation. It's more of a way of forcing the issue to be reopened in a couple of years.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 09:11:56 +1200

On 01-Aug-20 09:03, Ted Lemon wrote:
> On Jul 31, 2020, at 5:02 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:

Sure, but are they already fed into a database? If not, what tools are needed to do so?

   Brian

---

From: Michael StJohns <[email-masked]>
Date: Fri, 31 Jul 2020 17:18:23 -0400

> Sure, but are they already fed into a database? If not, what tools are needed to do so?

AIRC, they're scanned in and discarded.  There's no plain text of the 
sheets and not likely to be one given how bad our handwriting tends to be.

Mike

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 17:19:13 -0400

On Jul 31, 2020, at 5:04 PM, Michael StJohns <[email-masked]> wrote:
> Yes, but for example [email-masked] <mailto:[email-masked]> is an IETF community working group mailing list, and [email-masked] <mailto:[email-masked]> is an IETF community mailing list - need I go on?

Apparently, because I have no idea what your concern is. Do you mean that if people participate in RGs, that doesn’t qualify them for nomcom? Well then either we include RG mailing lists in the set, or we decide that indeed participating in RGs doesn’t qualify you for nomcom. This is just a decision to make—there’s nothing inherently difficult about it.

> Why is X on the list of qualified people - not why is X on the mailing list....    in any event, is "+1" a contribution?   Is "this idea is stupid for [list of reasons that have been debunked many many many times]" a contribution?   If so, I can automate a random +1 or -1 to be sent to the list every say 10 topics.
> 
Yes, you can. If people really want to kill the IETF, they can automate bots to kill the IETF. This is possible now—it just isn’t happening because nobody is motivated to do it. I agree that this is a potential attack surface, but there are things we can do about it: maybe we need to only allow addresses registered in the datatracker? Maybe only addresses registered in the datatracker that are for people who attended online or in person? If we’re so concerned that this is going to happen that we think it’s urgent to prevent it, that would make it a lot harder.

> But regardless, what you miss with your criteria are actual contributions: private notes to the WG chair, or back and forths with the authors on a document, or, for the GITHUB based WGs, you might miss pull requests or comments in the issues trackers.  Or the document that the contributor was working on is awaiting an action by the WG chair, by the AD, or by the RPC.  I don't find your criteria either useful or all that related to the concept of "contributing”.

You know, this is starting to feel like a pissing contest. Yes, what I suggested doesn’t capture every contribution, but it does capture an important type of contribution. Yes, it can be gamed. And the gaming can be mitigated. My concern is that right now we are headed in the direction of valorizing “showed up for the meeting” above “participated on the mailing list” and I’d like to address that. If you think participating on the mailing list doesn’t matter, why are you responding here?

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 17:20:11 -0400

> The general objection that people raised earlier** was that if sending email and/or writing reviews was counted, it would be easy for people to cheat the system by sending trivial email or reviews.…

Brian, I think I’ve said three or four times, please actually argue with me if you disagree with this: right now all you have to do to be counted for nomcom is show up at a meeting. That’s a _really_ low bar if we start counting online meetings.

If you union that with other participation measures, I think it raises the bar while still meaningfully including the right people. Mailing list participation is one such measure.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 17:21:43 -0400

> Sure, but are they already fed into a database? If not, what tools are needed to do so?

We were told for 108 that the blue sheets were being tracked automatically. That presumably means that they are indeed being fed into the database.

If we wanted to do this for in-person meetings, we would need some additional tooling. But there, the fact that you showed up and picked up your badge is a good indication that you were there, so probably we don’t need the blue sheets for in-person meetings.

---

From: Ted Hardie <[email-masked]>
Date: Fri, 31 Jul 2020 15:11:13 -0700

Howdy,

Some responses in-line.

On Fri, Jul 31, 2020 at 12:12 PM Alissa Cooper <[email-masked]> wrote:

> > Beginning with IETF 109, for any plenary meeting in which the IETF had
> no in-person participation, any registrant for the meeting will be treated
> as an in-person registrant for the purposes of the NomCom rules. The
> The third meeting in 2021 is scheduled to be IETF 112. Under your
> proposal, if IETF 112 is online, would it count as an in-person meeting or
> not?
>
> I was assuming not, but since this was not even a draft-00 proposal,
that's of course up for further discussion.

> I’m also wondering, regardless of which approach is taken, if it would
> make sense to scope one experiment just for the nomcom to be seated in
> 2021, rather than for the next two nomcoms. That way there is perhaps less
> danger of just codifying whatever gets specified in the experiment since it
> will have become the norm after two cycles, and it may make it easier to
> get something specified in time for the 2021 cycle while we spin up a
> parallel track where people can take more time to debate a longer term
> solution.
>

I very much agree that limiting the scope of the experiment will help with
the assessment.  I was focused on limiting the number of changes during the
discussion of longer term solutions, but limiting the time frame of the
experiment also makes sense to me.

> For your proposal specifically, scoping it to just the 2021 nomcom
> substantially reduces the potential for people to game the system. If your
> proposal extends to the 2022 nomcom and IETF 109, 110, 111, 112, and 113
> are all online meetings, your scheme could be easily gamed by someone
> registering sock puppets at any three of the next four meetings.
>
> There are a couple of different ways that the sock puppet threat model can
play out, and I think a critical part of analyzing the risk is an
estimation of what percentage of the pool ends up as sock puppets.   While
there is some risk of sock puppets, there's also some balance in that the
bar for actual participants to stand is simultaneously lowered.  That might
change the risk factor by keeping the sock puppets to a relatively small
part of the pool.  The other aspects of nomcom selection, especially the
random selection and affiliation rules, somewhat limit the risk.

As I said earlier in the thread, I don't think the risk here is zero, but I
do think we could test over a nomcom cycle and recover if we see evidence
of this behavior.    There also may be some mitigations we could put in
place, but I have not yet thought through that part of the problem space
very well.  A lot of my early thinking clearly has some potental to
suppress volunteers signing up, which is not desirable.

> I understand why draft-carpenter-eligibility-expand frames its proposal as
> a process experiment, but it strikes me that no matter what “experiment” is
> run we won’t be able to do a rigorous evaluation of the results because we
> don’t have agreed criteria that we’re optimizing for.

I'm still behind on the archives here--is there disagreement or a lack of
clarity?

That is, we don’t have agreement about what metrics characterize a “good”
> pool of nomcom volunteers — bigger or smaller, more or less “diverse,”
> better or worse familiarity with the community, etc. — other than perhaps
> by comparing to the size of the pool in recent years. With just months to
> get this done I don’t see a path to agreeing on other metrics. So despite
> the use of the “experiment” label, I think it will help us conclude this
> discussion in a timely fashion if we put the idea of experimental
> evaluation aside.
>
> Let me suggest, then, that one way to do this is to make serial additions
to the eligibility and assess after each one.  Those would not be
experiments but would be presumptive permanent additions, but each small
enough that we could assess whether it was sufficient or more additions
were required.  One approach to that based on the existing document would
be to restore the existing method and add only active working-group draft
authorship.  Those are both things that can be checked mechanically and
where the overlap can be derived.  After a nomcom cycle with just that
change, we could consider others as successive additions.

My apologies if all of this has been worked over; I hope to catch up with
the archives next week.

Ted

> Alissa
>
> >
> > Like for the "additional criteria" proposal, an assessment after the
> fact would be useful, but this is both a much smaller change from our
> current rules and seems likely to result in a larger pool.  Given the
> number of meetings in the longest possible period, there is some risk of
> gaming this, but the other nomcom limits and the limited time of the
> experiment mitigate this to some extent.
> >

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 31 Jul 2020 23:22:16 +0100

Hiya,

> change, we could consider others as successive additions.

I have to say I'd be against that approach. I can't see us
modifying this that many times, maybe this one experiment
and once more (perhaps in the schoo WG) in a year or two.

Can I ask a question Ted: For myself, I really do want to
see us make some moves towards nomcom eligibility for
people who have been and may continue to be habitually non
attenders at f2f meetings. Do you share that goal? Or do
you consider it a non-goal or perhaps even as a thing to
be avoided?

Cheers,
S.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 14:49:00 +1200

On 01-Aug-20 09:21, Ted Lemon wrote:
> On Jul 31, 2020, at 5:11 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:

Right, but I'd like to hear that this is up and running before suggesting to Robert Sparks that he might run a test like he did to generate the existing Venn diagrams in the draft. (I think that for IETF 107, we only have the registration data, not the actual participation data.)

> If we wanted to do this for in-person meetings, we would need some additional tooling. But there, the fact that you showed up and picked up your badge is a good indication that you were there, so p…

That's how it's worked up to now, as I understand it. There's always been some resistance to a technology alternative for individual session records (i.e. a badge reader at the door). Personally, I'd be fine with that, but there are people who actively refuse to sign blue sheets and would presumably refuse to use a badge reader.

   Brian

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 23:03:32 -0400

It seems to me that refusing to sign blue sheets is an explicit declaration that you are not participating and that someone who does so should not be eligible for nomcom on the basis of their in person participation. Not that we can enforce this for f2f meetings without digitizing blue sheets, but in principle I think this is true. 

Before we consider asking Robert to look at something, we should decide if we want him to. That’s where I am in this process anyway. I am suggesting something that seems like a good idea. If there were agreement that indeed it might be a good idea, and we had some clarity about how we think it would work in theory, then we could figure out how to investigate whether it works in practice. These seem to me to be two related but separate steps. 

> On Jul 31, 2020, at 22:49, Brian E Carpenter <[email-masked]> wrote:
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 15:04:36 +1200

Ted,

I have a specific question at the end:

On 01-Aug-20 09:20, Ted Lemon wrote:
> On Jul 31, 2020, at 4:59 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:
> Brian, I think I’ve said three or four times, please actually argue with me if you disagree with this: 

(I have found myself reluctant to express personal preferences as long as I'm editing the draft, since I'd prefer to edit it in the direction of possible consensus.)

> right now all you have to do to be counted for nomcom is show up at a meeting. That’s a _really_ low bar if we start counting online meetings.

Exactly right. And at the moment the draft ducks that question. If we want to add on-line attendance, we have to be careful what it means. And we hit the same problem as for email and reviews: what level of participation counts? Shows up and listens? Shows up and hums? Shows up and speaks?

> If you union that with other participation measures, I think it raises the bar while still meaningfully including the right people. Mailing list participation is one such measure.

Wait, doesn't union *lower* the bar by including a wider set of people? Intersection raises the bar.

I suppose that what we're trying to measure is really contribution, not participation. That's why it's tricky. If we gave up on that and only measured participation, life gets much easier. What is our goal?

Regards
   Brian

---

From: Ted Lemon <[email-masked]>
Date: Fri, 31 Jul 2020 23:19:02 -0400

Sorry, I meant intersection, not union. I think we are measuring participation. I don’t know how we could measure contribution. Maybe that’s our disconnect?  I might have misspoken on that point—it’s helpful that you made the distinction explicit. There is not way to do that that’s not subjective and I don’t blame you for questioning my sanity if that was what I seemed to you to be saying. 

> On Jul 31, 2020, at 23:04, Brian E Carpenter <[email-masked]> wrote:
> 
> Ted,
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Aug 2020 15:31:07 +1200

> Sorry, I meant intersection, not union. I think we are measuring participation. I don’t know how we could measure contribution. Maybe that’s our disconnect?  I might have misspoken on that point—it…

My sanity is at least as much in question ;-). Indeed, I think the objective has been a bit unclear in a lot of the discussion on this list. I'd be interested in other people's opinion on this point.

    Brian

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 1 Aug 2020 00:08:19 -0400

What I would like to measure is meaningful participation.  However, as 
far as I can tell directly measuring that is impossible.  So we are 
discussing various measures of contribution.  And some measures of 
participation that are not calibrated to "meaningful".  I personally 
tend to see the measures of contribution as better proxies for what I 
want.  But I may be wrong on that.

Yours,
Joel

---

From: Richard Outerbridge <[email-masked]>
Date: Sat, 1 Aug 2020 01:03:04 -0400

> On 2020-07-31 (213), at 17:04:35, Michael StJohns <[email-masked]> wrote:
> 
>> On Jul 30, 2020, at 9:17 PM, Michael StJohns <[email-masked] <mailto:[email-masked]>> wrote:

So far as I’ve been able to discern, IETF meetings are like the reading out of Orders to units on the Board in a game of Diplomacy :)
__outer