Subject: [Eligibility-discuss] Review of draft-carpenter-eligibility-expand-06

From: Barry Leiba <[email-masked]>
Date: Mon, 26 Oct 2020 15:33:21 -0400

Hi, Brian, and all,

I have some comments on the draft -- some editorial, and one
substantive that I have no solution for, so it's likely to be moot:

— Section 1 —

   Also according to [RFC8713], the qualification for signing a
   community petition for the recall of certain IETF office-holders is
   that same as for the Nominating Committee.  This document does not
   change that.

The last sentence could be misunderstood to mean that recall
eligibility remains tied to 8713, not to this document’s experiment.
I suggest rewording to make it explicitly clear, perhaps as:

NEW
   Also according to [RFC8713], the qualification for signing a
   community petition for the recall of certain IETF office-holders is
   that same as for the Nominating Committee.  This experiment retains
   that connection, and therefore applies to the recall process as well
   as to the Nominating Committee process.
END

— Section 3 —
The last list item is not parallel in construction to the others, each
of which starts with a verb; I suggest making it parallel:

NEW
   *  Provide algorithmic criteria, so that the Secretariat can check
      them mechanically against available data.
END

— Section 4 —

   *  Path 2: Has been a WG Chair or Secretary within the last 3 years.

I suggest spelling out “Working Group” here and in Section 5
(“Working-Group-adopted”).

   *  Path 3: Has been a listed author of at least 2 IETF stream RFCs
      within the last 5 years.  An Internet-Draft that has been approved
      by the IESG and is in the RFC Editor queue counts.

I find this troubling for a few reasons, though I’ll say up front that
I don’t have a suitable alternative that meets the “algorithmis” goal.

1. While I understand the reason, as stated in the notes, for making
this 5 years, it seems odd that someone whose name got on a draft 5
years ago becomes NomCom-eligible, while someone who was the working
group chair at the time but stepped down a year later does not.

2. This makes is important to wheedle one’s way into the author lists
of documents, and I hesitate to make that an attractive goal.

3. It’s often the case that people listed in a Contributors section
have been more active participants (and had more to do with shaping
the document) than some of those listed on the front page.

4. If we do a -bis of an RFC from 10 or 15 years ago, we will often
keep the original authors listed and add the -bis document’s editor.
This will have the unwanted effect of suddently making people who
haven’t been around the IETF in more than a decade NomCom-eligible.

As I said, though, I have no good alternative, and I’m not opposed to
trying it as an experiment.  I also think it’s unlikely that someone
enabled by my point 4 is likely to try volunteering.  I’m more
concerned about my points 1 and 3.

I’m also concerned, as Victor Kuarsingh is, that this is an unusual
sort of experiment in that “running it for a year” really means
“running a single instance of it.”  Again, I don’t know that there’s
anything we can do about that, but I worry that we simply will get
little or no data on the real effect of making this change — we can
only measure what it does to the volunteer pool, but not what effect
it has on the NomCom process or result.

-- 
Barry

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 28 Oct 2020 09:07:09 +1300

Hi Barry,

Thanks for the review:
>    as to the Nominating Committee process.

We also have a suggested rewrite from Victor's review, so we will
certainly fix this. (This point was mentioned in the now-deleted
Future Work section, which I think is why the main text phrasing
was sloppy.)

>       them mechanically against available data.

Sure.

> (“Working-Group-adopted”).

Ack.

> group chair at the time but stepped down a year later does not.

Yes, one could play with this for ever. Maybe we will get enough
IETF Last Call inputs to confirm the 5 years or reduce it.

> of documents, and I hesitate to make that an attractive goal.

Anything can be gamed, and author credits have always been a target.
But is being on NomCom really that big a prize compared to other
incentives to be a listed author?
 
> the document) than some of those listed on the front page.

That's true, but Contributors sections are relatively rare. Our goal
here is surely to have a big enough pool, not to collect every possible
name.

> haven’t been around the IETF in more than a decade NomCom-eligible.

True, but they are unlikely to actually volunteer.

> it has on the NomCom process or result.

The text allows the IESG to run it for a second year. Beyond that,
who knows what the IETF meeting patterns will look like?

Thanks
    Brian

---

From: Michael Richardson <[email-masked]>
Date: Thu, 29 Oct 2020 12:43:00 -0400

> IETF Last Call inputs to confirm the 5 years or reduce it.

We increased it from 3-years to 5-years for RFC-publication because we felt
3-4 years was a typical cycle time from ID-00 to RFC.  So if someone works on
one document at a time that 3-years might catch them between publication
times.

To me, the question is whether or not we should increase WG chair criteria
from 3-years to 5-years if you want parity.

    > here is surely to have a big enough pool, not to collect every possible

I'd really like to encourage the contributors section to always use the
available structure so that we can, going forward, collect that information
mechanically.  Then, we could collect some data.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Barry Leiba <[email-masked]>
Date: Thu, 29 Oct 2020 13:31:03 -0400

> from 3-years to 5-years if you want parity.

For me, it's not parity so much as what's a reasonable judgment of
active participation.  Do we think that someone whose document went
into the RFC Editor queue 5 years ago, and who has not been active
since, is an active participant who should be NomCom eligible?  Do we
think a working group chair who stepped down 3 years ago (or 5), and
who has not been active since, is an active participant who should be
NomCom eligible?

In general, I'm in favour of erring on the side of inclusion.  That
said, my preference is to stay with the spirit of including people who
remain active in the IETF, and I know that it's very hard to come up
with objective criteria for that.

> mechanically.  Then, we could collect some data.

Indeed.  I would prefer to include data from Contributors sections,
and, as Michael says, to encourage more use of that section to call
out people who were truly key contributors.  It can all be gamed, of
course, but a great many truly active and valuable participants simply
don't wind up as document authors, but we would be much poorer without
them nonetheless.

Barry

---

From: Ted Lemon <[email-masked]>
Date: Thu, 29 Oct 2020 13:52:10 -0400

> who has not been active since, is an active participant who should be

You could finesse this by adding the requirement that there be an RFC in the last five years and an I-D in the past, say, two years, but that might be more strict than is necessary.

---

From: Barry Leiba <[email-masked]>
Date: Thu, 29 Oct 2020 14:22:12 -0400

Yes, or, what I would really like, some sense of how active someone has
been in working group discussions.  But I know how bad the results are
likely to be if we key any of this on posting to mailing lists.

And I really don’t mean to have us rehashing things that have already been
discussed and ruled out, so I think that, lacking a really good suggestion
of how to make it better, I have to accept that we have a workable
experiment to proceed with.

I wanted to get my comments in now, because when this comes to the IESG my
review as an AD will almost certainly be a simple “Yes” ballot.

Barry

On Thu, Oct 29, 2020 at 1:52 PM Ted Lemon <[email-masked]> wrote:

> You could finesse this by adding the requirement that there be an RFC in
> the last five years and an I-D in the past, say, two years, but that might
> be more strict than is necessary.
>
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 29 Oct 2020 14:27:10 -0400

On Oct 29, 2020, at 2:22 PM, Barry Leiba <[email-masked]> wrote:
> Yes, or, what I would really like, some sense of how active someone has been in working group discussions.  But I know how bad the results are likely to be if we key any of this on posting to mailing lists.

I think that really isn’t a foregone conclusion. People suggest that there will be an influx of new garbage postings if we make this the rule, but I suspect that’s not at all the case. As an experiment, I think it would make a lot of sense to test this hypothesis. We could tie it to the datatracker to keep people honest: if you have an email address, you can claim it by creating a datatracker login, and then we can correlate addresses between mailing lists and the datatracker database. Still definitely gameable, but how serious of a risk is that? If we did this as an experiment for a year, we’d find out, and could easily stop doing it if it turned out to /actually/ cause a problem.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 30 Oct 2020 09:16:06 +1300

On 30-Oct-20 05:43, Michael Richardson wrote:
...
> mechanically.  Then, we could collect some data.

I agree with that for the longer term, and I'm gald you've started that as a tools discussion. But it's not a practical option for now.

    Brian

---

From: Michael Richardson <[email-masked]>
Date: Thu, 29 Oct 2020 18:22:16 -0400

Ted Lemon <[email-masked]> wrote:
    >> Yes, or, what I would really like, some sense of how active someone
    >> has been in working group discussions.  But I know how bad the results
    >> are likely to be if we key any of this on posting to mailing lists.

    > I think that really isn’t a foregone conclusion. People suggest that
    > there will be an influx of new garbage postings if we make this the
    > rule, but I suspect that’s not at all the case. As an experiment, I
    > think it would make a lot of sense to test this hypothesis. We could
    > tie it to the datatracker to keep people honest: if you have an email
    > address, you can claim it by creating a datatracker login, and then we
    > can correlate addresses between mailing lists and the datatracker
    > database. Still definitely gameable, but how serious of a risk is that?
    > If we did this as an experiment for a year, we’d find out, and could
    > easily stop doing it if it turned out to /actually/ cause a problem.

This is definitely the kind of data collection experiment that we should do.

We need to tell people about this, because it might be important for people
to know that from date X, if your ML posting address is not listed as an alias
in the DT, then your posting won't get counted.  So, please go fix your DT
entries.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 30 Oct 2020 11:32:03 +1300

> in the DT, then your posting won't get counted.  So, please go fix your DT

I'm all in favour of experimenting with this sort of thing. But not within the
timescale of getting the current draft approved, so that people know what the
criteria will be well in advance of next year's NomCom process.

    Brian

---

From: Ted Lemon <[email-masked]>
Date: Thu, 29 Oct 2020 18:34:20 -0400

> criteria will be well in advance of next year's NomCom process.

Okay, but I proposed the same thing six months ago and that’s what you said then too… :)

So maybe we should take an action item from this discussion to actually do this, if there’s a willingness to do it. I’m fine with not doing it as part of the current experiment.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 29 Oct 2020 20:19:56 -0400

> criteria will be well in advance of next year's NomCom process.

a) I agree that it can't go in as a criteria.
b) Ted is right, you said this six months ago.
c) I said a few months ago, that I wanted to collect a set of things that we
   thought about, but aren't doing now.  But that people should be aware of
   them.

"Tell me how you'll measure me, and I'll tell you how I'll act".
(I heard this attributed to W Edwards Deming, but the quotation cites don't agree)

So, if we can't collect the things possible into this document, do they go
into another document?  And if so, does it get published, or just talked
about loudly?

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 30 Oct 2020 00:25:44 +0000

FWIW, I very much agree with Brian. See below for why...

> into another document?  And if so, does it get published, or just talked

This document will undergo an IETF last call. I predict that
most people won't care and will say nothing. But many will
care a lot and say a lot about all of the text it contains.

Given we do need to solve this problem in the short term, any
and all extraneous text is IMO a risk not worth taking. It
is entirely reasonable that many of us reasonable people
disagree here and there about the margins between extraneous
and not of course, but personally I reckon collecting any
such additional information in a different draft is a way
better plan. I guess where that ends up will depend on a
bunch of things including the actual data found.

Cheers,
S.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 31 Oct 2020 09:27:30 +1300

Bron is shepherding this draft, and I'd like to know if he
sees any changes that we should make urgently in the next day or so.
Otherwise it won't be updated before the I-D cutoff.

There are numerous changes needed following reviews by Lars, Victor
and Barry, but those are really all non-contentious clarifications
of intent. There is a remaining sense of doubt, I think, about
path 3 (authorship) but I didn't see a concrete proposal to change
it.

Regards
   Brian

---

From: "Pete Resnick" <[email-masked]>
Date: Fri, 30 Oct 2020 15:58:28 -0500

I'm always wary about jumping into discussions where I'm acting as 
chair, but given that I don't think I'll be calling consensus on 
anything substantive in this document as one of the gendispatch chairs, 
I think I can safely take my hat off to give a bit of input below. If 
someone feels otherwise, let me know and I'll be happy to hand the reins 
to Francesca for any of the gendispatch discussion.

On 30 Oct 2020, at 15:27, Brian E Carpenter wrote:

> There is a remaining sense of doubt, I think, about

I think path 3 is perfectly reasonable as part of the experiment, and 
might be fine in the long term. Let me try to allay that remaining sense 
of doubt:

As part of the experiment, we are pretty sure that nobody in the last 5 
years has gamed the system in anticipation of this experiment by getting 
their name in the author list of a document in hopes of getting on a 
NomCom. Given that, the enfranchisement (I hate that word in this 
context, but whatever) of authors who do regularly participate in the 
IETF I think is more likely to increase good than the potential for 
"honorary" authors to increase badness by putting their names in for 
NomCom. (Given other discussion, I think it would be perfectly 
reasonable to add anyone who appeared in a Contributors section if such 
a section appears, but no need to decide that for the next version of 
the draft.)

I also think a good thing to add to the experiment is a design team to 
do an informal review of the past couple of years of RFCs and see they 
can determine how many "honorary" authors there have been versus actual 
contributors. We might also look to see if that number increases or 
decreases during the experiment. I suspect (with no evidence) that the 
number and percentage is actually pretty low, and in fact will get lower 
once chairs start realizing that the experiment is going on and might 
start pushing back on such "honorary" authors. Therefore, I think over 
the longer term path 3 will have minimal harms.

So I'm very much in favor of keeping path 3. The potential for harm I 
believe is low and the potential for good is pretty high. I also think 
it's one of the few objective metrics we have that might have a good 
correspondence to actual participation.

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: "Bron Gondwana" <[email-masked]>
Date: Sat, 31 Oct 2020 22:07:46 +1100

I'm seeing a lot of bikeshedding around the edges but not any significant proposal that I think needs changes to the experiment for one year.

As I've said before, we could micro-optimise this forever but I think this is a good experiment to run for this year in its current form, and we can run it again either the same or with modifications next year.  Either way, we'll need a new process to do that, so let's not overdo ourselves trying to predict the future now and do work that that we're just as likely to discard next year anyway.

Bron.

On Sat, Oct 31, 2020, at 07:27, Brian E Carpenter wrote:
> Eligibility-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/eligibility-discuss
> 

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]