Subject: [Eligibility-discuss] Eligibility strawman

From: Brian E Carpenter <[email-masked]>
Date: Mon, 16 Mar 2020 10:54:20 +1300

Hi,

I feel that we should move as fast as reasonably possible, rather than rehashing the arguments again.

Hence, here's a strawman. Just the bare bones, of course:

# Goals

- Resolve the issue of active remote (or rarely in-person) participants being disenfranchised in the NomCom and recall processes.

- Prepare for an era in which face-to-face plenary meetings are less frequent (thus extending the issue to many, perhaps a majority, of participants).

- Do this quickly in view of the risk of IETF 108 being cancelled too, but ensure that the result is flexible.

- Ensure that those eligible are true "participants" with enough current understanding of IETF practice and people to make informed decisions.

- Criteria must be algorithmic so that the Secretariat can check them mechanically.

# Criteria

To keep things simple, use the same eligibility rules for NomCom volunteers and for recall petitioners (no change). The issue of how many petitioners are required for a recall is separate and is not discussed here.

Define several alternative paths to eligibility, any one of which is sufficient (unless the person is otherwise disqualified):

- Path 1: as today, has attended 3 out the last 5 *in-person* plenary IETF meetings.

- Path 2: has been a WG Chair or Secretary within the last 3 years.

- Path 3: has submitted at least 6 reviews as a member of of an official IETF review team within the last 3 years.

- Path 4: has served in the IESG or IAB, or has been appointed to a formal role by the IESG or IAB, within the last 5 years.

- Path 5: has been a listed author of at least 2 IETF stream RFCs within the last 5 years.

# Criteria not included

- Individual I-D authorship (too easy).

- Sending lots of email (too easy).

- Remote attendance (no objective way to measure active participation).
 
Regards
   Brian Carpenter

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 15 Mar 2020 23:21:11 +0000

Hiya,

I could but won't quibble with the criteria. They'll
get sufficient quibbling in IETF LC:-)

I think the only thing missing is whether or not we
care by which path nomcom voting members qualify. I
think there's a reasonable argument that some subset
of 'em (maybe 50-60%) ought qualify via path#1, just
to de-risk things, and maybe reassure those who find
change problematic.

Other than that, write the I-D (or I'd be happy to
help with that), and I'll support it. I suspect events
and the long-duration discussions on this may mean
that this is an idea whose time has come.

Cheers,
S.

On 15/03/2020 21:54, Brian E Carpenter wrote:
> - Resolve the issue of active remote (or rarely in-person)
> participants being disenfranchised in the NomCom and recall
> processes.
> 
> - Prepare for an era in which face-to-face plenary meetings are less
> frequent (thus extending the issue to many, perhaps a majority, of
> participants).
> 
> - Do this quickly in view of the risk of IETF 108 being cancelled
> too, but ensure that the result is flexible.
> 
> - Ensure that those eligible are true "participants" with enough
> current understanding of IETF practice and people to make informed
> decisions.
> 
> To keep things simple, use the same eligibility rules for NomCom
> volunteers and for recall petitioners (no change). The issue of how
> many petitioners are required for a recall is separate and is not
> discussed here.
> 
> Define several alternative paths to eligibility, any one of which is
> sufficient (unless the person is otherwise disqualified):
> 
> - Path 3: has submitted at least 6 reviews as a member of of an
> official IETF review team within the last 3 years.
> 
> - Path 4: has served in the IESG or IAB, or has been appointed to a
> formal role by the IESG or IAB, within the last 5 years.
> 
> Regards Brian Carpenter

---

From: John C Klensin <[email-masked]>
Date: Sun, 15 Mar 2020 22:47:57 -0400

--On Monday, 16 March, 2020 10:54 +1300 Brian E Carpenter
<[email-masked]> wrote:

> Hi,
> 
> I feel that we should move as fast as reasonably possible,
> rather than rehashing the arguments again.
> 
>...
 
># Criteria
> 
> To keep things simple, use the same eligibility rules for
> NomCom volunteers and for recall petitioners (no change). The
> issue of how many petitioners are required for a recall is
> separate and is not discussed here.

See below about recalls.

> Define several alternative paths to eligibility, any one of
> which is sufficient (unless the person is otherwise
> disqualified):
> 
> - Path 3: has submitted at least 6 reviews as a member of of
> an official IETF review team within the last 3 years.
> 
> - Path 4: has served in the IESG or IAB, or has been appointed
> to a formal role by the IESG or IAB, within the last 5 years.
> 

Brian,

I think this works, more or less.  I.e., "close enough".   I
think one could tune by, e.g., allowing combinations of those
criteria, but probably not worth the effort.  Also, as we
discussed in a prior round of these issues long ago, I prefer
some sort of decay function to finding ourselves in a position
in which a lot of activity five (or three) years back counts but
similar activity six (or four) years ago doesn't and would
rather see some evidence of more recent involvement.   However,
that might be enough more complicated to not be worth the effort.

> - Remote attendance (no objective way to measure active

As to recalls, I think fairness and justice require that anyone
who can be hurt by an pattern of actions be able to initiate a
recall effort and take responsibility for petition-signing.  It
may be entirely reasonable to impose some criteria for
endorsements by some number of people who meet requirements
similar to the above in order to prevent assorted attacks, but
the idea that someone who feels injured or believes that
activities of someone in the leadership were sufficiently out of
line to justify removal might round up others to sign a petition
but not be able to sign it themselves and have people who are in
similar situations and similarly aggrieved just does seem right
to me.

   best,
    john

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 16 Mar 2020 16:44:50 +1300

Hi John,

> that might be enough more complicated to not be worth the effort.

Fair enough. If I turn this into a draft, I could add a section of
"Hard problems left for later."
 
> similar situations and similarly aggrieved just does seem right

Yes, (inferring a missing "not") I appreciate that, and it may well
be that the NomCom and petition-signing criteria should indeed be
different. But for present purposes I'm inclined to put that in
the "later" bucket.

Regards
    Brian

---

From: Michael Richardson <[email-masked]>
Date: Mon, 16 Mar 2020 00:09:15 -0400

Clipping the part about the goals, as I have no dispute at all with them.
Can we put that into a document as a problem statement?

> - Criteria must be algorithmic so that the Secretariat can check them mechanically.

I am very very much in favour of this.
I think that participants should be able to get this feedback directly from
the DT, and I think that it should probably be on the public dt/person page
should indicate the status.  Do we agree that the criteria are public?

Implicit in this is that you must have a DT login to be eligible.
(That's a change from the status quo, btw, as the Registration system and
it's database is not presently integrated into the DT, but I understand that
there is a plan)

    > - Path 1: as today, has attended 3 out the last 5 *in-person* plenary IETF meetings.

I'm actually rather not that attached to this criteria, btw.
I don't mind grand-fathering everyone who has this criteria already met in
some way.

    > - Path 5: has been a listed author of at least 2 IETF stream RFCs

I think that this list is pretty straightforward and uncontroversial.
It might not cast a wide enough net, but I think that we should figure out
in a document, if we need a specific amending formula for the document.
(do we need something more formal than IETF Consensus document?)

You have identified Paths, but I wonder if they can be combined.
What if I've written 1 IETF stream RFCs and done 3 reviews?
Would you want to instead establish a point system for each path?

What if my seven IETF stream RFCs have been suck in IESG review for too many
months or been in MISREF for 4+ years?   What if I publish 6 documents in one
year, and none of 4 years, because that's how long they take?

    > - Sending lots of email (too easy).

Agreed.  Might even be provide negative points :-)

    > - Remote attendance (no objective way to measure active participation).

By the same token, in-person attendance may not actually be that objective a
way to measure.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Mon, 16 Mar 2020 00:11:11 -0400

> to de-risk things, and maybe reassure those who find

There are a number of ways of doing this, but some ways appear to create two
pools of volunteers.   I may be okay with that, but I would prefer if the
subset was set on a declining schedule.

    > that this is an idea whose time has come.

+1

---

From: Michael Richardson <[email-masked]>
Date: Mon, 16 Mar 2020 00:48:35 -0400

> that might be enough more complicated to not be worth the effort.

Given the way that our technical specifications tend to have four or five
documents which reference each other, and this can cause multi-year MISREF
events, it is quite reasonable that we would observe fairly non-stochastic
RFC publication dates.
This also fails to acknowledge efforts of documents that might get stuck or
even abandonned at or near IESG review stage.
So tweak I would make to Path 5 would be to change this to be an author of a
document which has gone through an WGLC (or IETF LC for AD sponsored, I guess).

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Warren Kumari <[email-masked]>
Date: Mon, 16 Mar 2020 19:32:37 -0400

[ no-hats ]

> Would you want to instead establish a point system for each path?

I personally think I really like Brian's proposal,  but two thoughts:
1: I really don't like the (proposed) points tweak for a number of
reasons - the point distribution seems like it will be arbitrary and
lead to much bikeshedding; having eg 47.5 out of the needed 50 feels
weirdly nickel and diming, and just generally odd to me -- I think
that somehow it feels like it becomes a scavenger hunt / gamification
system. It also somehow feels like "I'll gladly pay you Tuesday for a
hamburger today."

2: I also don't really like Path 3 -- I love my directorate reviewers,
but the quality of the reviews is very variable - having incentives
where "come serve on a directorate and send 6 "No isses" reviews gets
you a prize" feels like completely the wrong set of incentives. Yes,
the obvious antidote is "Your reviews must be at least 324 words long,
and identify at least 3 factual errors and 3 nits" is, obviously,
worse...

> year, and none of 4 years, because that's how long they take?

Perhaps it should be from when the document enters the RFC Editor
queue, not from when the finished RFC pops out? Yes, this does not
address the stuck in eval forever issue, but hopefully that doesn't
happen *too* often. There are also 2 weird corner cases / incentives
here:
1: I've had a number of documents that completed AD review, IETF LC,
and / or IESG review and I was waiting on the authors to submit new
versions so I could progress them (in one case, it was simply
cutting-n-pasting in some editorial change - this took many months and
many many reminders by myself and the chairs) - having some sort of
incentive for authors to progress things might be really nice...
2: you may also have the other gaming issue -- e.g: I have 3 drafts
which are almost cooked -- instead of publishing them all now / in a
batch, I have an incentive to space them out over the next 6(?) years
to keep up my rolling average... I suspect that this is not that
likely though.

Other than that, I really like this idea...
W

> Eligibility-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/eligibility-discuss

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Ted Lemon <[email-masked]>
Date: Mon, 16 Mar 2020 20:31:55 -0400

might be nice to make working group participation part of the equation.  I think one of our biggest problems in the IETF is that work gets done and then nobody reads it because they checked out before it was finished.

Sent from my iPad

> On Mar 16, 2020, at 7:33 PM, Warren Kumari <[email-masked]> wrote:
> 
> [ no-hats ]
>

---

From: Warren Kumari <[email-masked]>
Date: Mon, 16 Mar 2020 20:53:04 -0400

On Mon, Mar 16, 2020 at 8:31 PM Ted Lemon <[email-masked]> wrote:

> might be nice to make working group participation part of the equation.

Define “participation”...

I think one of our biggest problems in the IETF is that work gets done and
> then nobody reads it because they checked out before it was finished.

I don’t disagree that this is an issue, but I don’t think that tying a
“prize” to participation gets a better outcome, and think it will make
things much worse...

It doesn’t seem like something that can be measured - there are many people
who post to mailing lists, referencing documents, but who clearly haven’t
actually thought about the issues and haven’t actually contributed - and
then you have the other side of the scale who say very little, but their
participation is deep and thoughtful.

Unless we have something like a test to determine if someone has *actually*
read the document, and has actually *thought* about it, I cannot see how
this doesn’t simply add noise without signal.
“I read your document, it was very nice. The font was especially pretty...”
is not something we want to reward, but “Sec.3, paragraph 2, bullet 5 is
wrong - bit 7 is already used for signaling fwirzles.” is — and I don’t
think that there is a way to deterministically differentiate these...

W

-- 
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
   ---maf

---

From: Ted Lemon <[email-masked]>
Date: Mon, 16 Mar 2020 21:06:03 -0400

If a review results in a change to a document, clearly it was useful.   If it results in no change to the document, perhaps not so useful, or perhaps the document was just done.  I think it’s okay if you don’t get credit for the “done” reviews—if you are an active participant, you’ll be doing reviews before the document is done.  Maybe if you did a useful review and then a final review, you get an additional point or something.

It’s worth pointing out that one of the ways to prevent this being gamed is for working group chairs to pay attention.  If being a WG chair gets you nomcom eligibility, or counts toward it, it should be the case that you are actually doing your job, including actively fostering good discussion and not just making demands and ignoring it when the mailing list explodes.

Tracking social capital is a hard problem, but it’s actually the problem you’re trying to solve.  The idea that the most valuable contribution a person makes to the IETF is showing up in person is pretty obviously wrong, and the reason this is hard is because we’re talking about doing better than that.

While we’re at it, implementing a draft and keeping the implementation up to date with the version that becomes RFC should give you at least as much credit as being an author on an RFC, because otherwise we are encouraging people to write RFCs that nobody uses.

Sent from my iPad

> On Mar 16, 2020, at 8:53 PM, Warren Kumari <[email-masked]> wrote:
> 
> 
> 
> 
>> I think one of our biggest problems in the IETF is that work gets done and then nobody reads it because they checked out before it was finished.
> 
> I don’t disagree that this is an issue, but I don’t think that tying a “prize” to participation gets a better outcome, and think it will make things much worse...
> 
> It doesn’t seem like something that can be measured - there are many people who post to mailing lists, referencing documents, but who clearly haven’t actually thought about the issues and haven’t actually contributed - and then you have the other side of the scale who say very little, but their participation is deep and thoughtful.
> 
> Unless we have something like a test to determine if someone has *actually* read the document, and has actually *thought* about it, I cannot see how this doesn’t simply add noise without signal.
> “I read your document, it was very nice. The font was especially pretty...” is not something we want to reward, but “Sec.3, paragraph 2, bullet 5 is wrong - bit 7 is already used for signaling fwirzles.” is — and I don’t think that there is a way to deterministically differentiate these...
> 
> W
> 
> 
>> 
>> 
>> Sent from my iPad
>> 
> I don't think the execution is relevant when it was obviously a bad idea in the first place.
> This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants.
>    ---maf

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 16 Mar 2020 21:16:07 -0400

Ted, your description below seems that we want to make people eligible 
for nomcom as a reward for the social contribution.  Huhh?

While rewarding contribution is a nice idea, the nomcom ought not be a 
tool for that.  The qnomcom eligibility rules, as I understand them and 
their purpose, are an effort to approximate selecting people who 
understand and care about how the IETF is working (and ways it can 
work.)   As many people have said, we know that the rules over-emphasis 
some things and miss some things that matter.  "Has contributed to 
making the community work" is a useful measure if it actually represents 
a good confidence that the person actually has seen the processes.

One of the reasons we tend to include document authors and WG chairs in 
crafting other metrics is that those are folks who have had to deal with 
the processes and the practicalities.

Yours,
Joel

On 3/16/2020 9:06 PM, Ted Lemon wrote:
> If a review results in a change to a document, clearly it was useful.   
> If it results in no change to the document, perhaps not so useful, or 
> perhaps the document was just done.  I think it’s okay if you don’t get 
> credit for the “done” reviews—if you are an active participant, you’ll 
> be doing reviews before the document is done.  Maybe if you did a useful 
> review and then a final review, you get an additional point or something.
> 
> It’s worth pointing out that one of the ways to prevent this being gamed 
> is for working group chairs to pay attention.  If being a WG chair gets 
> you nomcom eligibility, or counts toward it, it should be the case that 
> you are actually doing your job, including actively fostering good 
> discussion and not just making demands and ignoring it when the mailing 
> list explodes.
> 
> Tracking social capital is a hard problem, but it’s actually the problem 
> you’re trying to solve.  The idea that the most valuable contribution a 
> person makes to the IETF is showing up in person is pretty obviously 
> wrong, and the reason this is hard is because we’re talking about doing 
> better than that.
> 
> While we’re at it, implementing a draft and keeping the implementation 
> up to date with the version that becomes RFC should give you at least as 
> much credit as being an author on an RFC, because otherwise we are 
> encouraging people to write RFCs that nobody uses.
> 
> Sent from my iPad
> 
>> On Mon, Mar 16, 2020 at 8:31 PM Ted Lemon <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>     I think one of our biggest problems in the IETF is that work gets
>>     done and then nobody reads it because they checked out before it
>>     was finished.
>>
>>
>> It doesn’t seem like something that can be measured - there are many 
>> people who post to mailing lists, referencing documents, but who 
>> clearly haven’t actually thought about the issues and haven’t actually 
>> contributed - and then you have the other side of the scale who say 
>> very little, but their participation is deep and thoughtful.
>>
>> Unless we have something like a test to determine if someone has 
>> *actually* read the document, and has actually *thought* about it, I 
>> cannot see how this doesn’t simply add noise without signal.
>> “I read your document, it was very nice. The font was especially 
>> pretty...” is not something we want to reward, but “Sec.3, paragraph 
>> 2, bullet 5 is wrong - bit 7 is already used for signaling fwirzles.” 
>> is — and I don’t think that there is a way to deterministically 
>> differentiate these...
>>
>> W
>>
>>
>>
>>
>>     Sent from my iPad
>>
>>     > On Mar 16, 2020, at 7:33 PM, Warren Kumari <[email-masked]
>>     <mailto:[email-masked]>> wrote:
>>     >
>>     > [ no-hats ]
>>     >
>>     >> <[email-masked] <mailto:[email-masked]>> wrote:
>>     >>
>>     >>
>>     >> Brian E Carpenter <[email-masked]
>>     <mailto:[email-masked]>> wrote:
>>     which is
>>     to a
>>     >> Michael Richardson <[email-masked]
>>     <mailto:[email-masked]>>, Sandelman Software Works
>>     >> [email-masked] <mailto:[email-masked]>
>>     > [email-masked] <mailto:[email-masked]>

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 17 Mar 2020 14:21:59 +1300

> If a review results in a change to a document, clearly it was useful.   If it results in no change to the document, perhaps not so useful, or perhaps the document was just done.  I think it’s okay…

Not going to happen. We need the list of those eligible to be generated by a script that the Secretariat can run. I don't think that we know how to write such a script if the word "useful" is part of the criteria.

> Tracking social capital is a hard problem, but it’s actually the problem you’re trying to solve.  The idea that the most valuable contribution a person makes to the IETF is showing up in person is…

No, we're talking about how to fix the problem that our current criterion is now broken, and will become more broken as f2f meetings become rarer, while retaining the property that the criteria are objective and can be mechanically verified.
 
> While we’re at it, implementing a draft and keeping the implementation up to date with the version that becomes RFC should give you at least as much credit as being an author on an RFC, because oth…

Well, not really, if the IETF Last Call process is of any value. But anyway, implementation is not something that a Secretariat script can verify, so it's useless for present purposes.

   Brian

>> On Mon, Mar 16, 2020 at 8:31 PM Ted Lemon <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>     > On Mar 16, 2020, at 7:33 PM, Warren Kumari <[email-masked] <mailto:[email-masked]>> wrote:
>>     >
>>     > [ no-hats ]
>>     >
>>     >> Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:
>>     >> Michael Richardson <[email-masked] <mailto:[email-masked]>>, Sandelman Software Works

---

From: Ted Lemon <[email-masked]>
Date: Mon, 16 Mar 2020 21:37:55 -0400

> Not going to happen. We need the list of those eligible to be generated by a script that the Secretariat can run. I don't think that we know how to write such a script if the word "useful" is part…

I get that, but this is a feedback loop.  Behaviors that are selected for will be amplified.  Behaviors that are not selected for will be suppressed.  My concern is that the behaviors you’re selecting for aren’t necessarily the ones that, if amplified, would produce a good outcome.

We already have a problem with RFCs being published that don’t come from real-world experience and have the potential to be harmful if implemented; the reason they don’t cause harm is because when implemented, it’s clear that they are a bad idea, and so they’re dropped.  I’m not going to name names because I have my favorites and you probably have an intersecting but not congruent set, but I think you know what I mean.

Do we really want to amplify this phenomenon?

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 18 Mar 2020 13:51:18 +1300

Ted,

> Do we really want to amplify this phenomenon?

No, but do we really want to amplify the existing phenomenon of people attending f2f meetings who have nothong to contribute, but therefore become NomCom-qualified? I doubt if there is any way to avoid this problem.

The same goes for the comment about bogus reviews and reviewers. Yes, it's a problem. But pretty much any measurable criterion is subject to some sort of gaming. And if it's not measurable, the Secretariat can't apply it. So I think we're stuck with this problem whatever we do**.

(I personally believe that one of the bus companies in Auckland games its performance metric of on-time arrival by dispatching the buses on frequent routes ahead of time. This is a universal problem, not ours alone.)

**Imagine for example that we analyzed all email on IETF lists for the last 3 years, and declared that the volunteer pool was the 500 most frequent "To" and "CC" individuals. I think we can all see how to game that.

    Brian

---

From: John C Klensin <[email-masked]>
Date: Tue, 17 Mar 2020 23:17:42 -0400

--On Tuesday, March 17, 2020 14:21 +1300 Brian E Carpenter
<[email-masked]> wrote:

>...
> No, we're talking about how to fix the problem that our
> current criterion is now broken, and will become more broken
> as f2f meetings become rarer, while retaining the property
> that the criteria are objective and can be mechanically
> verified.  
>...

It also becomes more broken as the percentage of contributing (I
don't have a good way to measure that either, but some of the
surrogates for that which you've mentioned haven't worked)
remote participants rises.  I think that may be inevitable even
if we manage to hold the next several meetings nominally f2f:
(i) the recent economic shocks are not going to vanish, and
vanish from memory, overnight and it is almost inevitable that
travel funding will become harder to justify in some
organizations and (ii) some contributors may find, after next
week and a lot of interim meetings that showing up in person
isn't nearly as important for whatever they expect to contribute
to and get out of the IETF as they thought it was.

IMO, not just nomcom eligibility but the survival of the IETF as
an effective body may require getting serious about remote
participants and effective remote participation.  If we do, many
of the nomcom eligibility issues, including some of the issues
that have come up in the last few days, may be a lot more clear
in a year or two.   If we don't, well, it will be a different
IETF with a lot less breath of perspectives.

    best,
    john

---

From: Ted Lemon <[email-masked]>
Date: Tue, 17 Mar 2020 23:27:23 -0400

> No, but do we really want to amplify the existing phenomenon of people attending f2f meetings who have nothong to contribute, but therefore become NomCom-qualified? I doubt if there is any way to a…

I suggested some ways, which you asserted without evidence were too hard.

> The same goes for the comment about bogus reviews and reviewers. Yes, it's a problem. But pretty much any measurable criterion is subject to some sort of gaming. And if it's not measurable, the Sec…

There is no algorithm that can make this work. It requires people, and judgment. The algorithm is always part of the equation, but it can’t be the whole equation.

> (I personally believe that one of the bus companies in Auckland games its performance metric of on-time arrival by dispatching the buses on frequent routes ahead of time. This is a universal proble…

This is an example of something that, if a feedback opportunity is present, can be corrected for.  The reason it’s a problem is that there’s no feedback mechanism.

> **Imagine for example that we analyzed all email on IETF lists for the last 3 years, and declared that the volunteer pool was the 500 most frequent "To" and "CC" individuals. I think we can all see…

That’s an absurd suggestion which I did not make.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 18 Mar 2020 16:49:23 +1300

Ted,
On 18-Mar-20 16:27, Ted Lemon wrote:
> On Mar 17, 2020, at 8:51 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:

It's very possible that I missed some parts of what you sent in the flood of email. Feel free to repeat yourself.
 
> There is no algorithm that can make this work. It requires people, and judgment. The algorithm is always part of the equation, but it can’t be the whole equation.

Well, here we disagree. However desirable judgment is, it doesn't scale, except possibly by voting. So I don't think there is scope for non-algorithmic criteria.

> This is an example of something that, if a feedback opportunity is present, can be corrected for.  The reason it’s a problem is that there’s no feedback mechanism.

And our feedback mechanism (confidential comments to NomCom) has a time constant that makes it basically useless.

> That’s an absurd suggestion which I did not make.

No, I made it, because I was trying to think of something that is algorithmic and allows for some sort of measure of peer assessment. Of course it's absurd, but then PageRank is a pretty absurd way of measuring the *quality* of web pages.

Regards
   Brian

---

From: S Moonesamy <[email-masked]>
Date: Wed, 18 Mar 2020 01:12:44 -0700

Hi Ted,
At 08:27 PM 17-03-2020, Ted Lemon wrote:
>>No, but do we really want to amplify the existing phenomenon of 
>>people attending f2f meetings who have nothong to contribute, but 
>>therefore become NomCom-qualified? I doubt if there is any way to 
>>avoid this problem.
>
>>The same goes for the comment about bogus reviews and reviewers. 
>>Yes, it's a problem. But pretty much any measurable criterion is 
>>subject to some sort of gaming. And if it's not measurable, the 
>>Secretariat can't apply it. So I think we're stuck with this 
>>problem whatever we do**.
>
>There is no algorithm that can make this work. It requires people, 
>and judgment. The algorithm is always part of the equation, but it 
>can't be the whole equation.

I was looking for some information to illustrate a point.  I picked 
some historical information as it might be uncontroversial.  Ericsson 
and Futurewei volunteers were disqualified because the maximum number 
of selected volunteers by affiliation was reached.  I looked up 
previous years and I found out that there were at least one NomCom 
member from one of those companies.  There are probably other 
companies in the same situation.  The point is that the system which 
was in place since 2002 or before is not disadvantageous to such companies.

At the other end, there are people who buy into the "anyone can 
participate" argument.  Those people are not eligible for NomCom 
because they don't get corporate sponsorship.

It is implausible that a significant number of participants would 
support a change to a rule which would put their companies at a 
disadvantage.  This is where we get into the "gaming" and other 
arguments when a change is proposed.

Warren posted a comment on the thread about directorate reviews.  I 
used to do that type of work.  I could see which reviewers were 
submitting a "low-quality" reviews.  Most of the criteria I could 
think of ends up amplifying systemic issues.  As you mentioned above, 
we remain stuck with a problem.

The probability that the existing nomination process works well this 
year is low because the pandemic caused several countries to close 
their borders.  One alternative around that is a change to the 
rules.  Another alternative is to ignore the violation of the rules.

The NomCom volunteers are expected to be familiar with the IETF 
processes and procedures.    That expectation cannot be met through a 
mechanical process.

Regards,
S. Moonesamy

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 09:15:13 -0400

> It's very possible that I missed some parts of what you sent in the flood of email. Feel free to repeat yourself.

See below.

> Well, here we disagree. However desirable judgment is, it doesn't scale, except possibly by voting. So I don't think there is scope for non-algorithmic criteria.

Getting an RFC published is not a mechanical process.  It is an exercise in judgment, from start to finish.  It can be gamed, and we’ve seen it gamed.  But be that as it may, it’s an exercise in judgment, which you are proposing to then mechanize as a criterion for nomcom eligibility.

I don’t disagree with the idea of mechanizing this, but I think that, like the in-person mechanization, it has problems, and I wrote a series of suggestions for things we could do similar to “publish an RFC” that would be worth doing regardless if whether you mechanized them as you have proposed doing for RFC publication, and that would enfranchise a lot of IETF participants who currently aren’t enfranchised, and who deserve to be.

I would appreciate it if you could recall that the RFC process involves judgment, and hence is not purely algorithmic, and admit to the possibility that there are other things that are similar.

Can we actually do all the things I proposed?  Would they actually make things better?   I don’t know, but it was pretty rough of you to just belittle them as unrealistic when compared to your own, similar thing the only advantage of which is that a process for it already exists.

The message in which I described these was previous to the one you replied to, to which this is a reply. I’m sure you can find it.

> No, I made it, because I was trying to think of something that is algorithmic and allows for some sort of measure of peer assessment. Of course it's absurd, but then PageRank is a pretty absurd way…

I think the problem with this proposal was simply that it selects for something we don’t want: more noise on the mailing lists.

It’s not hard to track peoples’ contributions.  We’ve seen people try to game the “acknowledgements” process; when someone actively tried to game that a few years back, there was some pushback from people who found his “helpful reviews” completely unhelpful, while others just did as he asked.  If it were a nomcom criterion, I think he would have been shut down hard, and would have gotten zero acknowledgments.

Requiring working group chairs and authors to seriously consider whether someone’s contributions to their effort were helpful, and giving people credit for helpful contributions, would not be that hard, would not be perfect, but would definitely select for something we want: more constructive participation.

I don’t think this is that hard, and I was serious in suggesting it.

Similarly with implementation, if someone is implementing IETF standards and feeding back the results of that implementation, it’s not that hard to take note of this.  It would require some tooling, and some process.   Again, this is a serious suggestion, and I think it’s practical both in terms of what it incentivizes, and also in terms of its effect on the quality of the nomcom selection pool.

It’s true that both of these suggestions require more work than what you are proposing.   But what you are proposing creates a reward for doing something we don’t want: publishing useless RFCs.  If you really want to do that, then we need the working groups and the IESG to be more disciplined about publishing stuff that won’t get used, or that is harmful but will be (mostly) ignored.

The other problem with your proposal is that publishing RFCs is hard, and we have limits on how many authors there can be on an RFC, /precisely because we don’t want to valorize participation in name only/.

And so the number of people that this rule would enfranchise is always going to be much smaller than the number who will be enfranchised by the in-person rule, and hence it won’t really make any difference, on its own, except to a few individuals who will no doubt appreciate it.  It’s great from the perspective of fairness, but it doesn’t improve the /IETF/.

If you want to enfranchise people for participation (which  I support!), RFC publication is too high a bar.

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 18 Mar 2020 13:40:27 +0000

Hi Ted,

On 18/03/2020 13:15, Ted Lemon wrote:
> If you want to enfranchise people for participation (which  I
> support!), RFC publication is too high a bar.
I think qualifying via path#5 from the draft is a
pretty reasonable indicator for having been an active
IETF participant.

More generally, ISTM the disconnect between you and
Brian is that he (and I) envisage that the secretariat
remain the ones who check if someone is nomcom-eligible
or not, whereas you seem to be envisaging a judgement
call being involved in that, and we don't ask the folks
from the secretariat to make such calls, for good
reasons.

So, if e.g. you want "effective WG participation" to
provide a route to being nomcom-eligible, then you'd
have to have someone (a WG chair or AD?) check a box
for such a participant or something like that. I think
I'd be less comfortable with criteria like that than
with ones that could be evaluated by the secretariat.

However, if this draft progresses, I do agree it would
make sense to establish early on whether or not there's
rough consensus on this point, i.e. we should ask the
question "should new criteria be such that it's ok for
the secretariat to evaluate  'em."

Cheers,
S.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 18 Mar 2020 09:48:32 -0400

Personally, I can not imagine any nomcom-eligibility process that is not 
essentially algorithmic.  The consequences of a process that requries 
that someone make judgments about whether individuals qualify for nomcom 
strikes me as a disaster on many levels.  And even worse if it is for 
purposes of recalls.

One can, as currently, disagree with the algorithmic evaluation because 
some of the data is wrong.  But the data is (reasonably) verifiable. 
Getting into arguments about judgment calls?  Please, no.

Yours,
Joel

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 09:54:15 -0400

Joel, I believe you and Stephen have completely ignored my point that “publishing an RFC” is a judgement call and not up to the secretariat.  “Published RFCs” represent a corpus of data that can be harvested by the secretariat to determine eligibility.

I am suggesting not that we offer WG chairs a button they can push to make someone nomcom eligible, but an opportunity to build an additional, hopefully useful, corpus of data about constructive participation.

I agree that this is would require non-trivial amounts of work.  I do not find it helpful, however, that both you and Stephen (and Brian) are essentially ignoring the fact that what I am proposing is no different than what Brian is proposing with the exception that my proposal would require new process.

Specifically it is not different in terms of requiring judgment calls from IETF leadership, including working group chairs and the IESG.

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 18 Mar 2020 14:01:43 +0000

Hiya,

On 18/03/2020 13:54, Ted Lemon wrote:
> Joel, I believe you and Stephen have completely ignored my point that
> “publishing an RFC” is a judgement call and not up to the
> secretariat.  

I didn't ignore it (honest!), I don't agree with you is
all:-)

Yes there are judgement calls involved in the process of
publishing an RFC, but there are (almost) none involved in
the process of checking if Ted Lemon published some RFCs
in the last N years.

> “Published RFCs” represent a corpus of data that can be
> harvested by the secretariat to determine eligibility.

Right.

> 
> I am suggesting not that we offer WG chairs a button they can push to
> make someone nomcom eligible, but an opportunity to build an
> additional, hopefully useful, corpus of data about constructive
> participation.
> 
> I agree that this is would require non-trivial amounts of work.  

I'd need to see a more concrete proposal tbh. If such a
corpus had other uses and came into being then maybe it'd
be ok to re-use for nomcom eligibility. I can't see such
a thing being created and maintained just for this reason
though. And especially not in the short term, and we do
also have a short term need here.

> I do
> not find it helpful, however, that both you and Stephen (and Brian)
> are essentially ignoring the fact that what I am proposing is no
> different than what Brian is proposing with the exception that my
> proposal would require new process.

Again, I'm not ignoring it, I'm disagreeing with you:-) What you are
proposing definitely seems qualitatively
different to me from what's in the draft.

Cheers,
S.

> 
> Specifically it is not different in terms of requiring judgment calls
> from IETF leadership, including working group chairs and the IESG.
>

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 11:00:00 -0400

> the process of checking if Ted Lemon published some RFCs

This is what I mean by “ignoring it.”  When you make this assertion, you are ignoring what I said, not disagreeing with what I said.

Yes, looking at the output of a process that is entirely based on judgment calls can be entirely mechanical.   I agree with you completely on this.  What I am suggesting is not that we add a new judgment-based process for looking at the output of some other process.  I agree that the way to look at the output of the process is by mechanically iterating across its output, not by having humans look at its output and make decisions.

> also have a short term need here.

Again, as I said in my original message about this, the benefit to doing this is that it improves the IETF.   The ability to use that improvement to improve the quality of the nomcom selection pool is an added bonus, and the /fact/ that the output of this process would affect the nomcom selection pool is a way to get people to take it seriously instead of phoning it in, and a meaningful basis for providing feedback when people who should be taking it seriously fail to do so.

The proposal is that people who give /useful/ feedback should get credit for doing so.  “Useful” is a judgment call for feedback just as it is for potential RFCs, but it’s not hard.   I think that document editors and working group participants should have some say in what constitutes useful, as should working group chairs.  As a document author, I know what useful feedback is.  If I am small-minded, I might mark feedback I don’t agree with as “useless,” and that’s where the working group chair can step in.  The person giving the feedback will be motivated to ask for this intercession.  Similarly, as we saw some years ago with the “acknowledge me” kerfuffle, there was no confusion about whether those contributions were useful, and the person who was complaining got the IESG involved, and this was handled perfectly effectively through normal IETF process.

The datatracker already tracks directorate feedback.  Brian has already proposed using this data corpus to evaluate participation.  I’m simply suggesting that we do something similar for working groups.   Several people have pointed out that directorate reviews sometimes aren’t useful.  So we have the same problem with directorate reviews that we would have with draft comments in working groups.  We should do something about that, and not just bemoan it.

We can have a similar process for IETF review, which at present is kind of useless.  The responsible AD for a draft would be responsible for having the last word on whether a contribution at that stage was useful. Tracking IETF last call comments is something that is not systematically done right now, and I think that’s part of why we have so much noise on the list when something is controversial: people give feedback but don’t feel heard, so they repeat themselves over and over again.

As for implementation, the main difference with implementation is that it can happen after an RFC is published, and implementors who implement RFCs still ought to get credit for participating in the IETF.  Again, this doesn’t need to be much different than the directorate process we have now; indeed, the directorates could do the work of evaluating such contributions and advising the appropriate ADs on whether credit should apply for the work done.

The point of this is not to do new things.  It is to keep better track of the things we are doing, and do them better and more effectively, and as an extremely valuable side effect, use that to improve the quality of the nomcom selection pool.  If you agree that directorate reviews are an appropriate input to the nomcom eligibility decision, I don’t see why these other important IETF participation measures shouldn’t be as well.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 18 Mar 2020 11:14:16 -0400

1) It was completely unclear in your earlier email that what you were 
proposing was a new judgment and recording process earlier in the cycle, 
where chairs would try to record who had made useful contributions.

2) I think your proposed recording is a task that will be inherently 
uneven across chairs and very prone to unintended bias.  It strikes me 
as a very bad idea.  Even dedicated chairs will likely be uneven in 
applying this.  (As a side-effect of it being a bad idea, it becomes a 
bad idea as an input to nomcom eligibility.)

Yours,
Joel

On 3/18/2020 11:00 AM, Ted Lemon wrote:
> On Mar 18, 2020, at 10:01 AM, Stephen Farrell <[email-masked] 
> <mailto:[email-masked]>> wrote:
> This is what I mean by “ignoring it.”  When you make this assertion, you 
> are ignoring what I said, not disagreeing with what I said.
> 
> Yes, looking at the output of a process that is entirely based on 
> judgment calls can be entirely mechanical.   I agree with you completely 
> on this.  What I am suggesting is not that we add a new judgment-based 
> process for looking at the output of some other process.  I agree that 
> the way to look at the output of the process is by mechanically 
> iterating across its output, not by having humans look at its output and 
> make decisions.
> 
> Again, as I said in my original message about this, the benefit to doing 
> this is that it improves the IETF.   The ability to use that improvement 
> to improve the quality of the nomcom selection pool is an added bonus, 
> and the /fact/ that the output of this process would affect the nomcom 
> selection pool is a way to get people to take it seriously instead of 
> phoning it in, and a meaningful basis for providing feedback when people 
> who should be taking it seriously fail to do so.
> 
> The proposal is that people who give /useful/ feedback should get credit 
> for doing so.  “Useful” is a judgment call for feedback just as it is 
> for potential RFCs, but it’s not hard.   I think that document editors 
> and working group participants should have some say in what constitutes 
> useful, as should working group chairs.  As a document author, I know 
> what useful feedback is.  If I am small-minded, I might mark feedback I 
> don’t agree with as “useless,” and that’s where the working group chair 
> can step in.  The person giving the feedback will be motivated to ask 
> for this intercession.  Similarly, as we saw some years ago with the 
> “acknowledge me” kerfuffle, there was no confusion about whether those 
> contributions were useful, and the person who was complaining got the 
> IESG involved, and this was handled perfectly effectively through normal 
> IETF process.
> 
> The datatracker already tracks directorate feedback.  Brian has already 
> proposed using this data corpus to evaluate participation.  I’m simply 
> suggesting that we do something similar for working groups.   Several 
> people have pointed out that directorate reviews sometimes aren’t 
> useful.  So we have the same problem with directorate reviews that we 
> would have with draft comments in working groups.  We should do 
> something about that, and not just bemoan it.
> 
> We can have a similar process for IETF review, which at present is kind 
> of useless.  The responsible AD for a draft would be responsible for 
> having the last word on whether a contribution at that stage was useful. 
> Tracking IETF last call comments is something that is not systematically 
> done right now, and I think that’s part of why we have so much noise on 
> the list when something is controversial: people give feedback but don’t 
> feel heard, so they repeat themselves over and over again.
> 
> As for implementation, the main difference with implementation is that 
> it can happen after an RFC is published, and implementors who implement 
> RFCs still ought to get credit for participating in the IETF.  Again, 
> this doesn’t need to be much different than the directorate process we 
> have now; indeed, the directorates could do the work of evaluating such 
> contributions and advising the appropriate ADs on whether credit should 
> apply for the work done.
> 
> The point of this is not to do new things.  It is to keep better track 
> of the things we are doing, and do them better and more effectively, and 
> as an extremely valuable side effect, use that to improve the quality of 
> the nomcom selection pool.  If you agree that directorate reviews are an 
> appropriate input to the nomcom eligibility decision, I don’t see why 
> these other important IETF participation measures shouldn’t be as well.
> 
>

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 18 Mar 2020 15:16:27 +0000

Hiya,

>> I'd need to see a more concrete proposal tbh. If such a corpus had
>> other uses and came into being then maybe it'd be ok to re-use for
>> nomcom eligibility. I can't see such a thing being created and
>> maintained just for this reason though. And especially not in the
>> short term, and we do also have a short term need here.
> 
> Again, as I said in my original message about this, the benefit to
> doing this is that it improves the IETF.   The ability to use that
> improvement to improve the quality of the nomcom selection pool is an
> added bonus, and the /fact/ that the output of this process would
> affect the nomcom selection pool is a way to get people to take it
> seriously instead of phoning it in, and a meaningful basis for
> providing feedback when people who should be taking it seriously fail
> to do so.
> 
> The proposal is that people who give /useful/ feedback should get
> credit for doing so.  “Useful” is a judgment call for feedback just
> as it is for potential RFCs, but it’s not hard.   I think that
> document editors and working group participants should have some say
> in what constitutes useful, as should working group chairs.  As a
> document author, I know what useful feedback is.  If I am
> small-minded, I might mark feedback I don’t agree with as “useless,”
> and that’s where the working group chair can step in.  The person
> giving the feedback will be motivated to ask for this intercession.
> Similarly, as we saw some years ago with the “acknowledge me”
> kerfuffle, there was no confusion about whether those contributions
> were useful, and the person who was complaining got the IESG
> involved, and this was handled perfectly effectively through normal
> IETF process.
> 
> The datatracker already tracks directorate feedback.  Brian has
> already proposed using this data corpus to evaluate participation.
> I’m simply suggesting that we do something similar for working
> groups.   Several people have pointed out that directorate reviews
> sometimes aren’t useful.  So we have the same problem with
> directorate reviews that we would have with draft comments in working
> groups.  We should do something about that, and not just bemoan it.
> 
> We can have a similar process for IETF review, which at present is
> kind of useless.  The responsible AD for a draft would be responsible
> for having the last word on whether a contribution at that stage was
> useful. Tracking IETF last call comments is something that is not
> systematically done right now, and I think that’s part of why we have
> so much noise on the list when something is controversial: people
> give feedback but don’t feel heard, so they repeat themselves over
> and over again.
> 
> As for implementation, the main difference with implementation is
> that it can happen after an RFC is published, and implementors who
> implement RFCs still ought to get credit for participating in the
> IETF.  Again, this doesn’t need to be much different than the
> directorate process we have now; indeed, the directorates could do
> the work of evaluating such contributions and advising the
> appropriate ADs on whether credit should apply for the work done.
> 
> The point of this is not to do new things.  It is to keep better
> track of the things we are doing, and do them better and more
> effectively, and as an extremely valuable side effect, use that to
> improve the quality of the nomcom selection pool.  If you agree that
> directorate reviews are an appropriate input to the nomcom
> eligibility decision, I don’t see why these other important IETF
> participation measures shouldn’t be as well.

So just to check I understand, the above would imply that
we'd have a "useful" or "like" button in the datatracker
and maybe the mail archive tool that various people could
push relating to various artefacts, and that some scoring
system would exist with a threshold above which a person
becomes nomcom-eligible. (And maybe with a time-decay as
part of the scoring.)

I guess such a thing could be done. If it had existed for
a while and been shown to be useful and not to have been
gamed (for fun or profit:-) then I'd not entirely be
opposed to it being a path to nomcom-eligibility. However,
I just don't see that being done in the next year, nor
can I see it being done solely as part of a change to
nomcom-eligibility, so I'd be against aiming for this
for now.

And since you asked, yes, I'm also leery of the path#3
directorate reviews part of the draft - I think I agree
with Warren that that would be better omitted. If, OTOH,
lots of people wanted to keep it in, I could live with
it.

Cheers,
S.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Mar 2020 15:19:57 +0000

How about ICANN or ISOC or W3C or someone(s) get to appoint someone to nomcom.

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 18 Mar 2020 15:23:33 +0000

> How about ICANN or ISOC or W3C or someone(s) get to appoint someone to nomcom.

You mean they get to directly appoint a voting member or
that they get to make someone eligible for random
selection as a voting member?

I think I dislike the idea either way though;-) It seems
to add risk and complexity without much benefit.

S.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Mar 2020 15:26:19 +0000

There are three options: non-voting, voting, or "throw them into the pool."  I didn't express a preference.

If we're interested in making nomcom more representative, so that the IETF leadership can be more representative, then adding outsiders to process, those who we work with, makes sense to me.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 11:28:16 -0400

On Mar 18, 2020, at 11:14 AM, Joel M. Halpern <[email-masked]> wrote:
> 2) I think your proposed recording is a task that will be inherently uneven across chairs and very prone to unintended bias.  It strikes me as a very bad idea. Even dedicated chairs will likely be uneven in applying this.  (As a side-effect of it being a bad idea, it becomes a bad idea as an input to nomcom eligibility.)

And of course RFC publication isn’t prone to similar bias?

And attendance isn’t prone to massive bias?

Best is the enemy of good enough, Joel.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 18 Mar 2020 11:32:24 -0400

Rich, I think I am not understanding something about your idea.  Our 
normal assumption is that what is important for nomcom is caring about 
the IETF and having some understqnding of the likely impact of selecting 
people.  I do not see how that leads to wanting to have the W3C appoint 
someone to the nomcom pool.

Yours,
Joel

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 11:34:58 -0400

> nomcom-eligibility, so I'd be against aiming for this

If not now, then when?  I think we should re-evaluate it after a while, but it seems no less useful than an in-person head count, and it requires effort and collusion to game it, whereas gaming attendance requires only a modest amount of money. 

The main motivation for the current changes, particularly if you exclude directorate reviews, seems to be to make sure that people who’ve written RFCs recently aren’t disenfranchised.  It will have very little benefit in terms of improving the quality of the pool, because in-person attendees will still outnumber editors of RFCs, and of course we’re only enfranchising editors, not everyone else who worked hard on the document that was published, some of whom, as a matter of policy (the five editor rule), are excluded now.

I agree that this is a population we shouldn’t disenfranchise, but we’re still going to be disenfranchising a lot of important participants if that’s the /only/ change we make.

I think this should be very concerning for all involved.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Mar 2020 15:36:43 +0000

If we want to make the IETF more representative (and perhaps that's not a universal view), then the pool of people doing the picking should be more representative.  If the nomcom is limited to those who are very well acquainted with the IETF, then it is hard to see how we get leadership that is outside the norms.

We want people who care about the IETF to appoint the leadership of the IETF. I am suggesting that not all people who care are actual IETFers. But even if that is wrong, hearing from others about who should be involved in the leadership seems like a good idea, to me.

>  I do not see how that leads to wanting to have the W3C appoint someone to the nomcom pool.

Does that help?

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 11:37:07 -0400

On Mar 18, 2020, at 11:32 AM, Joel M. Halpern <[email-masked]> wrote:
> Rich, I think I am not understanding something about your idea.  Our normal assumption is that what is important for nomcom is caring about the IETF and having some understqnding of the likely impact of selecting people.  I do not see how that leads to wanting to have the W3C appoint someone to the nomcom pool.

Indeed, if other SDOs want in the pool, their members can attend or participate.  I don’t think we want to encourage /anyone/ being added to the pool who does neither.  There are people in the IETF leadership now who work for ICANN, and active W3C participants who are also in the pool.   I don’t think we have a problem here.

If we do valorize productive feedback, it’s likely that members of these SDOs would be eligible on that basis anyway.

---

From: Joel Halpern Direct <[email-masked]>
Date: Wed, 18 Mar 2020 11:46:27 -0400

That does explain the thought process better.

I don't think I agree with the premises.  I am not sure which regards we 
want to make IETF "more representative"?  I personally would like to get 
more operators involved.  That does not mean I think that the NOGs 
should get to appoint nomcom members.

On the other hand, I do think that the leadership should be 
representative of the folks that are actually doing the work and 
participating in the IETF.  Which is why I would like to find an 
effective way to enable remote participants to be in the nomcom pool.

Separately, I would like to make sure that the IETF is as effectively 
inclusive and encouraging of technical participation as possible.  While 
that probably does mean that the leadership could / should look outside 
for information as to why folks do not participate, it does not seem to 
me to lead to letting outside groups that do not participate in the IETF 
appoint folks to the IETF leadership.

Yours mileage may differ,
Joel

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 18 Mar 2020 15:52:47 +0000

Hiya,

On 18/03/2020 15:34, Ted Lemon wrote:
>> I guess such a thing could be done. If it had existed for a while
>> and been shown to be useful and not to have been gamed (for fun or
>> profit:-) then I'd not entirely be opposed to it being a path to
>> nomcom-eligibility. However, I just don't see that being done in
>> the next year, nor can I see it being done solely as part of a
>> change to nomcom-eligibility, so I'd be against aiming for this for
>> now.
> 
> If not now, then when?  

When there's a different stand-alone reason to want it
done that justifies the effort.

> I think we should re-evaluate it after a
> while, but it seems no less useful than an in-person head count, and
> it requires effort and collusion to game it, whereas gaming
> attendance requires only a modest amount of money.
> 
> The main motivation for the current changes, particularly if you
> exclude directorate reviews, seems to be to make sure that people
> who’ve written RFCs recently aren’t disenfranchised. 

I don't think you mean motivation there do you? I'm
motivated to help with this both because of the current
situation and because the current criteria are IMO
unfair, and I'd like if we improve that. So I guess
you mean "effect of" rather than "motivation for"?

> It will have
> very little benefit in terms of improving the quality of the pool,
> because in-person attendees will still outnumber editors of RFCs, and
> of course we’re only enfranchising editors, not everyone else who
> worked hard on the document that was published, some of whom, as a
> matter of policy (the five editor rule), are excluded now.
> 
> I agree that this is a population we shouldn’t disenfranchise, but
> we’re still going to be disenfranchising a lot of important
> participants if that’s the /only/ change we make.

How do you know it's "a lot"? The population concerned are
those that'd not qualify via one of the paths in the draft
and that'd want to volunteer for nomcom and where we think
there's a good enough chance they'd be good at being nomcom
voting members. I don't know that that's "a lot." I do
agree we should try calculate some lists of names and sets
of numbers for who'd be eligible or not according to new
criteria before we're done though, so we could each see
if we think the net effect is ok.

Cheers,
S.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Mar 2020 15:53:22 +0000

> it does not seem to 
  
I wasn't saying the other folks appoint people.  (Indication of some bias here?)  In particular, I didn't express an opinion about the degree of their involvement and mentioned three possibilities.
  
>    Yours mileage may differ,
 
Reasonable [people] may disagree.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 12:16:47 -0400

> done that justifies the effort.

I gave a clear reason.

> you mean "effect of" rather than "motivation for"?

This doesn’t offer a particularly meaningful improvement, except to a certain group of people whom I agree are potentially treated unfairly.  It’s a good thing to do, but I don’t expect it to make any practical difference to the makeup of nomcom, because it affects too few people.

> if we think the net effect is ok.

Well, indeed, we don’t know how big it is.  But suppose attendance drops to 300 per meeting, but participation remains the same.   “A lot” is now 700 people, minus those who have personally been listed as an author on an RFC in the past three years.  That’s definitely “a lot.”   And we aren’t even counting the people who are currently disenfranchised because they can’t attend meetings that involve flying halfway around the world and staying in hotels for a week.   This has been an ongoing problem.  I’m proposing that we seriously consider fixing it.  I do not mean to impugn your motivation.  I’m just saying, we have an opportunity here to do more, and I don’t think “we’re in a hurry” is a reason not to try to do better than what’s been proposed thus far.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 18 Mar 2020 15:49:08 -0400

> system. It also somehow feels like "I'll gladly pay you Tuesday for a

I would prefer a rather generous system that counted from many paths, but had
a limit from each path.  The goal would be to enfranchise enough people
easily such that anyone who was really 47.5/50 probably wasn't connected
enough to the IETF to even figure out how to game things.

"You must be <this> tall", where the height restriction is 57cm, at a
convention of tall people where the shortest is 162cm.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 18 Mar 2020 15:56:56 -0400

Ted Lemon <[email-masked]> wrote:
    > credit for doing so.  “Useful” is a judgment call for feedback just as
    > it is for potential RFCs, but it’s not hard.   I think that document
    > editors and working group participants should have some say in what
    > constitutes useful, as should working group chairs.  As a document
    > author, I know what useful feedback is.  If I am small-minded, I might
    > mark feedback I don’t agree with as “useless,” and that’s where the
    > working group chair can step in.  The person giving the feedback will
    > be motivated to ask for this intercession.  Similarly, as we saw some
    > years ago with the “acknowledge me” kerfuffle, there was no confusion
    > about whether those contributions were useful, and the person who was
    > complaining got the IESG involved, and this was handled perfectly
    > effectively through normal IETF process.

If each path to becoming eligible is restricted to be at most 40% (or so) of
eligibility, and getting enough "useful" points only takes 5-6 contributions,
then there is little reason to try to game this direction.

    > contributions and advising the appropriate ADs on whether credit should

So, I'd add filing errata (that is accepted).

    > of the things we are doing, and do them better and more effectively,
    > and as an extremely valuable side effect, use that to improve the
    > quality of the nomcom selection pool.  If you agree that directorate
    > reviews are an appropriate input to the nomcom eligibility decision, I
    > don’t see why these other important IETF participation measures
    > shouldn’t be as well.

---

From: Warren Kumari <[email-masked]>
Date: Wed, 18 Mar 2020 18:05:19 -0400

> So, I'd add filing errata (that is accepted).

So, one thing to keep in mind (for any and all of the metrics) is the
load that this puts on people / moves around.

https://www.rfc-editor.org/errata_search.php?rfc=5841
https://www.rfc-editor.org/errata_search.php?rfc=5513
https://www.rfc-editor.org/errata_search.php?rfc=3514

Are these errata? Yes. Are they accepted? Yes. Are they something that
we should reward? Dunno.
Looking through my snippets I've dealt with 90+ errata[0] - and
probably < 10 of these were actually substantive (likely to cause
confusion / implementation issues)

I personally don't care, but I keep hearing concerns from the
community that we cannot get enough people to run for AD positions
"because the workload is too high" and so the nomcom / community
doesn't have a good slate to choose from.
Having people do the equivalent of  'cat rfc*.txt | spell | mail
errata@' is not likely to make things better.

Processing Errata is an incredibly important part of being an AD, and
one of the few responsibilities that I take really seriously. They
almost always fall into two camps:
1: editorial things that take a few minutes to deal with (and often
feel like a waste of time)
or
2: places where considerable research and judgement, often with
following up with the WG to determine what the right answer is.
Sadly, the difference is not always obvious...

The Hold for Document Update vs Verified outcome also requires
considerable judgement, as does "is this actually an errata? Does it
change whatever the consensus was?".
It is also one of the few places where an AD can (presumably
accidentally) change the meaning of a published RFC.

Please let's tread carefully here and make sure that the incentives
work out correctly...

W

[0]: Actually. many Errata require a bunch of reading and research /
asking the WG / etc, and so the number of errata is less and this
counts followup / etc
E.g:
https://mailarchive.ietf.org/arch/msg/netmod/mjZWBQKHRMrn5dwGkoLTp0dzqkk/
https://mailarchive.ietf.org/arch/msg/bmwg/KprZwLwehbSMRqz_vdv9aQ4fxQY/

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 18:09:30 -0400

Yeah, I’m not convinced I’d want to incentivize errata—I do not remember errata processing fondly.  That said, Warren, why don’t you ask your directorate to take on some of the investigation work?

Sent from my iPad

> On Mar 18, 2020, at 6:05 PM, Warren Kumari <[email-masked]> wrote:
>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 19 Mar 2020 11:56:38 +1300

Hi,

Can we discuss Rich's idea separately please?

We have a short term problem caused by the probability of more meeting cancellations, which is why we have a draft on next week's gendispatch agenda. Adding non-IETF-qualified people to the volunteer pool sounds like a long-term proposal. Write a draft?

Regards
   Brian Carpenter

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 19 Mar 2020 12:02:15 +1300

Ted et al,

Yes, in the longer term, adding completely new mechanisms that solve
some of the issues you raised is an interesting idea. I'd say it's
a good candidate for a process experiment proposal under BCP93.

Right now I don't think we have the luxury of time. We need to deal
with the probability of more cancellations before the next NomCom
cycle. That's the focus of the current draft: what could be done
and approved before IETF 108?

Regards
   Brian

On 19-Mar-20 04:34, Ted Lemon wrote:
> On Mar 18, 2020, at 11:16 AM, Stephen Farrell <[email-masked] <mailto:[email-masked]>> wrote:

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 19:15:36 -0400

> and approved before IETF 108?

I don’t think this reasoning is at all sound.  You are proposing some fairly significant changes to the rules that will benefit certain individuals but leave quite a few others out who are equally affected.

Why do you think that the extensive set of changes your draft proposes should not include the additional changes I suggested?   I understand that you feel that your changes are easily implemented, but they also weight certain types of participation very strongly, while completely ignoring other arguably more important types of participation.

This feels a bit like using a crisis to force an change for the worse.

---

From: Warren Kumari <[email-masked]>
Date: Wed, 18 Mar 2020 19:54:36 -0400

> Yeah, I’m not convinced I’d want to incentivize errata—I do not remember errata processing fondly.  That said, Warren, why don’t you ask your directorate to take on some of the investigation work?

My standard work flow is to open the errata report (for those who
would like to play along at home:
https://www.rfc-editor.org/errata_search.php?rec_status=2&presentation=table
), and deal with the ones that are easy. The ones which require
investigation I generally send to the appropriate working group, but
it turns out to be surprisingly hard to get WGs to provide feedback on
if the Errata is valid, etc.

I might give the directorate idea a whirl, but they are also somewhat
overloaded / have not been given the care and feeding that they
deserve, and so I am trying not to impose too much on them... Rob
Wilton (the new Management AD) and I are planning on soliciting more
directorate members, etc..

W

---

From: Michael Richardson <[email-masked]>
Date: Wed, 18 Mar 2020 20:02:17 -0400

> errata@' is not likely to make things better.

It certainly would be nice to find a way, independently of anything else to
dis-incenticize stupid reports.  I like how uservoice / stackexchange works.
We literally could use either one for this.
We could probably drive a stackexchange instance via OpenID login from DT.
(We'd have to pay them for this.  Ubuntu does this as far as I can see)

    > Sadly, the difference is not always obvious...

I'd really like to remove ADs from being in the critical path for errata.
I would like to see WG chairs (or a design team they create) do the triage,
and there be a twice yearly sit-down with the ADs to go through the list of
things they couldn't resolve.

---

From: Warren Kumari <[email-masked]>
Date: Wed, 18 Mar 2020 20:22:06 -0400

> things they couldn't resolve.

That sounds nice -- but, sadly, I don't think it is really feasible; I
recently sat in a WG meeting where the WG (including the chairs) was
going down the path of completely changing a protocol using an errata
- they had crafted text changing "implementations MUST NOT..."  into
"implementations SHOULD..." - now, I happen to fully agree that the
right answer was that "implementations SHOULD...", but an errata
really isn't the right way to do that and I had to step in and put a
stop to it (one of the very few times I've had to wave my AD hat
around).

Assuming the load doesn't change drastically it currently isn't
onerous -- what *would* help is if WGs and chairs would respond when
an Errata notice comes in, and suggest a resolution to the issue, but
the actual "monkey clicking button" bit seems to me like it should
remain with the AD...

W

---

From: Michael Richardson <[email-masked]>
Date: Wed, 18 Mar 2020 20:51:08 -0400

> recently sat in a WG meeting where the WG (including the chairs) was

okay, fair enough.

I think we take your point that submitted errata should not be added to the
"Path" until we have a way to deal with this.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 20:59:18 -0400

> if the Errata is valid, etc.

I would say that chair should not be eligible for nomcom based on Brian’s criteria.

I know that sounds a little facetious, but if we have working group chairs who aren’t interested in either doing or driving the work, maybe those aren’t the right chairs, or maybe they don’t have the right motivation.  If the working group isn’t able to evaluate errata, maybe you should just close it…

Sort of kidding, but also sort of serious.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 19 Mar 2020 15:20:06 +1300

Ted,

> I don’t think this reasoning is at all sound.  You are proposing some fairly significant changes to the rules that will benefit certain individuals but leave quite a few others out who are equally…

Well, we apparently disagree about the urgency. Clearly, what we do in the coming weeks is not for ever; it's to fix a problem created by meeting cancellations (Spain is now in lockdown, so the July meeting is surely also at risk). Under urgency, we can only choose criteria that can be extracted from the records currently held by the Secretariat and the tracker.
 
> Why do you think that the extensive set of changes your draft proposes should not include the additional changes I suggested?   I understand that you feel that your changes are easily implemented,…

Assuming you mean:

"The proposal is that people who give /useful/ feedback should get credit for doing so.  “Useful” is a judgment call for feedback just as it is for potential RFCs, but it’s not hard.   I think that document editors and working group participants should have some say in what constitutes useful, as should working group chairs."

I agree that this is an interesting approach, but it's one that would take some months to design and a year or more to achieve results, so I believe it's out of scope for our draft. I'd be interested to see a fully worked out proposal.

> This feels a bit like using a crisis to force an change for the worse.

I don't really see why it's worse, rather than different.

If you're arguing specifically against path "3: Has submitted at least 6 reviews as a member of an official IETF review team within the last 3 years" because there is no quality control on reviews, we can have that discussion. I would expect ADs to remove useless reviewers from review teams, but maybe that doesn't happen.

   Brian

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Mar 2020 22:33:03 -0400

The problem is that your proposal doesn’t satisfy its stated goals, and furthermore if IETF 108 doesn’t happen, then the set of people eligible for nomcom doesn’t change.  Let’s look at your goals:

  o  Mitigate the issue of active remote (or rarely in-person)
      participants being disenfranchised in the NomCom and recall
      processes.

The proposed changes don’t meaningfully do that.  Certainly not in the short term.  They prevent certain key participants being disenfranchised, sure, but that’s a relatively small percentage of the total participants who would be disenfranchised.  So you haven’t actually solved this problem.

   o  Prepare for an era in which face-to-face plenary meetings are less
      frequent (thus extending the issue to many, perhaps a majority, of
      participants).

You’re actively arguing against my proposal to mitigate this problem.

   o  Do this quickly in view of the risk of IETF 108 being cancelled
      too, but ensure that the result is flexible for the future.

It’s not clear why IETF 108 being cancelled disenfranchise anyone.

  o  Ensure that those eligible are true "participants" with enough
      current understanding of IETF practice and people to make informed
      decisions.

While it will probably be the case that the people whose disenfranchisement this prevents will be participants in the sense that you mean, there will be many more such participants who are not helped by this.

   o  The criteria must be algorithmic so that the Secretariat can check
      them mechanically.

I think we agree on this, with the caveat that you don’t want to do any new work, whereas I think that if you don’t do sufficient new work to capture the other ways that people participate in the IETF, this is going to drastically reduce the number of active participants who would qualify for nomcom under the new criteria.   In-person attendance will still be mandatory for most active participants.

This is not that hard of a problem.  I’m afraid that your straw-man proposal will make things worse, not better.  That’s why I’m arguing that we do something a bit more even-handed, which is only a bit more work, rather than jumping into an emergency fix that makes things worse than they are now.  Once we have a temporary solution, there will be no urgency to making it better, and if history is any guide, that means that it simply won’t ever happen.

---

From: S Moonesamy <[email-masked]>
Date: Wed, 18 Mar 2020 23:02:24 -0700

Hi Stephen,
>different to me from what's in the draft.

Let's see whether you and I could agree. :-)

I have been thinking about the following questions for reasons 
unrelated to that draft:  The draft states that the IETF 107 meeting 
was cancelled.  Can a person attend a meeting if that meeting is 
cancelled?  If the answer is yes, does that person fulfill the 
requirements of RFC 8713?

Regards,
S. Moonesamy

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 19 Mar 2020 12:37:43 +0000

Hiya,

On 19/03/2020 06:02, S Moonesamy wrote:
> Hi Stephen, At 07:01 AM 18-03-2020, Stephen Farrell wrote:
>> Again, I'm not ignoring it, I'm disagreeing with you:-) What you
>> are proposing definitely seems qualitatively different to me from
>> what's in the draft.
> 
> was cancelled. 

Yeah I guess it'd be more accurate if we said "the
cancellation of the in-person aspects of the IETF107
meeting."

> Can a person attend a meeting if that meeting is
> cancelled? 

Many of us will I guess attend various online meetings
that are part of IETF107. So, my answer is yes but that's
because of the correction above.

> If the answer is yes, does that person fulfill the

That's the topic of a thread on [email-masked] isn't it? [1]
IIUC the IESG will decide how to handle that specific case,
and I think that's the right way to handle this case.

Cheers,
S.

[1] https://mailarchive.ietf.org/arch/msg/ietf/uBt8FeKeN4CRl7YTDMoy6m8leS0/

> Regards, S. Moonesamy

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 19 Mar 2020 12:56:55 +0000

Hi Ted,

I could quibble with a number of assertions in your mail
but instead I'll jump to the end:

On 19/03/2020 02:33, Ted Lemon wrote:
> 
> This is not that hard of a problem.

If by "this" you mean your proposal, then I disagree. I
think specifying something practical that takes into
account the kinds of things you want to include is a
hard problem. If, as you assert, it's not that hard, then
why don't you write that I-D? Not asking that you finish
that in the next week, but if it's not hard, then it
shouldn't be hard to make an I-D either. If the result
were really better and still practical I'd support that,
but I honestly can't see how you can achieve practicality.

> I’m afraid that your straw-man proposal will make things worse, not
> better.  That’s why I’m arguing that we do something a bit more
> even-handed, which is only a bit more work, rather than jumping into
> an emergency fix that makes things worse than they are now.

I very much doubt your proposal is "only a bit more work"
and don't agree that something derived from our draft that
had IETF consensus would be worse. (There's also no real
utility in exaggeration like "only a bit" and "emergency
fix" IMO - I can't see any reason why this discussion has
to be as emotive as that, but maybe that's just me reacting
to less well chosen language.)

> Once we have a temporary solution, there will be no urgency to making
> it better, and if history is any guide, that means that it simply
> won’t ever happen.

That last is a real risk, yes. I do agree that if we get
something done here, it'll be hard to change that again
for a few years (unless other events force change). So yes,
we need to figure out whether to be as ambitious as you'd
like, or not.

Cheers,
S.

---

From: S Moonesamy <[email-masked]>
Date: Thu, 19 Mar 2020 13:52:48 -0700

Hi Stephen,
>and I think that's the right way to handle this case.

Matters related to voting tend to generate disputes as it is 
something people might feel strongly about.  In this case, the way in 
which the words are formulated is important.

Regards,
S. Moonesamy