Subject: [Eligibility-discuss] some numbers

From: Stephen Farrell <[email-masked]>
Date: Wed, 22 Apr 2020 01:50:47 +0100

Hiya,

With many thanks to Robert Sparks for writing up some
initial code to query the datatracker DB exploring the
results of drafts such as the one Brian and I wrote,
we have some initial numbers to report.

If one applied the rules from our -02 draft [1] then
if seems we'd end up with a similarly sized set of nomcom
eligible folks without requiring f2f meeting attendance.

Details are at [2].

The status quo with only f2f meetings gives 647 people,
the draft-02 scheme gives us 742 eligible people without
requiring f2f attendance. 290 of those are eligible
both ways. From eyeballing the lists, the eligible
people who'd not usually be eligible seem no more
crazy than any random subset of IETFers;-)

I didn't put the actual lists of names up as I'm allergic
to publishing lists of people's names, but I'd be fine with
publishing hashes of them if that helped, or with answering
any on or off list queries. [2] does have instructions that
allow you to replicate creating the lists yourself if you
are willing to do a bit of work. (Before we could consider
ourselves "done" here, I do think we'd want someone to
have independently replicated results like this.)

For me, this means that draft-02 seems credible enough to
pursue some more, based on the data and not only based on
a hunch, which is good.

Happy to try answer queries or check out alternatives
if people have any to suggest...

Cheers,
S.

PS: I'm sure there'll be some errors in this somewhere.
Apologies in advance for those and they are of course
my fault and not Robert's nor Brian's ;-)

[1] https://tools.ietf.org/html/draft-carpenter-eligibility-expand-02
[2] https://github.com/sftcd/elig/blob/master/queries.md

---

From: Ted Lemon <[email-masked]>
Date: Tue, 21 Apr 2020 20:59:09 -0400

Stephen, what results do you get if you run this on historical data, say for the past two years?  The problem with this run is that it doesn’t really tell us who this would disenfranchise under normal circumstances, given that circumstances are anything but normal.

IOW, pretend that it’s just after IETF 106, and now run the old algorithm, run the new algorithm (in the second case pretending IETF 106 didn’t happen) and see who loses eligibility.  Do this again for IETF 105, 104 and 103, so that we have more than one test case.

Does that make sense?  Is it hard to do?

> On Apr 21, 2020, at 8:50 PM, Stephen Farrell <[email-masked]> wrote:
> 
> 
> Hiya,
> 
> <0x5AB2FAF17B172BEA.asc>-- 
> Eligibility-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/eligibility-discuss

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 22 Apr 2020 02:08:23 +0100

Hiya,

On 22/04/2020 01:59, Ted Lemon wrote:
> Stephen, what results do you get if you run this on historical data,
> say for the past two years?  The problem with this run is that it
> doesn’t really tell us who this would disenfranchise under normal
> circumstances, given that circumstances are anything but normal.
> 
> IOW, pretend that it’s just after IETF 106, and now run the old
> algorithm, run the new algorithm (in the second case pretending IETF
> 106 didn’t happen) and see who loses eligibility.  Do this again for
> IETF 105, 104 and 103, so that we have more than one test case.
> 

I'm not 100% sure I'm answering the right question but I
think that'd be the 357 people who are only eligible based
on f2f meetings and not otherwise. (See the Venn diagram
near the bottom of [2].)

Put another way: if we have no f2f meetings before the 2021
nomcom are selected, and we adopted draft-02, then those
are the people who'd "lose" (assuming nomcom-eligibility is
a "win";-).

It is absolutely a non-trivial, interesting and subtle
question as to whether we'd be ok with dropping that set
of 357 people and replacing them with the set of 452 who
would be eligible without having been at a f2f meeting
recently.

Optimistically, if we do have another f2f meeting before
nomcom2021 are picked, I'd guess many of those 357 people
might re-gain eligibility, but I'd have to change code
to put a number on that.

I guess I could also check how many of the 290 who are
in the intersection have previously volunteered to be
on a nomcom. (Another one for tomorrow or the next day:-)

Cheers,
S.

>> With many thanks to Robert Sparks for writing up some initial code
>> to query the datatracker DB exploring the results of drafts such as
>> the one Brian and I wrote, we have some initial numbers to report.
>> 
>> If one applied the rules from our -02 draft [1] then if seems we'd
>> end up with a similarly sized set of nomcom eligible folks without
>> requiring f2f meeting attendance.
>> 
>> Details are at [2].
>> 
>> The status quo with only f2f meetings gives 647 people, the
>> draft-02 scheme gives us 742 eligible people without requiring f2f
>> attendance. 290 of those are eligible both ways. From eyeballing
>> the lists, the eligible people who'd not usually be eligible seem
>> no more crazy than any random subset of IETFers;-)
>> 
>> I didn't put the actual lists of names up as I'm allergic to
>> publishing lists of people's names, but I'd be fine with publishing
>> hashes of them if that helped, or with answering any on or off list
>> queries. [2] does have instructions that allow you to replicate
>> creating the lists yourself if you are willing to do a bit of work.
>> (Before we could consider ourselves "done" here, I do think we'd
>> want someone to have independently replicated results like this.)
>> 
>> For me, this means that draft-02 seems credible enough to pursue
>> some more, based on the data and not only based on a hunch, which
>> is good.
>> 
>> Happy to try answer queries or check out alternatives if people
>> have any to suggest...
>> 
>> Cheers, S.
>> 
>> PS: I'm sure there'll be some errors in this somewhere. Apologies
>> in advance for those and they are of course my fault and not
>> Robert's nor Brian's ;-)
>> 
>> [1]
>> https://tools.ietf.org/html/draft-carpenter-eligibility-expand-02 
>> <0x5AB2FAF17B172BEA.asc>-- Eligibility-discuss mailing list

---

From: Ted Lemon <[email-masked]>
Date: Tue, 21 Apr 2020 21:14:12 -0400

I think you’re missing my point, or I’m misunderstanding your response.

The thing I’m concerned about is that this algorithm that’s been proposed is going to disenfranchise a lot of people who would have been nomcom eligible if there had been an IETF 107.  But we don’t really have a way to test that, because we don’t know who would have shown up for IETF 107.   So we can’t do an A/B test on IETF 107.  But we do know who showed up for IETF 106, and for IETF 105, and for IETF 104, and so on.

So what I’m proposing is that you do an A/B test comparison using your algorithm as if it were now just past IETF 106, which didn’t happen, using the list of RFCs published at that time, and make a list of all the people who would not be nomcom eligible using this scenario, but are noncom eligible because IETF 106 did happen.  And since each IETF has slightly different attendance implications, run the same scenario for a few IETFs before that.

If nobody is disenfranchised, then the algorithm is perfect.  If some people are disenfranchised, that would be useful to know as we evaluate the effectiveness of the proposal.

Given how late it is where you are, I would be happy to take my answer in the morning… :)

> On Apr 21, 2020, at 9:08 PM, Stephen Farrell <[email-masked]> wrote:
> 
> 
> Hiya,
> 
> <0x5AB2FAF17B172BEA.asc>

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 22 Apr 2020 02:20:47 +0100

Hiya,

I'll try figure out an answer for you tomorrow but just one
quibble in the meantime:-)

On 22/04/2020 02:14, Ted Lemon wrote:
> The thing I’m concerned about is that this algorithm that’s been
> proposed is going to disenfranchise a lot of people who would have
> been nomcom eligible if there had been an IETF 107.
The above is only true if we don't have more f2f meetings.
If those re-start in 2021 then many people who travel will
not lose eligibility. If those don't restart, as-is we'd
have nobody eligible. So it's not the scheme in draft-02
that'd be disenfranchising anyone, but rather covid-19.

Our draft *only* enfranchises people.

Cheers,
S.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 21 Apr 2020 21:33:00 -0400

> Our draft *only* enfranchises people.

That's a fair point, I guess, but you could also say that if we take no action, it will disenfranchise literally everyone.  So while I am sorry if my framing seemed unkind, the reason I didn’t think to frame differently is that the only thing I am really more concerned with is the outcome—I don’t care who is to blame. :)

---

From: Michael Richardson <[email-masked]>
Date: Tue, 21 Apr 2020 22:35:18 -0400

Thank you for doing this work.
I think that Ted's question is definitely an interesting one to answer.

I agree that persuing the draft is a good plan.

{My only quibble with the results is that sitting IESG members are not
eligible, nor will WG chairs that want to be selected want to serve}

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 22 Apr 2020 15:16:04 +1200

Michael,
On 22-Apr-20 14:35, Michael Richardson wrote:
> eligible, 

Yes, they would have to be deleted in a real run, of course, as the draft
says. But that's true for any algorithm we choose, and it's a filter that
can only be applied on the actual eligibility day.

> nor will WG chairs that want to be selected want to serve}

I expect so, although they are in fact eligible even under today's
rules. But Path 2 actually says "Has been a WG Chair or Secretary
within the last 3 years". Maybe it should be 5 years to catch a
few more well-informed volunteers.

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Wed, 22 Apr 2020 05:15:56 -0400

> few more well-informed volunteers.

I would be interested in what is the delta for each increase in years we look
at.  Clearly it's inclusive as you add years of history.
Given that experienced WG chairs are highly valued, I suspect that we seldom
send them out to pasture, except by their own decision and/or death.
(Competence tends to be a curse...)

Someone who has a WG chair 15 years go and has not been active since is
probably not well enough socially connected to be good for nomcom, so I want
to be clear that is a non-goal of my question, but I'm curious if there is
cliff in the data that would naturally appear.

And even then, we have a number of retirees that seem to continue to
contribute, even if they don't travel much, and this mechanism would include
their very mature experience input into nomcom. It's hard to undervalue the
worth of their 30+ years of industry experience, and social maturity.
(Yes, I'm talking about you Brian, but there are several others as well)

At the same time, I recognize that this could look like we are entrenching
some kind of oligarchy, but I think that our other criteria and random
selection should make this less of a concern.

---

From: Alissa Cooper <[email-masked]>
Date: Wed, 22 Apr 2020 13:17:20 -0400

Hi Stephen,

Thanks so much for doing this. 

> on a nomcom. (Another one for tomorrow or the next day:-)

If possible I think a good comparison with what you have would be to look at the 5- and 3-year periods that ended in April 2019 or 2018. I.e., generate the lists using the five meetings prior to that date and the 5- and 3-year periods prior to that date. Then we could see how similar the lists and the venn diagram look at two points in time, rather than one.

If you do that, then also looking at how the list of who actually did volunteer for the nomcom in that particular year and how that group breaks down between those who would be eligible via paths 2/4/5 and those who would not would be revealing. In other words, it would be interesting to see if most of the people who volunteer land at the center of the venn diagram or not.

None of this implies any normative judgment as you say, but at least it could help people understand numerically what the impact could be of the paths you propose.

Thanks,
Alissa

---

From: "Pete Resnick" <[email-masked]>
Date: Wed, 22 Apr 2020 12:44:07 -0500

On 21 Apr 2020, at 20:33, Ted Lemon wrote:

> That's a fair point, I guess, but you could also say that if we take 
> no action, it will disenfranchise literally everyone.

I don't think that's necessarily true. If we take 8713 as the draft 
describes it:

    *  Path 1: As per [RFC8713], the person has attended 3 out of the
       last 5 in-person IETF meetings.

The "last 5 in-person IETF meetings" will always be 102, 103, 104, 105, 
and 106 until we have another in-person IETF meeting. The result for who 
ends up eligible would be increasingly non-useful, but it is true that 
we'll have 647 people who will be eligible for a really long time.

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 22 Apr 2020 20:03:21 +0100

Hiya,

On 22/04/2020 18:17, Alissa Cooper wrote:
> If possible I think a good comparison ...

Ack. I'll try include that too. I had a quick look at
this today but got side-tracked;-)

Just to set expectations: I'm assuming that getting
these answers before the end of the month(-ish) is
fast enough, given the ongoing LC related to the 2020
nomcom ends then.

S.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 22 Apr 2020 15:31:35 -0400

On Apr 22, 2020, at 1:44 PM, Pete Resnick <[email-masked]> wrote:
> The "last 5 in-person IETF meetings" will always be 102, 103, 104, 105, and 106 until we have another in-person IETF meeting. The result for who ends up eligible would be increasingly non-useful, but it is true that we'll have 647 people who will be eligible for a really long time.

That amounts to disenfranchising all new participants.

I wonder how many relative newcomers who signed the blue sheets for IETF 107 or a subsequent interim are not eligible.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 23 Apr 2020 08:57:50 +1200

On 23-Apr-20 07:31, Ted Lemon wrote:
> On Apr 22, 2020, at 1:44 PM, Pete Resnick <[email-masked] <mailto:[email-masked]>> wrote:

The problem here (at the moment) is getting a reliable database of those signatures. If Mickey Mouse "signs" the blue sheet portion of an Etherpad for a few sessions, he will become eligible.

We'd need a sign-in system based on datatracker logins. That doesn't sound like an impossible dream, but it's also probably more than a few lines of Python.

We could mock up such a database by doing a sort-merge of the lists in the IETF 107 Etherpads and manually checking for bogons, but I don't think that would tell us much.

    Brian

---

From: "Pete Resnick" <[email-masked]>
Date: Wed, 22 Apr 2020 16:08:54 -0500

On 22 Apr 2020, at 15:57, Brian E Carpenter wrote:

>>> The "last 5 in-person IETF meetings" will always be 102, 103, 104, 
>>> 105, and 106 until we have another in-person IETF meeting. The 
>>> result for who ends up eligible would be increasingly non-useful, 
>>> but it is true that we'll have 647 people who will be eligible for a 
>>> really long time.
>>

Oh, absolutely. I was not indicating that this was a good result, simply 
that it's what the current interpretation of RFC 8173 amounts to and why 
it's important for us to adopt something for remote participants 
generally.

>> I wonder how many relative newcomers who signed the blue sheets for 
>> IETF 107 or a subsequent interim are not eligible.
>
> The problem here (at the moment) is getting a reliable database of 
> those signatures. If Mickey Mouse "signs" the blue sheet portion of an 
> Etherpad for a few sessions, he will become eligible.
>
> We'd need a sign-in system based on datatracker logins. That doesn't 
> sound like an impossible dream, but it's also probably more than a few 
> lines of Python.

As Andy posted over on manycouches 
<https://mailarchive.ietf.org/arch/msg/manycouches/0FUXKXQclMqjbjmNCpJMtImOFQg>, 
if we were using Meetecho, we get most of it for free. The more 
interesting question is whether there's an easy way to do that for f2f 
meetings.

pr

---

From: Michael Richardson <[email-masked]>
Date: Wed, 22 Apr 2020 20:33:40 -0400

> sound like an impossible dream, but it's also probably more than a few

I was thinking about this today, and I think we should "sign" the Bluesheets
with our Datatracker URL. Mine would be:
   https://datatracker.ietf.org/person/Michael%20Richardson

1) it contains my name.
2) it is unique.
3) easily processed by secretariat.
4) URL provides useful information to people who want to know who I am.

---

From: Robert Sparks <[email-masked]>
Date: Thu, 23 Apr 2020 11:19:52 -0500

>     https://datatracker.ietf.org/person/Michael%20Richardson

Look at

https://datatracker.ietf.org/person/Matt%20Larson

That's really two people.

We don't have this happen much, but it does happen.

Using an email address the datatracker knows about would be less ambiguous

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 23 Apr 2020 16:25:50 +0000

The bluesheets serve two purposes: capacity planning for later IETF's, and perhaps fodder for IPR legal actions.  In the first, the name doesn't matter as long as only physical attendees sign. In the second, someone might try to hide/dissemble their email to claim that they weren't there, but the legal process has steps to (try to) find out the facts of the matter.

With virtual meetings we don't need the first purpose. For legal matters, things like IP address have been shown to be useful (e.g., copyright/takedown notices).  Should we care?  I don't know, but if we do, then we should think about recording IP address.

---

From: S Moonesamy <[email-masked]>
Date: Thu, 23 Apr 2020 09:57:08 -0700

Hello,
At 09:25 AM 23-04-2020, Salz, Rich wrote:
>The bluesheets serve two purposes: capacity planning for later 
>IETF's, and perhaps fodder for IPR legal actions.  In the first, the 
>name doesn't matter as long as only physical attendees sign. In the 
>second, someone might try to hide/dissemble their email to claim 
>that they weren't there, but the legal process has steps to (try to) 
>find out the facts of the matter.

If the "blue sheets" are used for capacity planning, it does not make 
sense to publish them.  If the purpose is for legal action, there 
isn't a need to publish them given that subpoenas are generally used 
for legal actions.

Regards,
S. Moonesamy

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 23 Apr 2020 17:00:17 +0000

Informal discussion, or reviewing attendee list to figure out "who was that person I spoke with".  That's also important.  Publishing also emphasizes the public nature of our meetings.  That's also important.  I didn't mention those, perhaps I was wrong to not repeat those.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 24 Apr 2020 08:42:58 +1200

> With virtual meetings we don't need the first purpose. For legal matters, things like IP address have been shown to be useful (e.g., copyright/takedown notices).  Should we care?  I don't know, but…

Please not. Judges and lawyers may be under the illusion that IP addresses identify people, but we're not. In fact, we published RFC4941 to make sure of it.

   Brian