Subject: [Mtgvenue] New Draft: draft-elkins-mtgvenue-participation-metrics

From: <[email-masked]>
Date: Tue, 12 Jul 2016 07:37:22 +0000 (UTC)

All,
I am posting this draft to the list as the date to submit officially passed last week.
WG chairs, please let me know if you want to discuss this at IETF96 or if we should wait until IETF97. Thanks,
Nalini ElkinsInside Products, Inc.www.insidethestack.com(831) 659-8360

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 12 Jul 2016 21:27:34 +1200

One thing that needs to be written down very clearly is that this usage
of 'participate', 'participation', and 'participant' has no relationship
to the formal usage in RFC 3979 or draft-bradner-rfc3979bis-08 and has no
impact on IPR disclosure requirements.

    Brian
On 12/07/2016 19:37, [email-masked] wrote:
> _______________________________________________
> Mtgvenue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mtgvenue
>

---

From: Nalini Elkins <[email-masked]>
Date: Tue, 12 Jul 2016 12:24:05 +0200

Brian,

Sent from my iPhone

> On Jul 12, 2016, at 11:27 AM, Brian E Carpenter <[email-masked]> wrote:
> 

Is there another word for 'participation' better to use to eliminate confusion?

Nalini

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 13 Jul 2016 03:28:16 +1200

> Is there another word for 'participation' better to use to eliminate confusion?

I can't think of one. In the legal text there's an important difference between
'participant' and 'contributor', and we might be looking to mix the two concepts
somewhat in the Mtgvenue context. It will just avoid future confusion if we add
a one-sentence disclaimer. OK, let me try:

Note that in the present document, the words "participate" and "contribute"
and their derivatives are not used in their formal sense as defined
in the IETF's intellectual property rules [RFC3979].

   Brian

---

From: <[email-masked]>
Date: Tue, 12 Jul 2016 15:44:51 +0000 (UTC)

>in the IETF's intellectual property rules [RFC3979].

Thanks, Brian.  I will add that to the draft.

Nalini

---

From: S Moonesamy <[email-masked]>
Date: Tue, 12 Jul 2016 10:23:55 -0700

Hi Nalini,
>I am posting this draft to the list as the date to submit officially 

In Section 1: "One of the criteria for choosing a location is the 
amount of participation by the people in that region."  The draft 
extends participation in the IETF process to: "email lists, writing 
drafts, physical or remote attendance at a meeting, chairing Working 
Groups and so on".  In general, participation is when the person 
expresses an opinion.  It is usually done to influence a decision 
which is being taken.  Attending a meeting, whether it is done in 
person or remotely, does not mean that the person is participating in 
the IETF process.

draft-sullivan-mtgvenue-decisions-00 argues for not having a meeting 
in a place for outreach purposes while Section 1.2 and 1.3 of this 
draft argues that having a meeting in a new geographic region could 
provide some motivation to participate.

The draft creates two categories of participation: "active 
contribution to substantive IETF work" and support to "improve the 
functioning of the IETF organism".  I read the draft and I could only 
find the following for the second category: "Involvement in process 
groups is very much needed by the IETF and it may be a way for new 
people to work their way towards fundamental participation".

Section 2.2 mentions "social sanctions for making comments without 
merit" on email lists or at the microphone in a meeting merely to 
improve the statistics for the region.  Does "social sanctions" 
include problematic comments?

Section 2.2.1 mentions economic cost.  If a person attends a meeting 
in person, the costs are travel expenses, meeting fee, and up to six 
days of work.  The draft argues that "an Internet connection of some 
type and an access device is all that is needed".  I would argue that 
the person also needs to have some understanding of the topic being 
discussed and be prepared to discuss about it.  That cost is not 
being taken into account.

Section 2.2.2 mentions that if an Internet Draft "is chosen for live 
presentation at a Working Group session, then that is a high degree 
of participation".  That is a presentation and not a review of the 
Internet Draft.  It is not worth doing a (free) review if the live 
presentation is viewed as higher on the participation scale.

I'll make a few suggestions:

   (i)   Please assume that there will be attempts to "game the 
system".  If the
         number is relatively low it is not a significant concern.

   (ii)  If the objective is to have a measure of participation it is 
better not
         to get into a discussion about the level of participation.

   (iii) It might be easier to "count" comments received during 
WGLCs.  Not having
         Area Directors in the "count" does not significantly affect 
the "count" as
         I assume that it is a large number.

One of the issues with (iii) is that the person commenting does not 
usually provide location information.

Regards,
S. Moonesamy

---

From: Melinda Shore <[email-masked]>
Date: Tue, 12 Jul 2016 11:54:46 -0800

On 7/11/16 11:37 PM, [email-masked] wrote:
> WG chairs, please let me know if you want to discuss this at IETF96 or
> if we should wait until IETF97.

Jonne is still on holiday with poor connectivity and I haven't
had a chance to discuss this with him, but I'm going to go
ahead and put this on the agenda.  Please upload the draft as
soon as submissions re-open, and in the meantime if you can
get slides to us ASAP that can help get information about the
draft out to people who are planning to attend the session.

One quick observation - your draft seems to cover two topics,
1) advocacy for meeting in new places, and 2) arguing for the
development of participation metrics.  You may want to split
those out, and possibly move section 1 into a separate document.

Thanks,

Melinda

---

From: <[email-masked]>
Date: Tue, 12 Jul 2016 20:27:43 +0000 (UTC)

Thanks so much for your comments.  Mine inline.
Hi Nalini,

>person or remotely, does not mean that the person is participating in 

Yes, agreed.  I believe I was trying to say that if we use comments made in a WG meeting (expressing an opinion), then that is considered participation.  It is impossible to quantify "hallway conversations" and such.

>provide some motivation to participate.

I was trying to say that showing increased participation in a region is done by a group of committed people who try to organize the region and that has a lot of value in and of itself.   I was not saying that IETF meetings should be held in a region for the purposes of outreach.  But, I think if what I am saying is being misunderstood, then I should rephrase.

>people to work their way towards fundamental participation".

I will add more on "process" activities.  For example, helping with the mentoring program making matches, helping with outreach, etc.

>include problematic comments?

Do you mean that people will be rude if someone says something which does not have merit?  Or do you mean that if people make problematic comments then they will be the object of social sanctions?   I guess that also brings up the point, if someone makes problematic comments, then are those counted in a region's statistics?  And, how do you tell what is problematic? 

>being taken into account.

True.  The actual barrier to entry is that you should actually understand the protocol being discussed and have something somewhat intelligent to say about it.  

>presentation is viewed as higher on the participation scale.

I think this question of levels of participation is getting difficult, as you bring up below.  

>I'll make a few suggestions:

  >(i)  Please assume that there will be attempts to "game the system".  If the
  >       number is relatively low it is not a significant concern.

Sure.

> better not to get into a discussion about the level of participation.

I think you may be right on this.   At least to start with, we should just use what we can verify via written means (email lists, minutes, etc.) and let it all count the same.   
I guess the thing I was thinking is that then it makes being a WG chair count the same as one comment on an email list.  Is that really fair?   I don't know how to do this but I think there is something to be thought about. 

>WGLCs.  Not having Area Directors in the "count" does not significantly affect 
>the "count" as I assume that it is a large number.

Not following you here.  A lot of comments are made before WGLC.  What do you mean?  I am not understanding.

>usually provide location information.

Well, this is actually a big problem with the whole scheme!   Location is a piece of information we need to keep somehow associated with each person.   Maybe it could be asked when you sign up for an email list?  Or when you register?   I guess each person would say what region they are from.   I don't think that is asked today.   At least, I don't remember. 

I think some people have multiple "homes" but most of us are primarily from one geographic region.
Nalini

---

From: <[email-masked]>
Date: Tue, 12 Jul 2016 20:31:55 +0000 (UTC)

>draft out to people who are planning to attend the session.

Sure.  Will do.

>those out, and possibly move section 1 into a separate document.
I wasn't meaning to advocate but it seems that more than one person is seeing it that way.  So, let me think about this.  Maybe splitting this document into two is the way to go.
Thanks,Nalini

---

From: S Moonesamy <[email-masked]>
Date: Tue, 12 Jul 2016 16:26:15 -0700

Hi Nalini,
At 13:27 12-07-2016, [email-masked] wrote:
>Yes, agreed.  I believe I was trying to say that if we use comments 
>made in a WG meeting (expressing an opinion), then that is 
>considered participation.  It is impossible to quantify "hallway 
>conversations" and such.

For what is worth, I chose that description for participation as that 
word is already used in a BCP.  I prefer not to explore hallway 
conversations as it would be like tackling problems which cannot be solved.

>I was trying to say that showing increased participation in a region 
>is done by a group of committed people who try to organize the 
>region and that has a lot of value in and of itself.   I was not 
>saying that IETF meetings should be held in a region for the 
>purposes of outreach.  But, I think if what I am saying is being 
>misunderstood, then I should rephrase.

You captured what you were trying to say in the last paragraph of 
Section 1.3.

>I will add more on "process" activities.  For example, helping with 
>the mentoring program making matches, helping with outreach, etc.

It might make the usage of the word "participation" more difficult.

>Do you mean that people will be rude if someone says something which 
>does not have merit?

There has been a few cases previously where the receiver might be 
offended.  I haven't been keeping up with the meeting stuff to 
comment about whether it would happen nowadays.

>   Or do you mean that if people make problematic comments then they 
> will be the object of social sanctions?   I guess that also brings 
> up the point, if someone makes problematic comments, then are those 
> counted in a region's statistics?  And, how do you tell what is problematic?

There are process mechanisms to deal with problematic 
comments.  There isn't a count of problematic comments; it may be 
better not to have that.  The issue was about a person making 
comments without merit on email lists or at the microphone merely to 
improve the statistics.  It may be problematic if, for example, a 
person on a mailing list is singled out.

>True.  The actual barrier to entry is that you should actually 
>understand the protocol being discussed and have something somewhat 
>intelligent to say about it.

I would use the word "useful" instead of "intelligent".

>I think you may be right on this.   At least to start with, we 
>should just use what we can verify via written means (email lists, 
>minutes, etc.) and let it all count the same.

Ok.

>I guess the thing I was thinking is that then it makes being a WG 
>chair count the same as one comment on an email list.  Is that 
>really fair?   I don't know how to do this but I think there is 
>something to be thought about.

I did not get into the details.  For example, posting one comment or 
two over a year would not be regular participation.  Instead of 
trying to equate being a WG chair with posting one comment on a 
mailing list my suggestion was to go with some light definition of 
(WGLC) participation.  The suggestion requires much more discussion, 
in my opinion, as there are probably issues which I have not thought about.

>Not following you here.  A lot of comments are made before 
>WGLC.  What do you mean?  I am not understanding.

There are 15 Area Directors.  It should not affect the statistics if 
they are all from the same region if we look at participation in 
terms of magnitude.

>Well, this is actually a big problem with the whole 
>scheme!   Location is a piece of information we need to keep somehow 
>associated with each person.   Maybe it could be asked when you sign 
>up for an email list?  Or when you register?   I guess each person 
>would say what region they are from.   I don't think that is asked 
>today.   At least, I don't remember.

The only piece of information currently required for participation on 
a mailing list is an email address.  There are different ways to 
implement the scheme, e.g. the person can provide the additional 
piece of information if he/she is interested in it being counted for 
determining participation by country.

>primarily from one geographic region.

Yes.

Regards,
S. Moonesamy

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 13 Jul 2016 20:07:56 +0000

Thanks for this. On a quick read, it gives a pretty reasonable summary of the questions.

One issue I have with metrics is figuring out how to consistently and accurately measure and report them. For example, your paper says that posting to a working group list is participation but posting to the IETF list is not, and that specifically the posts that are relevant to an Internet Draft are "participation", while others may not be. That sort of implies that we should analyze list discussion (probably after the fact, processing archives) looking for posts relevant to an internet draft, and count those.

Two issues:

(1) Some threads put the draft name into the subject line or mention it in the body, but I can't guarantee that all do; when a thread mutates, the subject line is usually changed, and the new one usually identifies the topic, not the draft name. So I'm not sure that I can identify the draft a post is about, in an automated way, and I'm not sure I want to impose the tooling burden to create a way.

(2) IESG Last Calls happen on the IETF List, in order to get opinions from people that have not necessarily been active in the working group. As a result, there is often discussion of Internet Drafts that happens on the IETF list.

So I don't know how to calculate the information that your drafts suggests be used as a metric.

Rather than add complexity or overhead, I would prefer a simpler metric, one that is a lot more readily calculated and reported on. If you post to a mailing list @ietf.org, it's IETF participation. The comment may or may not persuade others, may or may not be seminal, and may or may not be relevant to a specific draft. It may be as short as "+1", with most of the content implied being in someone else's post and expressing support. But the IETF is first and foremost a body in which ideas are discussed, and that post is part of the discussion.

I have a similar comment on drafts as participation. As you note, a draft is an extended expression of an idea. It might be a bad idea or a good idea, it might be mathematically impossible (we have had some of those) or it might reflect operational realities that non-operators tend to miss, it might be a number of other things. I have heard it said that a person that knows "how" will always have a job - working for someone that knows "why". If you look at the Internet Drafts historically, or the statistics of the RFC series, there are drafts and RFCs on both "how" and "why", and the latter are often the most seminal.

Yoav commented a few days ago that the drafts that should count as "participation" are those that "progress", which is to say "become Working Group drafts", or at least promote substantive discussion. I have some sympathy for the position, in the sense that it is something I think a lot about as a working group chair. However, in terms of participation, I would think that a person that posts a draft, however poorly or well received, has expended effort in an attempt to participate, and put an idea out for consideration. That, in my book, is participation.

I might ask how we would classify the discussion of draft-elkins-mtgvenue-participation-metrics. Imagine that you never actually post it - someone else posts a better one that you defer to, or you get disgusted with the resulting email conversation, or whatever other scenario might cause that. By the argument that mailing list discussion of posted drafts is "participation" and other comments are not, your email, this response, and the others in the thread would not be "participation". I'm pretty sure that's not what you intend or what I would want, but a strictly legal reading would say that. It's not uncommon for several individual submissions to bat variations on an idea around, and a subsequent working group draft combine them. I would think that each of those individual submissions was a contribution to discussion and the ultimate working group draft. So I would want to count them all as participation.

So on drafts, I would similarly go for "simple": if you post a draft, you have participated.

In general (we can discuss this f2f next week if you like) any draft that proposes metrics should describe how they might be measured, so that we have some idea of the mechanism and hoop-jumping involved.

My two yen.

---

From: Melinda Shore <[email-masked]>
Date: Wed, 13 Jul 2016 12:16:40 -0800

> So on drafts, I would similarly go for "simple": if you post a draft,

I have a few concerns about the draft, and one of the primary
ones is that I'm hopeful that "participation" is being defined in
service of the broader goal of figuring out who ought to be
at meetings, but I'm not sure it is.  There may well be value in
defining "participation" more generally, within the overall context
of the organization (and I do think it would be useful to figure
out what the activity levels actually are), but I hope we don't
lose sight of which problems we're trying to solve in this particular
working group.

Melinda

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 13 Jul 2016 22:35:10 +0000

> On Jul 13, 2016, at 1:16 PM, Melinda Shore <[email-masked]> wrote:
> 
> at meetings, but I'm not sure it is.

I *think* the discussion has to do with where meetings should occur, based on the rubric that meetings should happen in places where we have participants - "so who is a participant". "Where meetings should occur" is central to the charter, whether IETF participation is directly so or not.

> There may well be value in

---

From: Melinda Shore <[email-masked]>
Date: Wed, 13 Jul 2016 16:03:43 -0800

On 7/13/16 2:35 PM, Fred Baker (fred) wrote:
>> I have a few concerns about the draft, and one of the primary ones
>> is that I'm hopeful that "participation" is being defined in
>> service of the broader goal of figuring out who ought to be at
>> meetings, but I'm not sure it is.
>
> I *think* the discussion has to do with where meetings should occur,
> based on the rubric that meetings should happen in places where we
> have participants - "so who is a participant". "Where meetings should
> occur" is central to the charter, whether IETF participation is
> directly so or not.

I think so, as well, but I have a personal concern about meetings
being effective, and being places where we do things that can't
be done on mailing lists or by use of other mechanisms, and I'm
not sure that what's under discussion captures that.  As chair
of this particular working group I have concerns about whether or
not we should be defining what a "participant" is, both as a matter
of scope and as a matter of not heading into ratholes.  That said I do
think metrics can be extremely useful, so I'm interested to
see where this discussion goes.

Melinda

---

From: <[email-masked]>
Date: Thu, 14 Jul 2016 06:47:52 +0000 (UTC)

Fred,

>One issue I have with metrics is figuring out how to consistently and accurately measure and report them. 

Yes.  This is a big issue.  I am trying to start with written material that can be perused via a tool.  For example, WG minutes, drafts and email list contributions.   I believe that once we agree on the principles (which list, how / if to weight items, then we (someone) can develop software that uses the written material as input.   I wanted to agree on principles first.   And, whether we should even do it!

>For example, your paper says that posting to a working group list is participation but posting to the IETF list is not, and that >specifically the posts that are relevant to an Internet Draft are "participation", while others may not be. That sort of implies >that we should analyze list discussion (probably after the fact, processing archives) looking for posts relevant to an internet >draft, and count those.
>Two issues:

>(1) Some threads put the draft name into the subject line or mention it in the body, but I can't guarantee that all do; >when a thread mutates, the subject line is usually changed, and the new one usually identifies the topic, not the draft >name. So I'm not sure that I can identify the draft a post is about, in an automated way, and I'm not sure I want to impose >the tooling burden to create a way.

Good point but I don't know that we need to be that specific.  Why do we need to have a metric per contributor (participant) per draft?  Why not just per WG email list per contributor?  Then, it doesn't matter what the subject line is.

>(2) IESG Last Calls happen on the IETF List, in order to get opinions from people that have not necessarily been active in >the working group. As a result, there is often discussion of Internet Drafts that happens on the IETF list.

True.  So, maybe contributions to the IETF list need to be counted somehow. 

>Rather than add complexity or overhead, I would prefer a simpler metric, one that is a lot more readily calculated and >reported on. If you post to a mailing list @ietf.org, it's IETF participation. The comment may or may not persuade others, >may or may not be seminal, and may or may not be relevant to a specific draft. It may be as short as "+1", with most of the >content implied being in someone else's post and expressing support. But the IETF is first and foremost a body in which >ideas are discussed, and that post is part of the discussion.

Sure.

>I have a similar comment on drafts as participation. As you note, a draft is an extended expression of an idea. It might be >a bad idea or a good idea, it might be mathematically impossible (we have had some of those) or it might reflect >operational realities that non-operators tend to miss, it might be a number of other things. I have heard it said that a >person that knows "how" will always have a job - working for someone that knows "why". If you look at the Internet Drafts >historically, or the statistics of the RFC series, there are drafts and RFCs on both "how" and "why", and the latter are >often the most seminal.

>Yoav commented a few days ago that the drafts that should count as "participation" are those that "progress", which is to >say "become Working Group drafts", or at least promote substantive discussion. I have some sympathy for the position, in >the sense that it is something I think a lot about as a working group chair. However, in terms of participation, I would >think that a person that posts a draft, however poorly or well received, has expended effort in an attempt to participate, >and put an idea out for consideration. That, in my book, is participation.

>I might ask how we would classify the discussion of draft-elkins-mtgvenue-participation-metrics. Imagine that you never >actually post it - someone else posts a better one that you defer to, or you get disgusted with the resulting email >conversation, or whatever other scenario might cause that. By the argument that mailing list discussion of posted drafts is >"participation" and other comments are not, your email, this response, and the others in the thread would not be >"participation". I'm pretty sure that's not what you intend or what I would want, but a strictly legal reading would say >that. It's not uncommon for several individual submissions to bat variations on an idea around, and a subsequent working >group draft combine them. I would think that each of those individual submissions was a contribution to discussion and the >ultimate working group draft. So I would want to count them all as participation.

>So on drafts, I would similarly go for "simple": if you post a draft, you have participated.

I am good on simple but I am trying to do metrics that somehow reflect our intuitive understanding of the level of contribution.  It is of course true that it may be that a one line contribution in one RFC is equivalent in impact to 10 RFCs which see no implementation in reality.   Quality is hard to quantify.

What IS possible to quantify is how many people refer to your work.  I suppose also how many people list you in the acknowledgments.  As you well know, there is much interesting discussion that happens in private before a draft is posted.  Such people, if they don't become co-authors, are often acknowledged.   These IMO are very important contributions.

Again, I am trying to propose metrics that can be gathered if we see drafts, email lists, etc. as data sources which can be mined for information.

>In general (we can discuss this f2f next week if you like) any draft that proposes metrics should describe how they might be >measured, so that we have some idea of the mechanism and hoop-jumping involved.

Yes.  Let's discuss F2F.  I am nearly done with the presentation & will send off to WG chairs later today.

>My two yen.

Thank you for thinking about this problem.

Nalini

---

From: <[email-masked]>
Date: Thu, 14 Jul 2016 07:05:17 +0000 (UTC)

>see where this discussion goes.

The reasons I wanted to start this discussion is:

1.  As Fred said (and as the other drafts in this WG say), where an IETF meeting is physically held depends on where the participants are.  So that really begs the question: "What do you mean by participant?"

As far as I know, historically, a participant has been someone who attended physically and / or was in a leadership position.  I believe there was also some consideration given to RFCs written.  (All, please correct me if I am wrong on this.)

To my mind, that is not fair to some regions of the world which are emerging economies and cannot participate physically in person as readily as others.  So, if such regions want to grow their IETF participation, they can most readily grow via electronic means - email, remote hubs, etc.

As many have said, much of the IETF work is done via email lists.  So, if we have good strong contributors participating via email, then why should that NOT be counted?  So, I want to get consensus that we should count such things and then consensus on how to count, weigh, etc.

2.  There is also much discussion of the type "We held a meeting in such and such a place & how do we know if we have additional contributors?"

My personal feeling is that there ARE additional contributors.  But, that is only my feeling and I am only one data point.  So, let's find out if there really are additional contributors.  Why have a fact-free discussion?

3.  There is quite a bit of outreach being done by various parts of the world - India, Africa, Asia-Pacific, Russia and of course Latin America.  And, that is only what I personally know about.  I am sure that there is much more that I do not know about.

I think it is good for these regions to know what are the specific metrics of IETF contribution so that they can gauge their own progress and also show others of the progress in their region.  

4. We need consensus.  We should have metrics that we all agree on are "good enough".  We will never have perfect metrics.

I do not believe we need to rathole.  I understand your concern.  Let's stay focused.

Thanks,
Nalini

---

From: S Moonesamy <[email-masked]>
Date: Thu, 14 Jul 2016 05:27:33 -0700

Hi Nalini,

It may seem like I am opposing you as I replied to several of your 
emails.  That is not my intent.

At 00:05 14-07-2016, [email-masked] wrote:
>1.  As Fred said (and as the other drafts in this WG say), where an 
>IETF meeting is physically held depends on where the participants 
>are.  So that really begs the question: "What do you mean by participant?"

That is a good question.

>As far as I know, historically, a participant has been someone who 
>attended physically and / or was in a leadership position.  I 
>believe there was also some consideration given to RFCs 
>written.  (All, please correct me if I am wrong on this.)

I don't recall it being documented except in a IPR context or for Nomcom.

>To my mind, that is not fair to some regions of the world which are 
>emerging economies and cannot participate physically in person as 
>readily as others.  So, if such regions want to grow their IETF 
>participation, they can most readily grow via electronic means - 
>email, remote hubs, etc.

Agreed.

>As many have said, much of the IETF work is done via email 
>lists.  So, if we have good strong contributors participating via 
>email, then why should that NOT be counted?  So, I want to get 
>consensus that we should count such things and then consensus on how 
>to count, weigh, etc.

In order to argue for that I would have to attend several IETF 
meetings to have discussions with a significant number of 
attendees.  Another alternative is to ask the question (remotely) at 
the plenary.  Based on my experience of the IETF I'd say that it is 
unlikely that I would get any support from the floor.  I suggest that 
you have a conversation with Margaret and as she did say something 
which is factual during a plenary meeting.

For what it is worth, there was a person in my ex-Working Group which 
I consider as a strong contributor.  The person and I are neither 
from the same company nor from the same region.

>2.  There is also much discussion of the type "We held a meeting in 
>such and such a place & how do we know if we have additional contributors?"
>
>My personal feeling is that there ARE additional contributors.  But, 
>that is only my feeling and I am only one data point.  So, let's 
>find out if there really are additional contributors.  Why have a 
>fact-free discussion?

As an anecdote I met someone new to the IETF.  I doubt that the 
person would not be described as not contributing to the IETF as the 
person has been used as an example by an IETF Chair.

There are people who are interested in contributing and they have the 
potential to do it.  It is unfortunate that those persons are not 
counted; at some point it may viewed that it is better to go away.

I looked at the numbers for [email-masked].  There was only one person 
from China who commented during the last month.  If I go back to the 
beginning of the year, the number would be around three.  The number 
looks insignificant in comparison with comments from persons residing 
in the United States.  The numbers for IETF Last Calls used to be 
insignificant in comparison with other discussions on 
[email-masked].  Although the numbers have improved, probably due to 
the willingness of the IESG to make some changes, participation for 
an IETF Last Call is mainly from less than a handful of countries.

>3.  There is quite a bit of outreach being done by various parts of 
>the world - India, Africa, Asia-Pacific, Russia and of course Latin 
>America.  And, that is only what I personally know about.  I am sure 
>that there is much more that I do not know about.
>
>I think it is good for these regions to know what are the specific 
>metrics of IETF contribution so that they can gauge their own 
>progress and also show others of the progress in their region.

I have done outreach in my region.  If I encourage people to 
participate in the IETF and mention that there is low participation, 
I would be asked about what the IETF considers as "participation".  I 
would have to provide some facts to explain why I said "low 
participation".  Should I use the statistics for recent RFCs?  Should 
I use the attendance statistics which are presented by the IETF 
Chair?  I used to follow all the working groups in an IETF Area.  I 
knew how many persons from my region participated on the working 
group mailing lists.

https://www.iab.org/documents/minutes/minutes-2016/iab-minutes-2016-05-11/ 
states that there were people from 11 countries, including Brazil, 
Costa Rica, Bolivia, Venezuela, Argentina, India, Pakistan, Morocco, 
Ethiopia, and Benin.  According to the minutes, presenting a draft is 
a working group activity.  If several persons from South America put 
some effort to join a working group mailing list discussion to 
understand a protocol better so that they can make useful comments, 
is that a contribution from that region?  Would it be positive for 
the region and for the IETF?

>4. We need consensus.  We should have metrics that we all agree on 
>are "good enough".  We will never have perfect metrics.

Agreed.

Regards,
S. Moonesamy

---

From: Vinayak Hegde <[email-masked]>
Date: Thu, 14 Jul 2016 19:44:50 +0530

Replies inline.

On Thu, Jul 14, 2016 at 5:57 PM, S Moonesamy <[email-masked]> wrote:
>> 2.  There is also much discussion of the type "We held a meeting in such
>> and such a place & how do we know if we have additional contributors?"
>>
>> My personal feeling is that there ARE additional contributors.  But, that
>> is only my feeling and I am only one data point.  So, let's find out if
>> there really are additional contributors.  Why have a fact-free discussion?

One good metric to look for "hard/substantial" (please suggest a
better word) is to look at the list of contributors by countries.
http://www.arkko.com/tools/stats/countrydistr.html

> I looked at the numbers for [email-masked].  There was only one person from
> China who commented during the last month.  If I go back to the beginning of
> the year, the number would be around three.  The number looks insignificant
> in comparison with comments from persons residing in the United States.  The
> numbers for IETF Last Calls used to be insignificant in comparison with
> other discussions on [email-masked].  Although the numbers have improved,
> probably due to the willingness of the IESG to make some changes,
> participation for an IETF Last Call is mainly from less than a handful of
> countries.

Again the link above might be useful.

>> 3.  There is quite a bit of outreach being done by various parts of the
>> world - India, Africa, Asia-Pacific, Russia and of course Latin America.
>> And, that is only what I personally know about.  I am sure that there is
>> much more that I do not know about.
>>
>> I think it is good for these regions to know what are the specific metrics
>> of IETF contribution so that they can gauge their own progress and also show
>> others of the progress in their region.

Since you mentioned India, I checked and was pleasantly surprised to
find India at No. 6 for eg.
http://www.arkko.com/tools/allstats/india.html

But This is only one data point. We will need a melange of metrics.
This can be aggregated at region level as well.

> I have done outreach in my region.  If I encourage people to participate in
> the IETF and mention that there is low participation, I would be asked about
> what the IETF considers as "participation".  I would have to provide some
> facts to explain why I said "low participation".  Should I use the
> statistics for recent RFCs?  Should I use the attendance statistics which
> are presented by the IETF Chair?  I used to follow all the working groups in
> an IETF Area.  I knew how many persons from my region participated on the
> working group mailing lists.

WG participation and in-person attendance could be other metrics.
Maybe we can start collecting data bout remote participation as well.

> states that there were people from 11 countries, including Brazil, Costa
> Rica, Bolivia, Venezuela, Argentina, India, Pakistan, Morocco, Ethiopia, and
> Benin.  According to the minutes, presenting a draft is a working group
> activity.  If several persons from South America put some effort to join a
> working group mailing list discussion to understand a protocol better so
> that they can make useful comments, is that a contribution from that region?
> Would it be positive for the region and for the IETF?
>
>> 4. We need consensus.  We should have metrics that we all agree on are
>> "good enough".  We will never have perfect metrics.
> Agreed.

I believe the consensus would be "rough". But if it is directional
enough even if not completely representative.

Thanks
Vinayak

---

From: S Moonesamy <[email-masked]>
Date: Thu, 14 Jul 2016 08:04:57 -0700

Hi Vinayak,
>http://www.arkko.com/tools/stats/countrydistr.html

You could use the word "authors".

Regards,
S. Moonesamy

---

From: <[email-masked]>
Date: Thu, 14 Jul 2016 20:18:53 +0000 (UTC)

>emails.  That is not my intent.
Not at all.  I feel that you are very interested in the topic (as am I) and I am very glad to have your thoughtful comments.   That is how we will come to the best conclusion.
<snip>

>from the same company nor from the same region.

But I thought this MTGVENUE WG was formed to define and discuss such issues.  Is this not the right place for this conversation?  Why should we do it at the Plenary?

>counted; at some point it may viewed that it is better to go away.

Why do you say that they are not counted?  Is it because they are participating remotely?   Do you think they THEY feel that it is "better to go away"?   I think that is what we very much want NOT to happen.

>in the United States.  
I am going to say that [email-masked] is a somewhat special case.   Some of us in the U.S. (and I count myself among them!) feel quite free to voice our opinions on all kinds of topics.   We take our freedom of speech seriously!   I know I feel sometimes that I should probably "cool it" and go write some code instead of spouting off on the [email-masked] list.  So, I suppose there may be some cultural issues here. I think it is probably more important to look at contributions to actual WG lists.  I know there is the point about WGLC comments on the [email-masked] list.  That complicates the issue.
<snip> 

>participation".  

Yes.  We should have some fair, agreed on metrics.

>Should I use the statistics for recent RFCs?  Should >I use the attendance statistics which are presented by the IETF 

>the region and for the IETF?

IMHO, I think the kind of effort that you are talking about is wonderful.   I don't know how others feel.

The IETF Mentoring program has been trying to put together Internet Draft Review Teams with just the intent that you state above.   We just started a team with 2 people from Africa, one from India and one from Latin America and two mentors.   I have high hopes for such teams.   Although, it takes some coordination to figure out how the team can work together.  We also need some tools!!!   BTW, if anyone wants to help with this, please contact me unicast.    There is quite a demand for this kind of team - I have plenty of people who want to be on a team that I haven't matched up.  Unfortunately,  I need to do my day job every once in a while!

Thanks,Nalini

---

From: Melinda Shore <[email-masked]>
Date: Thu, 14 Jul 2016 12:31:13 -0800

On 7/14/16 12:18 PM, [email-masked] wrote:
> But I thought this MTGVENUE WG was formed to define and discuss such
> issues.  Is this not the right place for this conversation?  Why should
> we do it at the Plenary?

I could be wrong but I don't think he was suggesting that,
but rather putting it out as a hypothetical.

That said, it really isn't clear to me that a measurement activity
actually would belong in this working group, but I'm comfortable
that it's the right place to start a discussion and I do think
that there's value in this kind of metric if it's done well.
So, let's talk about it during the session on Monday and see how
it goes.

Melinda

---

From: S Moonesamy <[email-masked]>
Date: Thu, 14 Jul 2016 18:12:28 -0700

Hi Nalini,
At 13:18 14-07-2016, [email-masked] wrote:
>Not at all.  I feel that you are very interested in the topic (as am 
>I) and I am very glad to have your thoughtful comments.   That is 
>how we will come to the best conclusion.

Thanks.

>issues.  Is this not the right place for this conversation?  Why 
>should we do it at the Plenary?

There were three points on which I commented:

   (a) we have good strong contributors participating via email

   (b) why should (a) not be counted?

   (c) I want to get consensus that we should count such things ...

The problem was about the select of meeting venues for physical 
meetings and the fact that it is a recurrent topic of 
discussion.  That is not related to (a).  At some point in the 
discussions it may be possible to figure out further discussion about 
(b) is okay or not.  You already identified what you should do 
((c)).  It is not a matter of right place for the conversation; it is 
about what would gain IETF Consensus.  I was not suggesting to try 
and do that at the plenary.

>Why do you say that they are not counted?  Is it because they are 
>participating remotely?   Do you think they THEY feel that it is 
>"better to go away"?   I think that is what we very much want NOT to happen.

There isn't any count of mailing list participation.  I would have to 
agree with the person if he/she decides that it is better to go away.

>I am going to say that [email-masked] is a somewhat special 
>case.   Some of us in the U.S. (and I count myself among them!) feel 
>quite free to voice our opinions on all kinds of topics.   We take 
>our freedom of speech seriously!   I know I feel sometimes that I 
>should probably "cool it" and go write some code instead of spouting 
>off on the [email-masked] list.  So, I suppose there may be some 
>cultural issues here.

That would be the First Amendment.  My comment was about IETF Last 
Call participation. I do not consider it (last sentence) as cultural 
issues.  I don't think that you did or said anything to discourage 
other persons from participating in any such thread.  To be clear, 
please do not read anything I said as implying that it is better for 
you or anyone else not to comment on [email-masked].

>I think it is probably more important to look at contributions to 
>actual WG lists.  I know there is the point about WGLC comments on 
>the [email-masked] list.  That complicates the issue.

WGLC comments are on the Working Group mailing list.  My suggestion 
was not to take the [email-masked] IETF Last Call participation into 
consideration for statistical purposes.

>wonderful.   I don't know how others feel.

It is worthwhile to consider whether the effort would produce results.

>The IETF Mentoring program has been trying to put together Internet 
>Draft Review Teams with just the intent that you state above.   We 
>just started a team with 2 people from Africa, one from India and 
>one from Latin America and two mentors.   I have high hopes for such 
>teams.   Although, it takes some coordination to figure out how the 
>team can work together.  We also need some tools!!!   BTW, if anyone 
>wants to help with this, please contact me unicast.    There is 
>quite a demand for this kind of team - I have plenty of people who 
>want to be on a team that I haven't matched up.  Unfortunately,  I 
>need to do my day job every once in a while!

It was one suggested to me to have review teams.  I tried it and it 
did not work as expected.

Regards,
S. Moonesamy