Subject: [Manycouches] Survey of remote then not remote

From: Jay Daley <[email-masked]>
Date: Mon, 7 Sep 2020 15:11:59 +1200

Previously, Mirja suggested [1]:

> But maybe it would actually be interesting to survey both - reasons why people used the waiver last time and also those who usually participate remotely but didn’t last time.

For the second part of that the intent is to directly email all those email addresses used for a remote registration for IETF 103, IETF 104, IETF 105 or IETF 106 that were not also used to register for IETF 108, and ask the recipients to fill out a survey to understand why they did not participate.  

To see what we’re planning to ask, you can either go direct to the SurveyMonkey preview [2] and comment there or try to read the survey plan below.  Your feedback is welcome.

Jay

[1]  https://mailarchive.ietf.org/arch/msg/manycouches/y7gCXC0dsxdZcTZbCV6DTzleHuk/
[2]  https://www.surveymonkey.com/r/Preview/?sm=bKEqxglR93Ag7tYyNpjoU4IQI0oIfeYRDCouRQrnfSk7gLilLx0b_2Bm3AkmQM1_2BTc

—— 

[PAGE 1]
Quick Check

[HELPTEXT]
Thank you for taking part in this very short survey with at most four questions.  We are hoping to hear from people who participated remotely in IETF 103, IETF 104, IETF 105 or IETF 106 and who did not participate in IETF 108 (which was a fully remote meeting held in July 2020) to understand why you did not participate.

As we have used email addresses of registered remote participants for this survey and people use more than one email address, these first two questions quickly check if this survey is for you.

PLEASE NOTE: This survey is 100% ANONYMOUS - we do not track you at all.

[QUESTION 1 - Mutlple Choice]
Did you register for IETF 108?
- Yes
- No

[QUESTION 2 - Multiple Choice]
Did you participate in IETF 108?
- Yes
- No

[LOGIC]
If QUESTION 1 is "Yes" or QUESTION 2 is "Yes" then skip to end of survey.

[PAGE 2]
Reason for not participating

[QUESTION 3 - Multiple Choice]
What is the primary reason why you did not participate in the IETF 108 meeting?
- I did not know about the IETF 108 meeting
- I did not want to participate in IETF 108 because I do not participate in the IETF any more
- I did not want to participate in IETF 108 because IETF meetings are not productive enough
- I did not want to participate in IETF 108 because the agenda was not of interest to me
- I wanted to participate in IETF 108 but I had to do something else
- I wanted to participate in IETF 108 but the registration fee was a barrier to participation
- I wanted to participate in IETF 108 but the time of day of the meeting was too difficult for me to participate
- I wanted to participate in IETF 108 but I knew I would have problems with technology / distractions / location
- Other (please specify)

[LOGIC]
If QUESTION 3 is NOT "I wanted to participate in IETF 108 but the registration fee was a barrier to participation" then skip to end of survey

[PAGE 3]
Fee Waivers

[HELPTEXT]
For IETF 108 we offered a fee waiver to anyone who felt that the registration fee was a significant barrier to participation.  These fee waivers were automatically granted to anyone who applied and no checks on eligibility were made.  There were an unlimited number of fee waivers available.

[QUESTION 4 - Checkboxes]

4. Why you did not apply for a fee waiver? (check all that apply)
- I did not know about the fee waivers
- I did not think that I was eligible
- I did not want to stop someone else from getting a fee waiver
- I did not want to explain why I needed a fee waiver
- I did not want someone to decide if I was eligible
- I did not want anyone to know that I needed a fee waiver
- I could not find out how to apply
- The process to apply was too much work / too difficult
- Other (please specify)

[END]

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 7 Sep 2020 10:15:12 +0100

I know that the plan is to address the cost of remote participation, but there is parallel issue with F2F meetings that maybe we should discuss before they resume again.

There may be a number of people that have contributed significantly to the IETF over the years, who wish to continue to help the IETF, but no longer have a sponsor or even regular work. I was thinking of the retired community but this also applies to the unemployed.

It would be good if we could find a way to significantly reduce the fee to this group.

To keep this manageable in terms of numbers and impact, I would suggest that somehow we attach a work contribution criteria to such a concession such as proof of earlier significant contribution, for example some number of accepted drafts, or service through holding IETF office, or some other form of recognised service. Members of such a community would likely only take the concession if they needed to, and would likely only attend if they were still contributing, but I would like to find a way of welcoming to continue to engage with the active IETF community.

- Stewart

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 7 Sep 2020 12:28:20 +0100

Hi Jay,

Good idea to do a survey.

Question 3 omits some cases, e.g. "I would have paid a
smaller fee that better matched my level of interest in
the agenda and the agenda timing." It might be better
to try tease apart which factors influenced decisions
rather than try enumerate all possibilities and then ask
everyone but 1 case to provide no other information.

IOW, maybe make Q3 a "tick all that apply" with a re-spin
on the categories and then lead into Q4 with "if you
considered asking for a fee waiver..."

Cheers,
S.

---

From: Jay Daley <[email-masked]>
Date: Tue, 8 Sep 2020 00:08:19 +1200

> On 7/09/2020, at 11:28 PM, Stephen Farrell <[email-masked]> wrote:
> 
> 
> Hi Jay,
> 
> the agenda and the agenda timing."

To put it more neutrally - "the cost/benefit wasn’t good enough".  I will add something to that effect.

> It might be better
> on the categories 

You probably know this, but in order to explain myself, I need to contextualise my answer as follows:

Both "multiple choice - single answer" questions and "multiple choice - multiple answer" (checkboxes in my plan below) have their pluses and minuses.  With multiple answer it is often easier for people to answer if the choices do not match their intended response well enough, which I think is your point above, but it can bias low importance factors in a multi-factor response because there is no weighting.  Single answer pushes people to given the single most important factor but runs the risk that people cannot answer (and so choose the 'Other' option) and needs enough responses to get a useful result.

In this particular case, we’ve already asked one question about non-participation in the post-108 survey and while that wasn’t a lot of responses it does tend to validate the possible answers.  There’s also been good analysis on this list and ietf@ that seems to have enumerated the full set of concerns.  And one piece of knowledge I haven’t shared is that this will go out to >2000 email addresses which should provide enough response for a useful result.  Consequently, in this case I’ve gone for the single answer variant as most likely to deliver the cleanest result with the risk being that I need to read through multiple 'Other' answers in order to categorise those.

> and then lead into Q4 with "if you

I think it’s important we get data from that well-defined group who did not participate because of the fee rather than those who may have been affected but weren’t.

Jay

> <0x5AB2FAF17B172BEA.asc>

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Jay Daley <[email-masked]>
Date: Tue, 8 Sep 2020 00:34:11 +1200

Following up on this - I’ve now changed the plan as follows to address your first point and to pick up on an implicit suggestion about the importance to people of what they were missing out on:

—— 

[PAGE 1]
Quick Check

[HELPTEXT]
Thank you for taking part in this very short survey with at most five questions.  We are hoping to hear from people who participated remotely in IETF 103, IETF 104, IETF 105 or IETF 106 and who did not participate in IETF 108 (which was a fully remote meeting held in July 2020) to understand why you did not participate.

[LOGIC]

[PAGE 2]

[QUESTION 3 - Multiple Choice]
What is the primary reason why you did not participate in the IETF 108 meeting?
- I did not know about the IETF 108 meeting
- I do not participate in the IETF any more
- The time of day of the meeting was too difficult for me to participate
- The agenda was not of interest to me
- I was busy with something else
- IETF meetings are not productive enough to justify the fee
- I could not afford the registration fee
- I knew I would have problems with technology / distractions / location
- Other (please specify)

[QUESTION 4 - Multiple Choice)
Did you want to participate in IETF 108?
- Definitely yes
- Probably yes
- Not sure
- Probably not
- Definitely not

[LOGIC]
If QUESTION 3 is NOT "I could not afford the registration fee" then skip to end of survey

[PAGE 3]
Fee Waivers

[HELPTEXT]

[QUESTION 5 - Checkboxes]

[END]

> On 8/09/2020, at 12:08 AM, Jay Daley <[email-masked]> wrote:
> 
> 
> 
> _______________________________________________
> Manycouches mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/manycouches

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: "John Levine" <[email-masked]>
Date: 7 Sep 2020 14:40:28 -0400

In article <[email-masked]> you write:
>To keep this manageable in terms of numbers and impact, I would suggest that somehow we attach a work contribution
>criteria to such a concession such as proof of earlier significant contribution, for example some number of accepted
>drafts, or service through holding IETF office, or some other form of recognised service. ...

This strikes me as both unworkable and counterproductive. For one
thing, this is exactly the kind if thing that the IETF can't resist
turning into an endless rathole.  If being a WG chair counts, how
about being a RG chair?  Or if a former chair, how recent does it
have to be?  I have no interest in arguing about stuff like this.

If someone asks for a waiver and we say, sorry, you only had two
drafts accepted but you need three, the response is far more likely
to be "never mind, I won't come" than "oh, then I'll pay the $1000."
I would prefer to set the speed bumps as low as possible unless we
have evidence that they need to be higher.

R's,
John

---

From: Keith Moore <[email-masked]>
Date: Mon, 7 Sep 2020 14:44:19 -0400

> have evidence that they need to be higher.
+1

Keith

---

From: Michael Richardson <[email-masked]>
Date: Mon, 07 Sep 2020 14:45:39 -0400

Jay Daley <[email-masked]> wrote:
    >> Question 3 omits some cases, e.g. "I would have paid a smaller fee
    >> that better matched my level of interest in the agenda and the agenda
    >> timing."

    > will add something to that effect.

I think that that's perhaps two choices.
"Cost/benefit wasn't big enough" implies that the benefit the IETF provides
could have been bigger.  There are other places.

For instance, it could also be that there are bumps in expense claim bureaucracy.  
A $199USD non-travel related expense might be easier to claim than a $201USD expense.

Stephen's original wording has within it, that the timezone might have meant
that the person would have only attended a very few sessions.

So, include your more neutral wording as an option if you like, because it
might capture other concepts, I'd keep Stephen's query.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 8 Sep 2020 09:32:21 +1200

Stewart,

As a possibly affected party, I would say that the meeting fee is largely irrelevant. If I attend a f2f meeting as a private (retired) individual, my main costs will be airfare and hotel. It won't be the meeting fee that determiness my decision whether to attend.

It's a different issue for remote attendance, where the fee is the only significant cost.

Regards
   Brian Carpenter

---

From: Jay Daley <[email-masked]>
Date: Tue, 8 Sep 2020 10:56:41 +1200

In response I’ve restructured the survey to try ti split out the various things people want to see.  The new plan is now

[PAGE 1]
Quick Check

[HELPTEXT]
Thank you for taking part in this very short survey with at most eight questions.  We are hoping to hear from people who participated remotely in IETF 103, IETF 104, IETF 105 or IETF 106 and who did not participate in IETF 108 (which was a fully remote meeting held in July 2020) to understand why you did not participate.

[QUESTION 1 - Mutlple Choice]
Did you still participate in the IETF?
- Yes
- No

[QUESTION 2 - Multiple Choice]
Did you know about IETF 108?
- Yes
- No

[LOGIC]
If QUESTION 1 is "No" or QUESTION 2 is "No" or QUESTION 3 is "Yes" then skip to PAGE 4

[PAGE 2]

[QUESTION 4 - Matrix/RatingScale]
How much did each of the following aspects of the meeting affect your decision not to participate?
Items:
- Registration fee
- Timezone
- Meeting schedule (5 days, 6 hours per day)
- Agenda
- Productivity of IETF meetings in general
- Being fully online
Rating:
- Extremely
- Very
- Moderately
- Slightly
- Not at all

[QUESTION 5 - Multiple Choice]
What is the primary reason why you did not participate in the IETF 108 meeting?
- Registration Fee
- Timezone of the meeting
- Schedule of the meeting
- Agenda
- Productivity of IETF meetings in general
- Because it was a fully online meeting
- I was busy with something else
- I knew I would have problems with technology / distractions / location
- Other (please specify)

[QUESTION 6 - Multiple Choice)
Did you want to participate in IETF 108?
- Definitely yes
- Probably yes
- Not sure
- Probably not
- Definitely not

[LOGIC]
If QUESTION 4, "Registration fee" is NOT "Extremely" or "Very" AND QUESTION 5 is NOT "Registration fee" then skip to PAGE 4

[PAGE 3]
Fee Waivers

[HELPTEXT]

[QUESTION 7 - Checkboxes]

[PAGE 4]
Finish

[QUESTION - Comment Box]
Is there anything else you would like to say?

[END]

Jay

> On 8/09/2020, at 6:45 AM, Michael Richardson <[email-masked]> wrote:
> 
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 8 Sep 2020 00:15:44 -0400

I am in the same situation as Brian. The cost of attending virtual IETFs is
much less than the cost of attending in person.

There is a bigger problem with the business plan for participation which I
have been thinking about at length. I can afford to work without a salary,
the lone inventor is a well established archetype for my background. Most
do not have that fortune.

Lets redo DNS and make sure that the inevitable surplus that is created by
sale of names goes to benefit Internet users and developers.

On Mon, Sep 7, 2020 at 5:32 PM Brian E Carpenter <
[email-masked]> wrote:

> Stewart,
>
> As a possibly affected party, I would say that the meeting fee is largely
> irrelevant. If I attend a f2f meeting as a private (retired) individual, my
> main costs will be airfare and hotel. It won't be the meeting fee that
> determiness my decision whether to attend.
>
> > I know that the plan is to address the cost of remote participation, but
> there is parallel issue with F2F meetings that maybe we should discuss
> before they resume again.
> >
> > There may be a number of people that have contributed significantly to
> the IETF over the years, who wish to continue to help the IETF, but no
> longer have a sponsor or even regular work. I was thinking of the retired
> community but this also applies to the unemployed.
> >
> > To keep this manageable in terms of numbers and impact, I would suggest
> that somehow we attach a work contribution criteria to such a concession
> such as proof of earlier significant contribution, for example some number
> of accepted drafts, or service through holding IETF office, or some other
> form of recognised service. Members of such a community would likely only
> take the concession if they needed to, and would likely only attend if they
> were still contributing, but I would like to find a way of welcoming to
> continue to engage with the active IETF community.
> >
> > - Stewart
> >
> >
> >
> >
> >
> >

---

From: Keith Moore <[email-masked]>
Date: Wed, 9 Sep 2020 11:30:44 -0400

> As a possibly affected party, I would say that the meeting fee is largely irrelevant. If I attend a f2f meeting as a private (retired) individual, my main costs will be airfare and hotel. It won't…

Mostly agree, but for me it is the total cost of attending that 
matters.   Often I can find a good deal on a nearby hotel; sometimes I 
can find a good deal on travel or use some of my remaining frequent 
flyer miles.   But there's no way to reduce meeting fees.   So meeting 
fees do matter, even if they're not usually the highest cost.

Keith

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 9 Sep 2020 16:55:03 +0100

> On 9 Sep 2020, at 16:30, Keith Moore <[email-masked]> wrote:
> 
> Mostly agree, but for me it is the total cost of attending that matters.   Often I can find a good deal on a nearby hotel; sometimes I can find a good deal on travel or use some of my remaining frequent flyer miles.   But there's no way to reduce meeting fees.   So meeting fees do matter, even if they're not usually the highest cost.
> 
> Keith
> 

I am approaching the point where I may also have a stake in this as well.

Last time I did an an analysis, using a cheap but reasonable hotel and an economy flight to Europe or the US from London, the meeting fee represented about 30 to 50% of the cost.

Of those components the meeting fee is something that the IETF can help with.

- Stewart

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 10 Sep 2020 09:57:07 -0500

On Wed, Sep 9, 2020 at 10:55 AM Stewart Bryant <[email-masked]>
wrote:

> >> As a possibly affected party, I would say that the meeting fee is
> largely irrelevant. If I attend a f2f meeting as a private (retired)
> individual, my main costs will be airfare and hotel. It won't be the
> meeting fee that determiness my decision whether to attend.
> >
> matters.   Often I can find a good deal on a nearby hotel; sometimes I can
> find a good deal on travel or use some of my remaining frequent flyer
> miles.   But there's no way to reduce meeting fees.   So meeting fees do
> matter, even if they're not usually the highest cost.
> >
> > Keith
> >
>
> Last time I did an an analysis, using a cheap but reasonable hotel and an
> economy flight to Europe or the US from London, the meeting fee represented
> about 30 to 50% of the cost.
>
+1

Meeting fee  does become a factor in F2F meetings.

Behcet