Subject: Re: [admin-discuss] [110all] IETF 110 survey reminder

From: John C Klensin <[email-masked]>
Date: Sun, 14 Mar 2021 19:17:09 -0400

TL;DR summary: Please consider that many of us have work to do
to follow up on substantive technical issues resulting from the
meeting.  If a survey is time-consuming and has no partway exit
path, at least for people with other commitments, you force a
choice between "complete survey" and "get IETF work done".
There is no winning choice.

Rant below.

--On Monday, March 15, 2021 09:44 +1300 IETF Executive Director
<[email-masked]> wrote:

> This is a reminder to please help us understand and improve
> the meeting experience by filling out this short survey while
> it is fresh in your mind
> 
> 	https://ietf.iad1.qualtrics.com/jfe/form/SV_6hTsDUOGTJJtTh4
> 
> We are particularly interested to understand what impact was
> on the meeting experience of the new pre-meeting system
> testing, new conflict resolution technology and new integrated
> helpdesk system.

> We've shortened this survey since the last meeting while
> keeping plenty of opportunity for you to share your views.  We
> need as many responses as possible please.

I don't know if I'm the only one, but it is still too long, with
too many mandatory questions and two many questions that take a
good deal of time to answer adequately.  If the survey
instrument is set up so that someone who gets about halfway
through then gives up because trying to move off a given page
gets stuck with a message of "Please answer this question", you
not only don't get answers to the remaining questions but don't
get the first half of the answers (the ones already filled in)
either.  Maybe that is what you want, but it creates a serious
case of what professional survey statisticians and researchers
call sample bias.  If the goal is to have something to count,
that is fine; it is is to get information about the whole set of
IETF participants...  Well, you might consider offering an
alternate survey with no more than the participant-identifying
information at the beginning and a question of why people were
unwilling to invest the time and effort to complete it.  Or,
better yet, provide an exit path after each page of questions
that captures questions already answered and "why did the
respondent not go any further" information without that being
burdensome.

If, as indicated above, you are particularly interested in three
sets of questions, then decide whether, if you cannot get every
respondent (and, ideally, every participant, to respond to all
three, organizing the survey so that people can respond to only
one or two would be a good idea no matter how much you encourage
responding to all three.  Or, taking opinion measurement
sophistication up a notch, think about sampling, subsets of
questions for different respondents, etc.

best,
   john

---

From: Jay Daley <[email-masked]>
Date: Mon, 15 Mar 2021 15:03:12 +1300

> On 15/03/2021, at 12:17 PM, John C Klensin <[email-masked]> wrote:
> 
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
> too many mandatory questions

There were a grand total of four mandatory questions (out of a maximum of 32), two of which were necessary for branching and two for meaningful demographics.  The latter has now been reduced to 1 leaving three mandatory questions. (the one I made optional was how many conflicts did you experience).

> and two many questions that take a
> good deal of time to answer adequately.

The median time to complete is currently 382 seconds and the first quartile is 259 (that includes several outliers of over an hour that I have not cleaned up).  As a reminder, someone can spend up to 108,000 seconds participating in an online meeting, so this is asking them to spend 0.35% of that time on making it better for the next time around.

>  If the survey

The survey software identifies partial vs complete answers and all partial answers are discarded to avoid early answer bias.

> questions for different respondents, etc.

The three things above were highlighted to preview the contents of the survey and encourage people to respond.  

I think it’s important to remember that we’re not doing surveys that determine critical resource allocation such as funding formulae for hospitals.  What we’re doing is feeding a process of continuous improvement - we find out what the issues are, make changes to address them, measure the impact of the changes - all in overlapping cycles.  For that purpose this form of surveying is perfectly acceptable and does not need the level of sophistication, effort and resources you propose.

However, as always, thank you for the feedback.

Jay

> 
> best,
>   john
> 
> 
> -- 
> admin-discuss mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/admin-discuss <https://www.ietf.org/mailman/listinfo/admin-discuss>
-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 15 Mar 2021 02:06:34 +0000

On 15/03/2021 02:03, Jay Daley wrote:
> The median time to complete is currently 382 seconds

Bugger. Now I wish I'd recorded how long it took me:-)

It wasn't long though.

S.

---

From: John C Klensin <[email-masked]>
Date: Sun, 14 Mar 2021 23:21:16 -0400

> Bugger. Now I wish I'd recorded how long it took me:-)

It may be, and probably is, that I'm just not working as rapidly
as I used to and/or that I was trying to answer questions
carefully and that sometimes took going back to other files and
looking things up, but I was over eight minutes when I gave up
and, if the little bar at the bottom is to be believed, that was
about at the halfway point (and at one of the questions that
turned out to be mandatory after I determined that I didn't want
to go back to my agenda notes looking for conflicts that I had
put aside once I determined which of the sessions I was going to
attend.

And, fwiw, medians are interesting but, if one is going to keep
enough statistics to compute and report on them, understanding
what was going on around the upper hinge, even if not the
maximum, would be worthwhile).

    john