Subject: [Manycouches] Revisiting the "show of hands" tool

From: "Bron Gondwana" <[email-masked]>
Date: Sat, 21 Aug 2021 13:19:57 +1000

Hi All,

I made myself a note after the last IETF to propose that we revisit the show of hands tool based on our experience at the past few online IETFs.  Based on discussions on the mailing lists and in gathertown, here's what I think people want from a tool:

 * more than two options
 * text descriptions for each option (not just Yes/No)
 * ability to select more than one option

To use the example from all the chair trainings early on: icecream flavours.  Rather than asking three questions:

1) do you like icecream Y/N
2) do you like chocolate icecream Y/N
3) do you like vanilla icecream Y/N

You could run a single poll:

Do you like icecream?

[ ] Chocolate
[ ] Vanilla
[ ] Some other flavour

In which everyone can select any of the above.

This is kind of like a bunch of hums - but since it's not "sound", we can run them all in parallel.  Yay technology.

Given that we still have many months until the next IETF, which will be virtual again, and Meetecho's implementation is now based on an expired draft -- we should issue guidance on what we want from the tool for the next IETF if we as a WG want something different than what's there now.

Martin - if you wanna do it, that's great.  Otherwise, I'm happy to write up a draft of how I think it should work - and chat with Meetecho about what's easy to do at their end.

Cheers,

Bron.

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 20 Aug 2021 23:29:39 -0400

I think that this is an attempt to refine something that is deliberately 
supposed to be a crude tool.  Given a choice between making it elaborate 
or simply removing it, I would be inclined to remove it.

Humms are just to try to get a sense of things.  Real decisions have to 
be made on the list.

Yours,
Joel

On 8/20/2021 11:19 PM, Bron Gondwana wrote:
> Hi All,
> 
> I made myself a note after the last IETF to propose that we revisit the 
> show of hands tool based on our experience at the past few online 
> IETFs.  Based on discussions on the mailing lists and in gathertown, 
> here's what I think people want from a tool:
> 
>   * more than two options
> To use the example from all the chair trainings early on: icecream 
> flavours.  Rather than asking three questions:
> 
> This is kind of like a bunch of hums - but since it's not "sound", we 
> can run them all in parallel.  Yay technology.
> 
> 
> Given that we still have many months until the next IETF, which will be 
> virtual again, and Meetecho's implementation is now based on an expired 
> draft -- we should issue guidance on what we want from the tool for the 
> next IETF if we as a WG want something different than what's there now.
> 
> Martin - if you wanna do it, that's great.  Otherwise, I'm happy to 
> write up a draft of how I think it should work - and chat with Meetecho 
> about what's easy to do at their end.
> 
> Cheers,
> 
> Bron.
> 
> --
> _______________________________________________
> Manycouches mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/manycouches
>

---

From: "Bron Gondwana" <[email-masked]>
Date: Sat, 21 Aug 2021 13:42:53 +1000

> Humms are just to try to get a sense of things.  Real decisions have to 

I haven't had to use the tool myself in groups that I chair, so I could also support that. But the cases where I have seen it used, it's been used as a really poor version of a poll to see which of N options have support.

But yeah, I'd also support removing it entirely.  A chair could always post a link to an anonymous poll outside meetecho entirely into the jabber channel.

Either way, the current tool is a poor user experience every time it gets used.

Bron.

---

From: John C Klensin <[email-masked]>
Date: Sat, 21 Aug 2021 02:18:49 -0400

--On Saturday, August 21, 2021 13:19 +1000 Bron Gondwana
<[email-masked]> wrote:

> Hi All,
> 
> I made myself a note after the last IETF to propose that we
> revisit the show of hands tool based on our experience at the
> past few online IETFs.  Based on discussions on the mailing
> lists and in gathertown, here's what I think people want from
> a tool:
> 
>  * more than two options
> To use the example from all the chair trainings early on:
> icecream flavours.  Rather than asking three questions:
> 
> This is kind of like a bunch of hums - but since it's not
> "sound", we can run them all in parallel.  Yay technology.

Bron,

While I generally agree with Joel that trying to turn a
deliberately crude tool into a precision one and about sorting
out details on mailing lists, your example above illustrates one
of the traps of trying to turn crude hums into a multiple-choice
question.  Suppose someone loathed ice cream.  With the existing
arrangement, they pick "N" to each question.  With your list,
there is no "none of the above, I hate the stuff" option.  Now,
with that insight, one could obviously just add another option,
But it is easy to imagine more complex and less obvious
questions and I can remember in-person meetings in which an
announcement of a hum causes an immediate (and often
time-consuming) debate about exactly how the hum should be
stated.    A sequence of Y/N questions does not eliminate that
problem, but it actually simplifies things considerably and
lowers the risk of getting answers that it is not possible to
interpret accurately.

best,
   john

---

From: "Bron Gondwana" <[email-masked]>
Date: Sat, 21 Aug 2021 22:06:05 +1000

> lowers the risk of getting answers that it is not possible to

Oh, I think there should always be a "none of the above" option, I should have included that in my message.  Of course explicitly not answering any is the same as that (with a multiple choice) but generally the way I've heard this done in working groups is something like.

Hum if you like option A
Hum if you like option B
Hum if you think the IETF shouldn't do this work at all

Which is three hums.  In a room, that can be done more quickly than in an online tool where half the people are faffing around saying "what does raising a hand mean again", "what was the question", "...".

Whereas:

( ) None of the above (aka, no opinion); or

[ ] Option A (we post everybody a chocolate icecream along with their shirt)
[ ] Option B (we post everybody a vanilla icecream long with their shirt)
[ ] Option C (posting icecream is a crazy idea and the IETF should not do this)

Allows everybody to have their say in a single pass, and still allows for the possibility of "multiple hums" - anybody can hum for more than one thing in a hum, though obviously humming for both C and either of A or B is logically inconsistent - but the same applies for humming in a room.

Bron.

---

From: John C Klensin <[email-masked]>
Date: Sat, 21 Aug 2021 10:58:51 -0400

(TL;DR warning: This note goes some distance into illustrating
why opinion measurement is hard. Those who are uninterested in
that subject should probably skip the message or jump to the
summary comment at the end.)  

--On Saturday, August 21, 2021 22:06 +1000 Bron Gondwana
<[email-masked]> wrote:

>> out details on mailing lists, your example above illustrates
>> one of the traps of trying to turn crude hums into a
>> multiple-choice question.  Suppose someone loathed ice cream.
>> With the existing arrangement, they pick "N" to each
>> question.  With your list, there is no "none of the above, I
>> hate the stuff" option.  Now, with that insight, one could
>> obviously just add another option, But it is easy to imagine
>> more complex and less obvious questions and I can remember
>> in-person meetings in which an announcement of a hum causes
>> an immediate (and often time-consuming) debate about exactly
>> how the hum should be stated.    A sequence of Y/N questions
>> does not eliminate that problem, but it actually simplifies
>> things considerably and lowers the risk of getting answers
>> that it is not possible to interpret accurately.
> 
> Oh, I think there should always be a "none of the above"
> option, I should have included that in my message.  Of course
> explicitly not answering any is the same as that (with a
> multiple choice) but generally the way I've heard this done in
> working groups is something like.

I don't want to get pedantic about this, just to point out the
difficulties in designing a UI and underlying software to set
polls up in real time [1].  There really are some questions with
binary choices out there in which "none of the above" makes no
sense.  So, think for a bit about the difficulties of a chair or
co-chair trying to set one of these up, asking those attending
if the formulation is correct, adjusting things as needed, etc.
Might still be better in some situations, but I'm not sure that,
overall, it saves any time.

> Which is three hums.  In a room, that can be done more quickly
> than in an online tool where half the people are faffing
> around saying "what does raising a hand mean again", "what was
> the question", "...".

Noting that faffing around happens in-room too (and thinking
about the implications for hybrid meetings with many people
in-room and many online), yes, maybe.

> the IETF should not do this)

But, pushing this example a bit further (and extrapolating only
a bit from some other discussions we have had about food).

 [ ] Option D (I think this is a great idea in principle, but
I'm lactose-intolerant or vegan and there should be a frozen
yogurt or non-dairy alternative, also with flavor choices)
 [ ] Option E (ice cream really doesn't ship very well but we
should optionally send cookies, which would be more traditional
anyway)

Note that neither of these is "None of the above" and that E,
while it can reasonably be seen as overlapping C, is a bit
different and contains far more information.

> Allows everybody to have their say in a single pass, and still
> allows for the possibility of "multiple hums" - anybody can
> hum for more than one thing in a hum, though obviously humming
> for both C and either of A or B is logically inconsistent -
> but the same applies for humming in a room.

But where it fails is "everyone has their say" as those
additional questions point out.

Pushing this example even further, if one were trying to extract
maximum information while ensuring that everyone has their say,
this particular question would probably be structured as:

(1) Do you like the idea of sending out food as well as shirts 
   [yes]  [no]

(2) (Only if there are sufficient "yes" answers to make
continuing useful:
   How about foods that require refrigeration or special
packaging like ice cream?  Worth the expense and trouble?  
   [yes] [no]

(3) Contingent on that answer, maybe you ask about ice cream and
flavors along the lines of the above (including D and E).  Or
you go down the cookies and cake or fruit paths.  Or maybe you
need to ask first about sending coffee (in liquid form and
warm/hot) versus sending something frozen.

Now _that_ would let everyone have their say.  It is also
time-consuming and tedious.  And, from observation, even many
people who are experienced professionals in opinion measurement
and instrument design who have been doing that stuff for years
typically need multiple passes and "what if" examinations to get
things right.  Most of them would never expect to formulate such
a hierarchy of questions in real time and I'm quite confident
they would poke holes in my quickly constructed example (I would
probably even do so if I put this note aside for 24 hours and
then reread it).

Conclusions:

(i) Designing a compound question that captures a broad and
comprehensive range of opinions is hard, getting it right in
real time is harder, and designing a software system and UI to
allow such a question to be easily stated but not fall into
obvious traps (like leaving out "none of the above") is harder
than it might seem.

(ii) There are many circumstances (of which your ice cream
questions might not be a good example) in which cascading
questions are really needed rather than a single multiple-choice
question.  Again, the decision about how to formulate such
questions may be hard to make well in real time in the middle of
a meeting.

(iii) Probably, coming back to Joel's comment, we should keep
our expectations for capturing information of any significant
complexity (even multiple choices) in hum-like activities fairly
low. 

(iv) Your comments may suggest something else which, IMO, could
be extremely useful if WG Chairs could get themselves organized
early enough.  With many WGs, the pattern is that the Chairs get
together a few days before an IETF meeting (or later), throw
together an agenda, and post it either by the nominal deadline
or within less than 24 hours of the meeting itself, relying on
taking up meeting time for agenda bashing and refinement.
Suppose, instead, they posted, a couple of weeks before the
meeting, a summary of what they thought were hot issues,
including questions of the type you are suggesting about them,
when there seemed, from list discussions, to be clusters of
opinions.  For topics on which there appeared to be general
agreement (remembering that this is on the mailing list), they
could ask questions about whether the topic needed more
discussion or make a consensus call and avoid spend meeting time
on them.  For more controversial topics, the answers might help
inform a more efficient and effective agenda.

Summary comment: Partially because of the issues illustrated
above, I have never been a fan of using hums (even in-room) for
anything other than confirmation questions such as "are we
sufficiently agreed that, pending confirmation on the mailing
list, we can move on?" or "are we ready to move to the next
topic?".  Even then, meetings with the vast majority of
participants in-room are better venues for hums that online
meetings because a chair can look around and make guesses from
the facial expressions of those who are not making hum-type
noises [2].  Even then, they are sometimes hard to interpret.  I
don't know if this parallels Joel's reasoning, but I am guessing
that adding things to Meetecho that would encourage asking more
complex questions would lead to more answers that are hard to
interpret accurately.  Either the people in the meeting would
detect that, leading to either debates about the questions or
debates about interpretation and the need to do more hums (both
costly in time) or they wouldn't and the result would be on-list
debates about whether the hum meant anything.  And, again, I
think we have many worked examples of those things happening.

best,
   john

[1] To the extent the difference is important, as soon as we get
to the niceties of multiple choice answers, we are talking about
polling, not traditional-style humming.

[2] This is another situation in which great care is needed to
prevent input from remote participants from getting lost.

---

From: Bob Hinden <[email-masked]>
Date: Sat, 21 Aug 2021 11:02:26 -0700

I have mixed feeling about improving the tool.

It’s hard to design effective questions, especially in real time.

We probably have some common questions that could be selected from.  Some examples:

o Do you support [adopting|advancing|] <draft-wgname-xxxx>?

   Yes
   No
   [No response means don’t care]

o Have you read <draft-xxxx>?

   Yes
   No
   [No response means don’t care]

But asking complicated effective questions is hard.  As other have pointed out the ice cream question in Bron’s email, it should have been “what flavor of ice cream do you like”.   Asking if you like ice cream and then give the choices for flavors, presupposes that one already likes ice cream.  It’s like the old car salesman trick of asking about what color car do you want, skipping the do you want to buy the car question.  It’s very easy to do this badly.

I could support some pre-formatted common questions chairs might ask the w.g., but expecting w.g. chairs to make good use of more free form questions is not likely to be very useful.

Bob

---

From: "John Levine" <[email-masked]>
Date: 21 Aug 2021 14:39:50 -0400

>It’s hard to design effective questions, especially in real time.

Qualtrics has built an entire business around designing and evaluating survey questions.
I don't think we want to go there.

I'm with the minimalists, hum if you agree with X, with no more options than loud, quiet, none.
Allow perhaps 10 seconds for people to choose one, not the very long countdown timer they tried before.

R's,
John

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 23 Aug 2021 10:34:49 +0200

The one change I think is really needed is to change “Raise hand/Don’t raise hand/Don’t participate” to “Yes/No/Don’t participate” because the current labels are constantly confusing to people.

> On 21. Aug 2021, at 20:02, Bob Hinden <[email-masked]> wrote:
>

---

From: Martin Duke <[email-masked]>
Date: Mon, 23 Aug 2021 10:31:38 -0700

Hi Bron,

Thanks for your continued interest.

I'm happy to fire up the markdown again, and bring you on as a co-author if
you like.

I agree that the current setup, though faithful to the in-person hum, is a
bit clunky. At the same time, I share the concern that we might build a
foot gun for chairs. But something useful can be done here, and I can take
the end result to Jay for implementation.

On Mon, Aug 23, 2021 at 1:35 AM Mirja Kuehlewind <[email-masked]>
wrote:

> The one change I think is really needed is to change “Raise hand/Don’t
> raise hand/Don’t participate” to “Yes/No/Don’t participate” because the
> current labels are constantly confusing to people.
>
>
>
> > But asking complicated effective questions is hard.  As other have
> pointed out the ice cream question in Bron’s email, it should have been
> “what flavor of ice cream do you like”.   Asking if you like ice cream and
> then give the choices for flavors, presupposes that one already likes ice
> cream.  It’s like the old car salesman trick of asking about what color car
> do you want, skipping the do you want to buy the car question.  It’s very
> easy to do this badly.
> >
> > I could support some pre-formatted common questions chairs might ask the
> w.g., but expecting w.g. chairs to make good use of more free form
> questions is not likely to be very useful.
> >
> > Bob
> >

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 24 Aug 2021 21:03:40 +1000

> different and contains far more information.

You're right.  In the room there's usually a degree of grumbling / heckling if anyone feels that the options are good enough.  Which basically means that you need (as I just suggested in another message) an:

"Other"

and a

"Reject the premise of the question".

Which can probably be included with "none of the above" - because both mean "there choices offered are not a complete set".

Either way - any of those responses should be considered very seriously by the chairs - and we can definitely include wording along the lines of "if you the chair get one of these in your result set, the other values are meaningless for consensus building purposes until you have asked those who disagree to propose alternative options to be added - and re-run the question with those options present".

Bron.

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 24 Aug 2021 20:59:02 +1000

I mean - chairs can still create a Yes/No easily with something that allows them to name multiple options.

I definitely agree that the "Yes/No" is clearer than "Raise/Don't Raise" -> and questions can easily be phrased as a Yes/No if they're a hum equivalent.

But .... there's not much point dusting off the markdown if we are going to get strong pushback from this group, so maybe we should keep discussing the objections to allowing multiple options.

It sounds like every time there need to be the following options available:

(*) do not participate (DEFAULT)

[ ] option A
[ ] option B
[ ] option C
[ ] Other

( ) None of the above / reject the premise

Obviously if there's any "Other" or "Reject the premise / none of the above" that's a clear message to the chairs that they haven't captured the various opinions that are in the room adequately.

Bron.

On Tue, Aug 24, 2021, at 03:31, Martin Duke wrote:
> I'm happy to fire up the markdown again, and bring you on as a co-author if you like.
> 
> I agree that the current setup, though faithful to the in-person hum, is a bit clunky. At the same time, I share the concern that we might build a foot gun for chairs. But something useful can be done here, and I can take the end result to Jay for implementation.
> 
> On Mon, Aug 23, 2021 at 1:35 AM Mirja Kuehlewind <[email-masked]> wrote:

---

From: John C Klensin <[email-masked]>
Date: Tue, 24 Aug 2021 08:56:37 -0400

Bron,

I think this is closer to being on the right track if there is
consensus to move in that direction.   However, I continue to
believe that we are better off having discussions than either
voting (however fuzzy or disguised) or trying to serially filter
out alternatives with multiple-choice questions.  Consequently
my instinct is to generally confine in-room humming to
confirmatory questions and to question whether it is actually a
useful mechanism for online meetings.

best,
   john

--On Tuesday, August 24, 2021 21:03 +1000 Bron Gondwana
<[email-masked]> wrote:

>> But, pushing this example a bit further (and extrapolating
>> only a bit from some other discussions we have had about
>> food).
>> 
>> should optionally send cookies, which would be more
>> traditional anyway)
>> 
> You're right.  In the room there's usually a degree of
> grumbling / heckling if anyone feels that the options are good
> enough.  Which basically means that you need (as I just
> suggested in another message) an:
> 
> "Other"
> 
> and a
> 
> Which can probably be included with "none of the above" -
> because both mean "there choices offered are not a complete
> set".
> 
> Either way - any of those responses should be considered very
> seriously by the chairs - and we can definitely include
> wording along the lines of "if you the chair get one of these
> in your result set, the other values are meaningless for
> consensus building purposes until you have asked those who
> disagree to propose alternative options to be added - and
> re-run the question with those options present".
> 
> Bron.
> 
> --

---

From: Barry Leiba <[email-masked]>
Date: Tue, 24 Aug 2021 09:55:20 -0400

I have very strong agreement with this.

When I'm chairing, I very rarely use hums/hands, and very much prefer
to assess the discussion, state what the dominant proposal appears to
be, and then ask for those who object to that to take the mic and
state their objections.

I find that "Does anyone object to vanilla?" works far better than
"Who likes vanilla / who likes chocolate?"  I'll take raises of hands
to assess who's actually willing to do the editing/reviewing/whatever,
but that's a very different sort of question.

Barry

---

From: Michael Richardson <[email-masked]>
Date: Tue, 24 Aug 2021 14:06:01 -0400

Bron Gondwana <[email-masked]> wrote:
    > But .... there's not much point dusting off the markdown if we are
    > going to get strong pushback from this group, so maybe we should keep
    > discussing the objections to allowing multiple options.

I think that we have identified that there are a couple of different polls^Whums
that chairs might want to do.   Some thoughts:
a) chairs should be able to set them all up ahead of time.
   (Let participants see them in advance too)

b) we need to capture the results better.  This means getting it into the
   video feed.  An automated summary to the jabber would also help.
   An automated "poll is open" to the jabber might also help.

There is a further type of survey that is the "who will review this
document", which is a poll in the sense that if the number is below some
threshold, then the document might now advance.  At the same time, the people
who volunteered need to be minuted. (It's *not* anonymous)

I suggest that maybe the situation is complex enough that it needs to be
split out of *meetecho* itself.   Could the polling system be a separate
application, which produced a video/screen-share feed that the chairs could
then mix in?

This would benefit because:
1) we could have many different such systems, which different people evolve according
   to their desired style.
   {we have oauth into DT, so much of the hassle of such a system is
   delegated to DT}

2) we'd have to create this interface, which might have other benefits, yet
   to be imagined.

3) the polling system could be in a different tab.  I've missed polls because
   I forgot to switch within meetecho.

4) the chair, when configuring the poll, could be sharing that interface with
   the WG, in order to get the question right.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Michael Richardson <[email-masked]>
Date: Tue, 24 Aug 2021 14:08:43 -0400

> I have very strong agreement with this.

Also I.

    > but that's a very different sort of question.

Chairs need more training on doing this.
(including me)

We do need a way to capture the raises hand to volunteer thing.
The polls don't get into the jabber log or the video, unless it is copied there.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Fred Baker <[email-masked]>
Date: Tue, 24 Aug 2021 13:03:07 -0700

I would agree with your viewpoint. A hum is a quick “sense of the room”, nothing more. If one is going to ask several questions and can do it in one, fine. But the quip about lipstick on a pig comes to mind; when we ask for a hum, we’re asking for what direction a conversation should turn, not finalizing a ballot.

Sent using a machine that autocorrects in interesting ways...

> On Aug 20, 2021, at 8:29 PM, Joel M. Halpern <[email-masked]> wrote:
> 
> I think that this is an attempt to refine something that is deliberately supposed to be a crude tool.