Subject: WG Meeting Structure / Alternate Room Layout

From: Adam Roach <[email-masked]>
Date: Thu, 27 Jun 2019 16:30:53 -0500

Hi. I'm following up from a recent thread on this mailing list ("WG meeting
structure" [1]) discussing the prospect of running additional meetings 
with a
"U-Shaped" table structure for the purpose of facilitating more natural
back-and-forth discussion of topics.

As many of you will probably recall, we ran an experiment at IETF 98 in 
Chicago
in which several working groups opted in to using such a layout. Those 
groups
that participated were QUIC, REGEXT, NFV, LAMPS, and HTTPBIS.

The stated goal of this experiment was basically the same as the goal put
forth in the thread cited above: to reduce the formality of a strict
microphone queue and encourage a more free-flowing back-and-forth 
exchange, in
the hope that this more informal style of discussion would lead to more 
rapid
progress.

The feedback that the IESG received immediately following the experiment was
mixed, with significantly more negative feedback than positive.

An exhaustive list of the positive feedback received follows:

   1.  A QUIC chair indicated: “seems to have worked well.”

   2.  One QUIC attendee noted that it “allowed a much greater
       volume of comments to be made by cutting down on overhead.”

   3.  Remote participants were able to more easily see expressions/
       reactions of people at the table

The negative feedback included the following points:

   1.  It is hard to tell who is speaking (almost everyone who gave
       feedback mentioned this)

   2.  Chairs found that sitting with their back to room was sub-optimal
       (almost all chairs indicated this). This had two reported impacts:

       –  The chairs can’t see people queuing at the floor mic

       –  The chairs don’t get a good feel for room assent/dissent

   3.  Single mic for non-table participants required long walks for
       some people (several participants indicated this)

   4.  Division between table and non-table perceived as elitist,
       especially when off-mic side-conversations broke out at the
       table (2 comments)

   5.  Having to turn head 90° to see the slides is uncomfortable
       (2 comments)

   6.  Chairs had no dedicated space, no dedicated mic, were pulled
       aside by neighboring IETFers into side conversations during
       meetings.

   7.  Several people who elected to sit at the table were not active
       participants, and took up space that was needed by active
       participants. Attempts to relocate these people were awkward
       and occasionally ineffective.

   8.  People were reluctant to fill in on “side” seating area, opting
       instead to stand at the back of the room.

While the volume of negative feedback was significantly greater than the
volume of positive feedback, there are two factors that should also be
taken into consideration. The first is that people are typically more
proactive about giving negative feedback than positive, so actual sentiment
on the topic may skew more positive than the preceding summary indicates.
The second is that several of the issues described in the "negative 
feedback"
list are not inherent in the revised layout, and may be possible to 
address if
the experiment were re-run.

Unfortunately, while there were plans to solicit feedback on this room 
layout
as part of the post-meeting survey for IETF 98, due to a technical 
issue, that
survey was never distributed to the IETF community.

In the more recent thread on this WG Chairs mailing list, the following 
issues
were either re-iterated from the initial feedback above, or mentioned by 
more
than one person:

     - Remote participants found meetings in the alternate layout more
       difficult to follow.

     - Many people who could not fit at the front tables felt excluded.

     - Everyone (both local and remote) had difficulty figuring out who was
       speaking, including Jabber scribes.

     - The style of queuing discipline required for this layout is
       unfamiliar to existing WG chairs, and many did a poor job with it.
       Non-front-row and remote participants particularly suffer from this.

     - There was no direction for chairs to face where they could see
       everything that was going on.

At the same time, there were approximately as many comments that such an
alternate layout seemed like a good idea to try. It was unclear in some 
cases
whether commenters were aware of the previous experiment or the community's
comments on it.

There were several suggestions for incremental improvements on the previous
experiment, both in the post-IETF-98 feedback and during the more recent
discussion on the WG Chairs list. Some of these included:

     * Place chairs where they can see the microphone

     * Give chairs dedicated table/mic

     * Experiment with shape of “U”
         – Face toward room instead of away from it
         – Make into open “V” facing room

     * Use two or more screens to display slides

     * Only allow this layout for very small groups

     * Use more and/or a different style of microphones

Given the foregoing, including in particular those aspects of negative
feedback that are inherent to the layout, the IESG would like feedback 
from the
chairs regarding whether a re-run of the experiment might be worthwhile.

/a
____
[1] 
https://mailarchive.ietf.org/arch/msg/wgchairs/rMPdilySo8XNU84623TQt5lRnJY

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 27 Jun 2019 22:03:11 +0000

Thanks for compiling the history of feedback
on this topic, Adam. 

> from the chairs regarding whether a re-run of the experiment might be 
> worthwhile.
>

I particularly resonate with:

... as an experiment for this meeting only. 

A lot of the who's-who and
entitlement issues may be solved or mitigated in small scale.
I have hosted and attended meetings with 10 to 20 people,
and the "U" worked well, but my guess is that 40 breaks the 
shared, single experience with some number of the negative 
issues mentioned.

regards,
Al

> -----Original Message-----
> From: WGChairs [mailto:[email-masked]] On Behalf Of Adam Roach
> Sent: Thursday, June 27, 2019 5:31 PM
> To: IETF WG Chairs <[email-masked]>
> Subject: WG Meeting Structure / Alternate Room Layout
> 
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__mailarchive.ietf.org_arch_msg_wgchairs_rMPdilySo8XNU84623TQt5lRnJY&d=D
> wIDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-
> 6zYMI&m=n-fEKQAt9NoRszV59Ac1D5d6611_ATKSOpDxztY1qUY&s=okK1nFCK-
> B46VCilSNKYrbfaPDcAy8IPPQJ6jGtGvfU&e=

---

From: Michael Richardson <[email-masked]>
Date: Thu, 27 Jun 2019 18:13:29 -0400

Adam Roach <[email-masked]> wrote:
    > structure" [1]) discussing the prospect of running additional meetings with a

Thank you for the great summary, I know how much work this takes!

    > Some of these included:

    > feedback that are inherent to the layout, the IESG would like feedback from
    > the

I would like to see more experiments of this kind run.
I sense that we aren't going to see an experiment like this at IETF105.
I agree that it would be easier to do this in small groups.

Can you define "very small groups"?  We do have WG that have only 10
people in the room (I recall nfsv4 was like that).  But, do those groups
actually have a problem?  Isn't the problem of facilitating better
participation an issue when we are at the ~75 people in the room with ~15
people queuing at the mic, and who knows how many who decided it wasn't worth
tripping over other people's knees/laptpos to get their?

We don't vote, we sometimes hum on issues.  The hum has significant social
advantages over showing hands.  I often wish I had a +1 / -1 button I could
push about what is being said that the mic.  Maybe also a "This is boring" button.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 28 Jun 2019 10:49:44 +1000

> shared, single experience with some number of the negative 

I have to second Al here.  This is a great setup for a small group, but my experience with the QUIC session was that it was a complete failure.  As someone privileged to have a seat at the table, I found the segregation really noticeable.  Depending on the subject, it was sometimes good, but more often it was acutely uncomfortable, and sometimes it was just completely unwieldy.  Talking to people who attempted to participate from the floor afterwards, their experience was much worse than my own.

To give a more concrete example: the improved flow of conversation at the table only really works if the rest of the room is spectating, but that was rare in QUIC.  Most conversations involved several people at a queue behind a floor mic.  Working out how to cede to the floor mic either resulted in starving the queue as the conversation at the table continued, or having the conversation at the table stilted by having to manage a queue.  The former was in some ways good (for those privileged few at the table).  The latter was strictly worse than just having a single queue, because tracking who was due to speak was extraordinarily difficult and made worse by having multiple active microphones.

I would say that while there is always some form segregation in these rooms (usually between participating and observing; also between remote and local), forcing that with a seating arrangement made things markedly worse.

This layout is great at interim meetings, where more time and smaller groups helps.  There, the segregation between local and remote is the only thing that remains uncomfortable.

---

From: Bob Hinden <[email-masked]>
Date: Thu, 27 Jun 2019 17:59:39 -0700

Adam,

Very nice summary.

> On Jun 27, 2019, at 2:30 PM, Adam Roach <[email-masked]> wrote:
> 
> As many of you will probably recall, we ran an experiment at IETF 98 in Chicago
> in which several working groups opted in to using such a layout. Those groups

…..

> feedback that are inherent to the layout, the IESG would like feedback from the

Reading this, seem many more negatives than positives.   The parts about making it hard for people not at the table or remote to speak, seems like showstoppers to me.

I have been to a lot of meetings like ICANN and IGF, where this is the normal format.   It may be great for people at the table, but felt to me like everyone was there to be in the audiance, and not participate.  That seems inappropriate for the IETF.

To answer your question, no don’t rerun the experiment, except for very small groups who want to do it.   However, I suspect for very small groups ( <40 ), I suspect the current meeting format isn’t a problem.   The room is small and people can sit at the front, easy to get to the mic, etc.

Bob

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 28 Jun 2019 12:55:32 +0100

Hiya,

> Thanks for compiling the history of feedback

Ditto. I also strongly agree with Al - only for
known v. small groups, e.g. with a history of
blue sheets with <N people on 'em. In that case
the IAB/IESG rooms could be used when free if
doing another experiment.

Cheers,
S.

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 28 Jun 2019 12:16:18 +0000

>>        especially when off-mic side-conversations broke out at the
   
As someone who has been involved in the EDU team and newcomer's events for a few years, this comment strongly resonates with me.

If there isn't room for "everyone" at the table, then you should not have a special table.

---

From: Mary Barnes <[email-masked]>
Date: Fri, 28 Jun 2019 08:17:37 -0500

On Fri, Jun 28, 2019 at 7:17 AM Salz, Rich <[email-masked]> wrote:

> As someone who has been involved in the EDU team and newcomer's events for
> a few years, this comment strongly resonates with me.
>
>  I agree.  This point was brought up during the early thread of discussion
but seems to have been ignored in this proposal.  I've been in other
meetings with this setup and it works very poorly.  I think once you hit a
threshold of 20-30 people, then it just doesn't work - you need the row
setup.

And, I also brought up how disruptive the cross talk is and the fact that
it's more of a problem for women.  I pointed to the research and it's been
my personal experience in lots of meetings.   So this experiment is flawed
and biased from the outset, since we have so few women participants.

Regards,
Mary.

---

From: "Black, David" <[email-masked]>
Date: Fri, 28 Jun 2019 13:25:55 +0000

> ... as an experiment for this meeting only.

+1 - as a co-chair of two WGs over the past year (one of the WGs recently closed), I would say that WG size definitely matters:

- TSVWG is a large WG with different communities of interest for different
   topics, as it serves as a catch-all WG for Transport Area work that doesn't
   fit into another WG or merit its own BOF/WG.  A U-shaped layout would
   not work well for TSVWG because it doesn't accommodate significant
   shifts in technical focus during the meeting where one wants different
   people at the tables for different topics.

- TCPINC was a small WG with only one topic in its latter stages (the TCPINC
  WG was recently closed) - a U-shaped layout would likely have worked
  better than the traditional room layout in order to maintain focus on
  that one topic area.

Thanks, --David

> From: WGChairs <[email-masked]> On Behalf Of MORTON,
> ALFRED C (AL)
> Sent: Thursday, June 27, 2019 6:03 PM
> To: Adam Roach; IETF WG Chairs
> Subject: RE: WG Meeting Structure / Alternate Room Layout
> 
> 
> [EXTERNAL EMAIL]
> 
> > 6zYMI&m=n-
> fEKQAt9NoRszV59Ac1D5d6611_ATKSOpDxztY1qUY&s=okK1nFCK-

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 28 Jun 2019 11:50:12 -0400

I also agree. Back almost 20 years ago I chaired the frnetmib wg, which had
about 8-10 regular attendees, all of whom were engaged in the work. We
would move our chairs into a circle and ignore the mikes, and just got
stuff done.  That wouldn't work as well today with all sessions being
broadcast and allowing remote participation, but a u-shaped table would
certainly work for a small group (with "small" being <= the number of
available chairs at the table), as long as speakers were well identified
for the remote participants - perhaps asking people to raise their hand and
be called on to speak by the chair would help. The chair(s) would also have
to make sure they keep an eye on the remote speaking queue.

Cheers,
Andy

---

From: Karen O'Donoghue <[email-masked]>
Date: Fri, 28 Jun 2019 16:19:17 +0000

Also from an EDU perspective… I strongly second Rich here… 

> On Jun 28, 2019, at 8:16 AM, Salz, Rich <[email-masked]> wrote:
>

---

From: Russ Housley <[email-masked]>
Date: Fri, 28 Jun 2019 12:37:00 -0400

>>       especially when off-mic side-conversations broke out at the

In LAMPS was that people not making presentation preferred the back row.  That contributed to the bad experience.

Russ

---

From: Michael Richardson <[email-masked]>
Date: Fri, 28 Jun 2019 15:12:04 -0400

Mary Barnes <[email-masked]> wrote:
    > I pointed to the research and it's been

What does research say is the best way to do things?
Pie in the sky, money no limit, what is the best way?
(And, what would be your preference?)

----

If we have to stick to rows of seated participants, can we consider:

a) having enough (more) space between chairs (columns) and between rows so that
   people can get in/out?

b) more wireless MICs. (I have ambivilent feelings about this. We need MIC jockeys)

---

From: Mary Barnes <[email-masked]>
Date: Fri, 28 Jun 2019 14:42:29 -0500

Comments below [MB].

> (And, what would be your preference?)
[MB]  The research I'm referring to, to which I had pointers in the last
discussion, indicates that cross-talk and background noise is a bigger
issue for women than men.  This would a particular problem for remote
participants.  I looked this up several years back after I tried to
participate in a meeting remotely and found it absolutely impossible due to
acoustics in the meeting room and the background chatter.  My point is that
we don't have the right set of subjects for this to be a valid test to
suggest this works for IETF participants, when the ones more likely to be
impacted makeup only 7% of the subjects.

The only time I've ever had experiences where the quality of participation
for remote participants in terms of audio in this sort of U shaped room
setup were ones where there are no back rows in a purpose built room, or
one where great effort was involved in installing the ceiling microphones
and great discipline was applied in ensuring that whomever was speaking had
the microphone nearby as you suggest below.  I don't think we have the
funds for that.  Audio problems in the meetings rooms are frequent even
with our fairly simple setup.     [/MB]

---

From: "Roni Even (A)" <[email-masked]>
Date: Sun, 30 Jun 2019 07:25:51 +0000

I agree with Al and likes Martin's summary. I was sitting in the QUIC session in the side rows  not facing eth monitors but the side of the table, the room was full, I could not follow the discussion on the table since there were multiple conversions at a time. I could not tell who is talking and there was no way to get to the mic.
I think this U layout works well in the QUIC Interim meeting when there are less participants and in that case even having two rows on each leg of the U works fine.

To have a successful U shape structure in a big room you may need a microphone for each of the participants and only one can work at a time and more spaces between rows in the rest of the room enabling better access to the mics for the non-privileged ones. 

Roni

-----Original Message-----
From: WGChairs [mailto:[email-masked]] On Behalf Of Martin Thomson
Sent: Friday, June 28, 2019 3:50 AM
To: [email-masked]
Subject: Re: WG Meeting Structure / Alternate Room Layout

On Fri, Jun 28, 2019, at 08:04, MORTON, ALFRED C (AL) wrote:
> I have hosted and attended meetings with 10 to 20 people, and the "U" 
> worked well, but my guess is that 40 breaks the shared, single 
> experience with some number of the negative issues mentioned.