Subject: Re: Fw: Interim meetings - changing the way we work

From: Alia Atlas <[email-masked]>
Date: Tue, 17 Feb 2015 09:37:38 -0500

[Moved from rtgwg to routing-discussion]

Hi Tom,

Thanks for pointing these out.  BIER was not an official IETF interim - it
was a follow-on meeting from a BoF
and not yet chartered as a WG nor a virtual BoF.  Nonetheless, minutes were
posted to the mailing list.

NVO3 has its minutes going out to the mailing list after - as well as
agenda and slides being posted to the
list beforehand.  I agree that they also should be uploaded to the Meeting
Materials tool and am cc'ing the NVO3
chairs and secretary so that they can check up on that.

Regards,
Alia

On Tue, Feb 17, 2015 at 7:32 AM, t.petch <[email-masked]> wrote:

> Adrian, Alia
>
> Looking for the outcome of an Interim meeting on the IETF website, I
> became aware of how rarely the proceedings are fully reported (as I
> posted to the main IETF list recently).
>
> Of the meetings in 2014 that produced no Minutes, three are in the
> Routing
> Area,
> 2014-12-18 teas
> 2014-12-15 bier
> 2014-10-21 nvo3
>
> These are not lists I normally follow but I did look at the Mailing List
> Archives and, for all three, there is mention of 'Rough' or 'Draft'
> minutes, in one case even adding
>
> 'I will post them to the meeting materials, by end of this week'
>
> Thoughts?
>
> Tom Petch
>
> ----- Original Message -----
> From: "t.p." <[email-masked]>
> To: "ietf" <[email-masked]>
> Sent: Sunday, February 15, 2015 1:57 PM
> Subject: Interim meetings - changing the way we work
>
>
> > There has been a marked increase in the number of  interim meetings.
> > Using
> > http://www.ietf.org/meeting/interim/proceedings.html
> > as a guide, there were
> > 18 in 2011
> > 35 in 2012
> > 45 in 2013
> > 84 in 2014
> > 13 in January 2015 alone.
> >
> > With them comes a change in the way of working, perhaps rendering some
> > of our practices historic.
> >
> > Of the 84 meetings listed for 2014, 21 left no other trace on the IETF
> > web site, no Agenda, no Minutes, no Proceedings.  Perhaps the WG
> > provided no materials, perhaps they did not happen; sometimes a
> > cancellation notice is apparent in the WG List Archives, other times
> > not.
> >
> > Of the 63 that have left a trace, 6 produced no Minutes but did
> produce
> > slides or recordings and so presumably happened.
> >
> > Of the 57 that produced Minutes, 18 produced no Agenda while in 13
> > cases, the Minutes contained no list of Attendees (goodbye Blue
> > Sheets?).
> >
> > Only 26 meetings left a complete record, of Agenda, Minutes and
> > Attendees.
> >
> > The meetings encompassed 30 Working Groups, of which 16 met once, 14
> > more than once, with one WG meeting 8 times.
> >
> > What is more subjective is that, with Virtual Interims, increasingly
> the
> > only kind, there is a tendency for the WG Mailing List to no longer
> > provide a record of discussions, choices, consensus.  For example,
> they
> > may make greater use of github so that the minutes record a discussion
> > of options 1, 2 and 3 for Issue 29 with no indication of what the
> issue
> > or options are; a while later, they may record an update to option 3
> so
> > it would now seem impossible to know what was discussed at the earlier
> > meeting.
> >
> > Even with the better minutes, they never give the same sense as posts
> to
> > a mailing list of who was or was not in favour and how strong their
> view
> > was.
> >
> > Of course, we still have WG Last Calls on the list but if at a future
> > date, an AD or GenArt reviewer wants to look back and see what options
> > were discussed and  how rough the consensus was, well, it may be
> > impossible.
> >
> > A post in another thread recently said
> >
> > > I do think that the increased significance of meetings
> > > in IETF participation (and here, I'm not talking about
> > > things like nomcom but about significance to our technical
> > > work) is a problem, both because it tends to marginalize
> > > people who can't come to meetings and because it slows
> > > work down.
> >
> > Well, I disagree about slowing the work down but certainly agree with
> > the marginalisation, that WGs holding multiple Interims may tend to
> > develop an in-crowd of those that can participate with the world at
> > large only seeing the end result without knowing how it was arrived at
> > by whom.
> >
> > Tom Petch
> >
>
>

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 17 Feb 2015 07:14:25 -0800

Hi Tom and Alia,

In regards to NVo3, I’ve been posting minutes and agenda, promptly.
However, I see minutes for 10/21/14 and agenda for 10/02/14 were missing.
I did send out the draft minutes for 10/21/14 to the mailing list (<http://www.ietf.org/mail-archive/web/nvo3/current/msg04084.html <http://www.ietf.org/mail-archive/web/nvo3/current/msg04084.html>>)
Agenda for 10/02/14 was sent out to mailing list as well (<http://www.ietf.org/mail-archive/web/nvo3/current/msg04000.html <http://www.ietf.org/mail-archive/web/nvo3/current/msg04000.html>)

Do not remember why they were not uploaded. Oversight on my part.
Will work with tools folks to have them uploaded.

cheers
-sam
> On Feb 17, 2015, at 6:37 AM, Alia Atlas <[email-masked]> wrote:
> 
> Thanks for pointing these out.  BIER was not an official IETF interim - it was a follow-on meeting from a BoF
> and not yet chartered as a WG nor a virtual BoF.  Nonetheless, minutes were posted to the mailing list.
> 
> NVO3 has its minutes going out to the mailing list after - as well as agenda and slides being posted to the
> list beforehand.  I agree that they also should be uploaded to the Meeting Materials tool and am cc'ing the NVO3
> On Tue, Feb 17, 2015 at 7:32 AM, t.petch <[email-masked] <mailto:[email-masked]>> wrote:
> Adrian, Alia
> 
> From: "t.p." <[email-masked] <mailto:[email-masked]>>
> To: "ietf" <[email-masked] <mailto:[email-masked]>>
> > http://www.ietf.org/meeting/interim/proceedings.html <http://www.ietf.org/meeting/interim/proceedings.html>
> > as a guide, there were
> > 18 in 2011
> > 35 in 2012
> > 45 in 2013
> > 84 in 2014

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 17 Feb 2015 07:26:06 -0800

They are now uploaded.

-sam
> On Feb 17, 2015, at 7:14 AM, Sam Aldrin <[email-masked]> wrote:
> 
> Hi Tom and Alia,
> 
>> On Feb 17, 2015, at 6:37 AM, Alia Atlas <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: t.petch <[email-masked]>
Date: Tue, 17 Feb 2015 18:19:46 +0000

Sam

Thanks for the uploads.   While having materials on the WG list
serves the immediate need well, consider if say an AD or Gen-ART
reviewer
(not something I will ever be:-) wants to look back a year or two later
to see how a decision was arrived at, then having them on the
proceedings page makes them easier to find (or even for me six months
later).

I commented earlier that only about a quarter
of interims in 2014 met the requirements as I understand them i.e.

"interim meetings are subject to the same rules for advance
notification, reporting, open participation, and process, which apply to
other working group meetings."

I wonder if this is sustainable given the explosive growth in Interim
Meetings, especially when a single WG holds many such between the IETF
Meetings.  Which brings me to

2014-10-02 nvo3 minutes yes (pdf Yuk), but I cannot see a list of
attendees
2014-09-11 nvo3 minutes but I cannot see a list of attendees

As I say, about three quarters of the Interim Meetings scheduled for
2014 would appear not to meet the requirements for them (so I am not
picking on nvo3:-)

Tom Petch

----- Original Message -----
From: "Sam Aldrin" <[email-masked]>
To: "Alia Atlas" <[email-masked]>
Cc: "t.petch" <[email-masked]>; "Adrian Farrel"
<[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Tuesday, February 17, 2015 3:26 PM

They are now uploaded.

-sam
> I did send out the draft minutes for 10/21/14 to the mailing list
(<http://www.ietf.org/mail-archive/web/nvo3/current/msg04084.html
<http://www.ietf.org/mail-archive/web/nvo3/current/msg04084.html>>)
> Agenda for 10/02/14 was sent out to mailing list as well
(<http://www.ietf.org/mail-archive/web/nvo3/current/msg04000.html
<http://www.ietf.org/mail-archive/web/nvo3/current/msg04000.html>)
>> Thanks for pointing these out.  BIER was not an official IETF
interim - it was a follow-on meeting from a BoF
>> list beforehand.  I agree that they also should be uploaded to the
Meeting Materials tool and am cc'ing the NVO3
>> > http://www.ietf.org/meeting/interim/proceedings.html
<http://www.ietf.org/meeting/interim/proceedings.html>

---

From: Benson Schliesser <[email-masked]>
Date: Tue, 17 Feb 2015 14:02:32 -0500

Hi, Tom -

> proceedings page makes them easier to find (or even for me six months

We agree. As Sam indicated, the lack of proceedings of our Oct-2014 
meeting was an oversight. I appreciate that Sam fixed the problem 
quickly - thanks, Sam.

> other working group meetings."

I don't have any insight on overall statistics, whether the 25% number 
is accurate etc. But based on my experience chairing NVO3 it seems like 
a fairly easy expectation to meet. Anything less than 100% should be 
considered a problem.

We announce the meeting in advance, post the access logistics (e.g. 
WebEx) to the mailing list, publish an agenda, upload slides, take 
minutes and blue sheets during the meeting, and upload them to the 
meeting materials manager tool. These are literally the same steps that 
we follow for a regular meeting, except that the virtual logistics (e.g. 
WebEx) may be different.

> 2014-09-11 nvo3 minutes but I cannot see a list of attendees

Attendee lists / blue sheets are not published in meeting minutes for 
regular meetings, and I don't see why interim meetings should be any 
different. Collecting the "blue sheets" from a virtual interim is a bit 
of a tricky problem, but regardless of how the names are collected I 
think the correct way to memorialize them is to give them to the 
secretariat - not publish them in the minutes.

I recall that until recently the blue sheets had to be emailed to the 
secretariat. But at some point recently the meeting materials manager 
tool was upgraded to include a "bluesheets" material type.

Cheers,
-Benson

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 17 Feb 2015 11:09:50 -0800

Tom

inline for my comments.
> On Feb 17, 2015, at 10:19 AM, t.petch <[email-masked]> wrote:
> 
> Sam
> 

In regards to attendees list, neither does the F2F meeting minutes capture that. The list of attendees are documented as part of Blue sheet.
Initially, the virtual meeting minutes did capture attendees list, but after discussing with Chairs, they were deliberately not added.
Two reasons
- Difficult to capture list of attendees on a virtual meeting (lot of folks are being missed)
- The list of attendees were being captured on virtual blue sheet, which we employed in the recent interims. (http://wiki.tools.ietf.org/wg/nvo3/trac/wiki/WikiStart <http://wiki.tools.ietf.org/wg/nvo3/trac/wiki/WikiStart>)

If pdf is yuk, then you might indicate to tools folks not to allow pdf’s. FWIW, had some difficulty in generating clean formatted plain text.

> 2014-09-11 nvo3 minutes but I cannot see a list of attendees

Same reason as above.
> 2014 would appear not to meet the requirements for them (so I am not

As we are talking about process, here is my gripe about it. (Not directed at you)
There are little or no volunteers to take notes/jabber. 
Whoever is taking is doing a great favor to the IETF community. 
Please be appreciative of the effort and not to wrap this effort in process quagmire.
If the process is that important, do change the rules to ensure WG meeting is held only if so and so following conditions are met. I haven’t seen one for minutes, yet.
</rant>

-sam

---

From: t.petch <[email-masked]>
Date: Wed, 18 Feb 2015 11:24:15 +0000

----- Original Message -----
From: "Benson Schliesser" <[email-masked]>
To: "t.petch" <[email-masked]>
Sent: Tuesday, February 17, 2015 7:02 PM
> a fairly easy expectation to meet. Anything less than 100% should be

The 25% figure is mine based on the 84 Interim Meetings listed on the
IETF website for 2014 (there are a few RGs in amongst the WGs which I
included,  more by accident than design).

I followed every link - WG, Agenda, Minutes, Proceedings - from the web
page to see what was there, Agenda, Minutes, Attendee list or some other
evidence of the meeting having happened (slides, recordings).

I regarded 26 as complete (Agenda, Minutes, Attendee list) while 21
showed no sign of having happened; perhaps they did not (but I would
value some sign of that on the website).

The rest lacked something, usually an Agenda (without which, I
understand some ADs have cancelled meetings!).

So that is where my figures come from, pursuing as many of the 336 links
as existed and to the extent that I can count, are accurate!

I have commented on Attendee lists separately.

Tom Petch

---

From: t.petch <[email-masked]>
Date: Wed, 18 Feb 2015 11:40:46 +0000

Inline <tp>

----- Original Message -----
From: "Sam Aldrin" <[email-masked]>
To: "t.petch" <[email-masked]>
Sent: Tuesday, February 17, 2015 7:09 PM

Tom

inline for my comments.
> 2014-10-02 nvo3 minutes yes (pdf Yuk), but I cannot see a list of

In regards to attendees list, neither does the F2F meeting minutes
capture that. The list of attendees are documented as part of Blue
sheet.
Initially, the virtual meeting minutes did capture attendees list, but
after discussing with Chairs, they were deliberately not added.
Two reasons
- Difficult to capture list of attendees on a virtual meeting (lot of
folks are being missed)
- The list of attendees were being captured on virtual blue sheet, which
we employed in the recent interims.
(http://wiki.tools.ietf.org/wg/nvo3/trac/wiki/WikiStart
<http://wiki.tools.ietf.org/wg/nvo3/trac/wiki/WikiStart>)

<tp>

The topic of attendee lists has surfaced (as Blue Sheets) on the main
IETF list, for example in April 2012.  The impression I get is that
there is obligation on the IETF to keep records if only against possible
future legal actions.

The 'Note Well' says

"A participant in any IETF activity acknowledges that written, audio and
video records of meetings may be made and may be available to the
public."

so a participant should not have an expectation of privacy.

I don't see a need for them to be in the minutes but note that that is
what most WG do; of the 14 WGs that held more than one Interim Meeting
in 2014, lmap was the only other one never to have provided an attendee
list.

This probably needs clarifying for the IETF at large.

Tom Petch

</tp>

If pdf is yuk, then you might indicate to tools folks not to allow pdf’
s. FWIW, had some difficulty in generating clean formatted plain text.

Same reason as above.
> 2014 would appear not to meet the requirements for them (so I am not

As we are talking about process, here is my gripe about it. (Not
directed at you)
There are little or no volunteers to take notes/jabber.
Whoever is taking is doing a great favor to the IETF community.
Please be appreciative of the effort and not to wrap this effort in
process quagmire.
If the process is that important, do change the rules to ensure WG
meeting is held only if so and so following conditions are met. I haven’
t seen one for minutes, yet.
</rant>

-sam

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 18 Feb 2015 11:27:53 -0800

Tom,

Inline for my comments.

> On Feb 18, 2015, at 3:40 AM, t.petch <[email-masked]> wrote:
> 
> Inline <tp>
> 
> From: "Sam Aldrin" <[email-masked] <mailto:[email-masked]>>
> To: "t.petch" <[email-masked] <mailto:[email-masked]>>
> <http://wiki.tools.ietf.org/wg/nvo3/trac/wiki/WikiStart <http://wiki.tools.ietf.org/wg/nvo3/trac/wiki/WikiStart>>)
> 
> <tp>
> 
%sam - I am by no means a lawyer or have position about privacy related topics, but not sure if meeting minutes is the right place to document attendee list.
As blue sheets are used for that and not part of minutes of F2F mtg, may be some guideline and clarification for IETF at large is needed, as you mentioned.
I like the idea of Etherpad for attendees to sign in and also the tools now support upload of blue sheet.

For NVo3, I’ll try to as gather as much data as I have, for the past interims, and upload them to proceedings page.
May be it is good idea to add a column for ‘Bluesheets’? <http://www.ietf.org/meeting/interim/proceedings.html <http://www.ietf.org/meeting/interim/proceedings.html>>

-sam