Subject: Minutes from the WG TOOLS BOF in At IETF 53 Minneapolis (fwd)

From: Bernard Aboba <[email-masked]>
Date: Mon, 25 Mar 2002 18:26:56 -0800 (PST)

---------- Forwarded message ----------
Date: Mon, 25 Mar 2002 19:06:06 -0800
From: Paul Hoffman / IMC <[email-masked]>
To: [email-masked]
Subject: Minutes from the BOF in Minneapolis

The following is the minutes that I took minutes during the BOF in 
Minneapolis. Bernard Aboba also took minutes as he ran the meeting, 
and the text of his minutes are after mine. We organized our topics 
somewhat differently, but I think we hit the same points. I have also 
put these minutes on the BOF's web site.

--Paul Hoffman

IESG tools BOF
--------------

Agenda bashing

What is the problem for WG chairs?
	SIP WG had lots of open issues that needed to be tracked concretely.
	Don't rely on authors to include changes and not to loose them
	SIP WG tracking 200 drafts, watching deliverable dates, interdepencies
	Just keeping track of the drafts
	Designated design teams working on particular problems
	Specialists in different knowledge domains
	Track of documents after they are out of your hands
	Look at overlapping topics in drafts
	Lots of options for going to draft standards status
	Associate message from the list to section and document
	Prevent same issue from coming up
	Working with ITU, converting between Word and text
	Generating reports for other bodies (not known problem yet)
	Concensus-finding tools; sign-off tools
	More formalized terms and processes for WG last call
	Design groups as a subset of the WG
	Automated turning repeated topics into actual FAQs
	Versioning for intra-version drafts
	Collaborative editing tools (Word, CVS/XML2RFC)
	Keeping copies of old drafts so resolution can be tracked
What is the problem for editors?
	Resolutions that do not appear in documents
	Track changes to a collection of drafts
What is the problem for IESG?
	Have the WG tools work with the IESG tools

Requirements
	Controlling who can input to various tools
	Controlling who can see the output of all the tools
	Email interface as well as web
	Link to WG archive
	Longterm archiving for after a WG disbands

The ways this might change things
	This will change the way we do our work; how we refer to things
	Some material will appear only in the tools, not on the WG list

Bernard talked about Diameter's experience with doing this by hand
	Typical problems
	To get an issue discussed, include "[Issue]" in subject
	Got linked to a thread in the mailing list
	Issue statuses for open issues, kept on the web page forever
	Pluses and minuses; generally life was better
	Lots of ideas for an "Ideal" solution

Dean's tool for tracking Internet Drafts
	Charter items have tags that can be associated with drafts
	Publicly accessible web page that is automatically generated; 
manual data
	Lots of attributes
	Links to design teams
	Morgue holds old versions of drafts
	Each file has a .info file with fields
	Ask draft authors to fill in .info files

Robert's demo of bugzilla for SIP
	Nice graphical tool'
	Has email interface
	Can be used as a nag tool; escalation nagging
	Wrapper around a database
	Database is a set of products
	SIP using it for 3 drafts
	Products are drafts, components are sections of a document
	Tracked open issues
	Good to dissect databases instead of having a monolithic database
	User self-provisioning by user name
	Concept of groups for visibility
	Can integrate with CVS commiting
	No backups

Paul talked a bit about RT

Bernard showed the IEEE tool
	Balloting tool
	Commenting tool that would mail to a database

Henning talked about CVS
	Discovered that Word change tracking devolved quickly
	Use a text format in LATEX, easy to use diffs
	Every section was a different document
	Committing the changes caused mail to all editors
	Served as sanity check about the changes actually be made
	Can get a high-level view of why changes were being made

Bill Fenner showed bits about CVS
	CVSweb shows summaries
	Added javascript for querying
	O&M Area has a tool for tracking I-Ds given to them
	Showed preview of new IESG tool

[[ Notes that Bernard Aboba took as he chaired the meeting ]]

WG TOOLS Discussion
March 17, 2002
IETF 53
Minneapolis, MN

Agenda
	What is the problem?
		Brainstorming
	What are the requirements?
		For WG chairs?
		For Editors?
		For WG members?
		For the IESG?
	Thinking about workflow: an example
	Demonstrations
		Draft management CGI: Dean Willis
		Bugzilla: Robert Sparks
		RT
		IEEE Comment Database: Bernard Aboba
		Inter-Group Tool (alpha)
		W3C SSL tool
	Next steps
		Internet draft on WG tools?
		IMC tools server
What is the Problem?
	For WG chairs
		Lots of open issues to concretely track (600+)
		Capture resolutions of issues as well as the 
conversation (email thread)
		Tracking lots of drafts (WG vs. individual submissions)
		Keep up with deliverable dates (milestones, WG last call)
		Keeps track of drafts in last call, can lose a draft 
or comments on it
		Keeping track of design team work (who is on it, what 
they're doing)
		To track what topics are in each document; topic coverage
		Specialists in different knowledge domains
		Tracking stuff that fell off RFC Editor, IESG lists 
mysteriously
		Keeping track of draft standard option implementation
		Formalize WG last call process (not only nos but yeses)
		Record issues to refer back and say it was resolved; 
integration with email and archives
		Signoff process (not just people who are posting or speaking)
		To track design group efforts
		Automated capturing of issues that come up frequently 
into a FAQ
What is the Problem (cont'd)
	For Editors
		Conversion between differing styles
		Collaboration between standards bodies
		Issue X comes up again, want to go back to the discussion
		Track changes to sections of draft since a date (not diff)
	For WG members
		Keeping track of draft standard option implementation
		Link a resolution to the document section that it settled
		Searchable email lists without having to download the 
whole thing
		Track changes to sections of draft since a date (not diff)
	For IESG
		Want full data for IESG or WG chairs, summarized data for WG
		Some comments that WG chairs can see, some that only 
IESG can see
		IESG tools that work together with WG tools to track updates
		Work item tracking integrated w/IESG tool

What Are the Requirements?
	For WG chairs
		Ability to determine who gets to input
		Want to be able to delegate to authors
		May want to restrict access to output, too
		Calendar generation
		Automated reporting for large efforts
	For Editors
		Versioning?
		Collaborative editing tools? (XML to RFC?)
	For WG members
		Email interface as well as Web
		Link to the WG archive
	For IESG
		Integration between IESG and WG tools
		Long term archiving of the results (in standard format? Text?)
	Design groups won't generate email on mailing list so tool 
may have unique discussions
One WG's Problem
	AAA WG history
		Mailing list discussion was diffuse
	Lots of discussion, but...
	Hard to tell if commentors were asking for a change to the 
documents or just asking questions about them
		When changes were requested, hard to track status
	Were changes made or was the request rejected?
	Was the original requestor satisfied with the changes? Or was 
the issue still open?
	Was WG consensus reflected in the documents?
		Document status was hard to gauge
	3G folks ask: "When will you be done?"
	Question was difficult to answer!
		How many issues were outstanding?
		How fast were issues being resolved?
		How fast were issues being raised?

A "Solution"
	A formal Document Change Request (DCR) process
	To get an issue considered, WG member MUST
		Post a message to the AAA WG mailing list with 
[Issue] in the Subject
		Message body includes a description of the issue, the 
affected document (s) and the proposed change
	Document number assigned to the issue
	WG discusses the issue
	Document editor proposes a resolution
	WG chair verifies WG consensus
	Issue status tracked by document on a Web page
		http://www.drizzle.com/~aboba/AAA/issues.html

DCR Schema
	DCR submission
		Description of issue
		Submitter name: Your_Name_Here
		Submitter email address: Your_Email_Address_Here
		Date first submitted: Insert_Date_Here
		Reference: URL to e-mail describing problem, if available
		Document: Document Requiring change [base, nasreq, 
mobileip, cms]
		Comment type: ['T'echnical | 'E'ditorial]
		Priority: ['S' Must fix | '1' Should fix | '2' May fix ]
		Section: Insert_Section_Number_Here
		Rationale/Explanation of issue:
		Description of problem
		Requested change:
		Proposal
	Issue status
		Text Proposed - Text has been proposed on the list, 
and no consensus has been reached
		Accepted - Text has been proposed, consensus has been 
reached, but the fix hasn't been included in a specification yet.
		Pending - Discussion is on-going, no proposed text 
has been proposed
		No Discussion - No discussion has been initiated on this issue.
		Clarifying Text Required - The issue can be closed if 
additional clarifying text is added to the draft
		Possible Reject - Issue will be rejected unless 
objections occur.
Better, but...
	Issues now tracked formally
		Now possible to distinguish between change requests 
and ordinary discussion
		Editors, Chairs can focus on email with [issue] in the subject
	But...
		Lots of manual labor required
	Issue #s manually assigned
	Issue descriptions cut and pasted into Issues list
	Issues web page manually maintained
		Not as integrated as it could be
	Issues web page not linked to WG discussion
	Can't click on an issue and see the discussion that ensued
	Not easy to verify that WG consensus was reflected in the 
issue resolution
		Errors possible
	Issues sometimes fall through the cracks
An "Ideal" Solution?
	WG member sends mail with [Issue] in the subject
	An automated attendant assigns an issue number, replies to 
the WG mailing list
	Attendant parses the body of the message, assigns the issue 
to a queue, depending on the draft
		If the issue relates to multiple drafts, it goes in 
multiple queues
	Editor receives mail that an issue has been assigned, with a 
URL pointing to a web page with all the issues assigned to him/her
	Editor can get status on all issues on the web page
		Issue list for each draft
		Open versus Resolved issues
		Details of each issue
	Status of each request can be reviewed
		Accept, Reject, Pending
		If Rejected, reason for the rejection
	Editors can resolve/assign issues
		Can assign the issue to someone to come up with a solution
		Can resolve issues
	Original poster verifies the resolution
		Editor can resolve the issue, but poster or Chair 
needs to "close" it
	Reports can be generated
		Incoming issues
		Resolved issues
		Open issues
		Rejected issues
	The whole process is secure
		An attacker can't delete or modify issues
		Editors can't touch issues outside their drafts

Draft CGI Demonstration
Dean Willis
Bugzilla Demonstration
Robert Sparks
RT Demonstration
IEEE Tools
	IEEE has developed their own tools over several years
	Solution derives from their definition of the problem, which 
is not the same as the IETF problem
	IEEE process
		Largely a batch process: comments are received in 
response to Ballot solicitations
		Task Groups meet in person to review and resolve comments
		Comments are reflected in the document, which goes to 
ballot again
		IETF process is much more interactive than this
		Still, productivity is impressive
	Solution
		Comment process is automated
		Members MUST submit comments for each ballot or they 
lose their votes
		Comments MUST be submitted in a rigidly defined format
	Ballots typically submitted in a spreadsheet
	To be valid, comment must specify the document and section, 
exactly what text is to be changed, and what it is to be replaced with
	Vague comments like "make it better" are classified as invalid
		Comments are automatically collated into a big 
spreadsheet or a database
		WG comes to consensus on how to deal with the changes
		Tools make it possible to resolve hundreds of 
comments within a few days
		Can track action items assigned to WG members, send 
prod-o-grams

Feedback?