Subject: [ietf-sow] Fwd: Agenda Development RFP SOW for Community Input

From: Robert Sparks <[email-masked]>
Date: Tue, 8 Nov 2011 15:09:11 -0600

I'm glad the IOAC is moving work on building a better scheduling system
forward. This is an improvement the community has been asking after for
a long time, and I believe it will help us create more effective meetings
with less effort. I have a long list of comments on the current sow document,
and believe the document needs significant revision before being issued as 
an RFP. Please accept these as suggestions for improvement, not as resistance
to the work.

High level points:

.) There are imbedded questions that aren't something the responder should
provide an answer to. (I'll comment on what I think the answers should be
below).

.) The document proposes changes to existing policy and practice.
(Such as not allowing a group to request a 3rd session until _after_ the
initial schedule has been prepared, and limiting 1st tier conflicts to 10 
groups, and requiring justificaiton for a second slot request.). Why? 

.) The document frequently makes UI decisions that it really should leave 
to the proposer. The use of "dropdown" to mean the user should select from
options is a particularly bad offender - the worst being when talking about
selecting groups for the conflict lists.

.) The document should make it clear early that the tool is expected to
automatically generate a proposed schedule for the meeting. Right now,
that doesn't really appear until section 5 (page 17). though it leaks 
forward a little into section 4.3.

.) The document is inconsistent on whether the automatically generated
schedule is created exactly once, or if it should be easy to say "no,
we don't like that at all", go back and tweak the conflict lists or apply
the "ignore" policies the document discusses and cause it to regenerate
a new proposal from scratch.

.) There is one major usecase for the tool that the document doesn't
really seem to address. Currently there is a major multiparty session
for working through conflicts, and we struggle to see the effects of
the changes being made in realtime. There should be some requirements about
allowing multiple remote users to review proposed changes in realtime.

.) There is insufficient detail around authorization for the different
access levels. The document needs to be much more specific about what
"own session" or "own session request" means. The document also assumes
a type of user that has no account - why? Is the tool going to reuse
datatracker accounts? (It should). Those are not hard to obtain, and
asking users to use them will improve transparency.

.) The document should call out early that the secretariat can take
any action on behalf of an AD or WG chair, and an AD can take any
action on behalf of a WG chair (this part would be new, but the tool
should allow the policy). 

.) It should be clear whether this tool or some other tool will be 
used to edit and adjust the schedule as the meeting progresses. If that's
part of what the RFP is asking the proposer to suggest, the question should
be explicit.

.) There are other tools (such as the agenda views in the tracker) that
depend on the schedule being populated in the database. The appendix
suggests that an existing application might extract information into the 
formats it needs. It should aso state that such an application needs to
put data back _into_ the database once it's done.

Details (in document order):

The note at the end of section 1.2 should be restated as the actual
requirement earlier in the section. What you're really requiring is
that the tool allow a list of room types (also called space) to be 
managed, and a separate list of room configurations to be managed. 
You have initial values for those lists that the tool should be configured
with. It should also be clear whether new names you add to either list
are only for the current meeting (that is the lists are meeting specific),
or persist between meetings.

Why do you need both the ability to specify available time ranges in 1.2
and blackout times in 1.3? If these are both being populate only with data
from the contract (as the sections indicate), this should be collapsed into
1.2 and the focus should be on capturing the available times. If the intent
of 1.3 is to mark space unavailable for some reason other than a meeting (and
if that can happen after the meeting template is created), the requirement
needs to be clarified.

Section 1.4 speaks of room and session names. It would help to clarify
the terms (consistent use and definition).  Does session name mean
something like "softwire" or something like "Afternoon Session I"?  
Similarly, does room name mean something like "3F Banquet" or 
something like "Breakout"?

There are multiple "types of space" lists that have different values
in them (one in 1.2 and one in 1.4). There should be one list.

Section 2 appears to be defining access levels for editing
session _requests_, but later parts of the document seem to be
reusing the levels to describe authorization for affecting the actual
schedule.

Are there any requirements for leaving a record of editing or deleting
a session request? Will there be a history log, and will the tool
expose it?

It might help to position section 2 after section 3 (so that "session
request" is defined).

The document asks {{{ Do sessions have different categories....
The answer to that affects the content of a session request, and depends
on whether we want to use session requests for leadership meetings or always
have the secretariat schedule those directly. Other parts of the document
assume the later, so I think the answer to the question (at least for this
iteration) is no, but that might change if we want to use the tool to schedule
the IAB/IESG breakout rooms or allow the nomcom to use it to schedule their
room.

Section 3.1 states that a Level 3 user can make an area meeting request.
I don't think that's what's intended.

As noted in the major observations above, the notion of the tool refusing
to allow the request of a 3rd session until after the first draft of the
agenda has been created seems detrimental. If such a session is known
to be needed (approved by an AD), not accounting for that in the initial
draft just adds pain when we have to throw large parts of the inital
draft away. The requirement instead should be to make it possible for ADs
to approve or reject exceptional requests between when they are submitted
and when the initial draft is created.

Again as noted in the major observations section, the tool should allow
the secretariat to take any action on behalf of an AD. Given that, the
requirement currently stated as "The Secretariat can also marked the
session request as approved in the tool, since on occasion approval is
given verbally or via email" is not needed, and could be dangerous. It 
could result in some special knob the secretariat could turn. Instead, 
the secretariat should just do whatever the AD would have done.

The section that talks about specifiying conflict lists requires
dropdown UI motifs, but shows examples of lists of checkboxes. All of these
should be carefully stated as either examples or suggestions. There are 
many other possible UI designs, and the RFP should allow the proposer to
propose one. It would be good to be explicit on whether the tool will allow
a requester to put a given group in both the 1st and 2nd tier conflict lists.

It seems odd that the document requires a requester to justify a 2nd slot
request (it shouldn't) but not justify a "Please do not schedule on" request
(which it should).

Under special requests, the document notes "There will be a statement
here noting that cost may apply for some items." Please provide additional
detail. Do you want this to appear when working group chairs are making
a normal working group session request, or was the intent for this to only
apply to breakout or side-meetings (such as corporate or affiliate meetings)?

There is design in the document around allowing requests to be made without
an account. As noted in the major comments section, this doesn't seem 
necessary - datatracker accounts are not hard to create, using them would
increase transparency and would avoid some of the edge conditions the document
current concerns itself with. The notion of including links that would return
to a request should be retained though - that will be useful even if you have
logged in.

The document asks {{ Is there any reason to involve this tool in side meeting
room reservations for the IESG, IAB, or IAOC }}. The answer depends on whether
this tool directly provides view of the agenda (that is, it maintains live data
during the meetings), or whether it is put aside once the printed agenda is
created. I think it would be more valuable to create a tool that would allow
managing the meeting in real time (dealing with room substitutions on the fly).

Section 3.5 says the tool will act automatically. It should explain what
automatic actions the tool will take.

In section 5, why should the tool wait until the secretariat "clicks a button"
to generate a proposed schedule. Why not have it build proposals on the fly as
each new request is received?

Under the agenda manipulation sections - why don't we allow users (of whatever
level) to save proposed agendas and refer to them by name (preferably with a URL 
that goes back to them). This would allow several people to propose conflict
resolutions. It would be even better if the tool provided a way to easily 
visualize the difference between any two saved agendas. This saving motif
could be leveraged to simplify some of the requirements around identifying
what the current draft or "final" agenda are.

The document, in section 5.2, asks whether manipulation should be limited
to ADs. I think the answer should be no - let anyone logged in play with the 
agenda to their hearts content and save it. Just don't make it where the ADs or
secretariat have to look at each saved thing (or fight through lists of saved
things to see what's officially on the table). 

There is detail missing from section 5.3 - are we requiring the tool to
know when the draft schedule has been published? Will it start behaving
differently? Will it behave differently after the "final" schedule has
been published?

When discussing creating ics files, the document calls out local timezones 
and conversions. Please point to the "floating" ical capability.

The document currently mixes discussing reporting on conflicts and
performing conflict resolution. These should be carefully kept separate.
The requirement to explore "if X moves to Y, show conflicts" should be moved
into the section discussing manipulating the schedule, and the requirement
should be to _always_ show conflicts on any given proposed schedule.

The document asks {{{ What should the conflict report look like...}}}
Adam Roach has a prototype that shows a graphical conflict report. Please
consider including a screenshot from it as an example.

Section 7.9 is too vague to implement. Since this is an RFP, could this
be cast as a "what other custom reporting capabilities can you support?"

Section 8.2 calls out [WG Acronym] but not all session requets are for
working group meetings.

Section 8.3 mentions the "draft agenda phase". Where are the phases
described?

Seciton 9 concerns itself with colored markup of the final agenda in word
and excel. It should instead focus on clear markup of the actual final agenda
on the website.

Please consider moving definitions early in the document.

Begin forwarded message:

> From: IETF Administrative Director <[email-masked]>
> Date: November 7, 2011 1:48:44 PM CST
> To: IETF Announcement list <[email-masked]>
> Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
> Subject: Agenda Development RFP SOW for Community Input 
> 
> REMINDER
> 
> The IETF Administrative Oversight Committee (IAOC) plans to issue a Request for Proposal (RFP) in November for a
> vendor to develop the IETF Meeting Agenda Scheduling Tool.  To that end the IAOC invites community comments on the 
> draft Statement of Work (SOW) that is located at <http://iaoc.ietf.org/rfpsrfis.html>.
> 
> The community comment period ends 14 November 2011. The IAOC will consider all comments it receives during that period.
> 
> The SOW describes the requirements for a Secretariat Meeting Agenda creation tool. The agenda tool will handle all meeting 
> scheduling, as well as management of any/all space in use during an IETF meetings. This includes working sessions, leadership 
> meetings, tutorials, BoFs, office hours, registration, breaks, and more.  Agenda scheduling is currently a manual process.  
> The Draft SOW includes some questions for which feedback and suggestions are desired.
> 
> Community comment is due no later than 14 November to [email-masked]. One may subscribe to that list here: 
> https://www.ietf.org/mailman/listinfo/ietf-sow. 
> 
> All information submitted in response to this announcement is voluntary and may be used in the development of the SOW 
> and a subsequent RFP.
> 
> Thanks for your continuing advice and support.
> 
> Ray Pelletier 
> IETF Administrative Director 
> Internet Society
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-announce

---

From: Russ Housley <[email-masked]>
Date: Fri, 11 Nov 2011 03:35:55 -0500

> provide an answer to. (I'll comment on what I think the answers should be

We asked if the community would like to suggest answers.  Thanks for doing so.

> groups, and requiring justificaiton for a second slot request.). Why? 

This is actually current policy, I believe, even if some ADs are approving a third slot for very busy WGs.

> selecting groups for the conflict lists.

I agree that we should allow some flexibility, but these came from the people that will be using the tool.

> forward a little into section 4.3.

Good suggestion.

> a new proposal from scratch.

This depends on the level of integration with the database, right?  However, I agree that it should be possible to rerun the schedule generation and overwrite a previous run.

> allowing multiple remote users to review proposed changes in realtime.

Yes, it should be added.

> asking users to use them will improve transparency.

When I reviewed it, I assumed that to be the case.  It should be explicit.

> should allow the policy).

Agree.

> part of what the RFP is asking the proposer to suggest, the question should

Agree.

> put data back _into_ the database once it's done.

Correct.  I wrote that part, and I missed that step.

> or persist between meetings.

Agree.

> if that can happen after the meeting template is created), the requirement

I think either way works.  This seems to be the way the users think about it.

> something like "Breakout"?

Yes, this does need clarification.  I think it means "Afternoon Session I".

> in them (one in 1.2 and one in 1.4). There should be one list.

Agree.

> reusing the levels to describe authorization for affecting the actual

We should add one section on login and authorization.

> a session request? Will there be a history log, and will the tool

I do not know if this is needed.  Would the Secretariat find this useful?

> It might help to position section 2 after section 3 (so that "session

Good idea.

> the IAB/IESG breakout rooms or allow the nomcom to use it to schedule their

We do not want this tool to schedule time in breakout rooms.  Maybe in the future, but I do not want to bite off that much right now.

> I don't think that's what's intended.

ADs or Secretariat should be able to do this action.

> and when the initial draft is created.

I addressed this above.

> the secretariat should just do whatever the AD would have done.

Agree.

> a requester to put a given group in both the 1st and 2nd tier conflict lists.

I addressed this above.

> request (it shouldn't) but not justify a "Please do not schedule on" request

I do not understand this comment.  Robert, please try again.

> apply to breakout or side-meetings (such as corporate or affiliate meetings)?

I think this is about side-meetings, not breakout rooms.

> to a request should be retained though - that will be useful even if you have

Agree.  Especially with the community enhancements that are underway.  Most active IETF participants will have datatracker accounts.

> managing the meeting in real time (dealing with room substitutions on the fly).

I think the Secretariat should set the requirements, and then the IAOC should decide whether it can implement this aspect with the current budget.  It might have to wait if it is too expensive.

> automatic actions the tool will take.

Yes, they need to be listed.

> each new request is received?

This depends on the approach taken -- two extremes are in the Annex.

> what the current draft or "final" agenda are.

I'm not sure.  This could lead to a very significant amount of storage.  Also, it will not be useful after a few days.

> things to see what's officially on the table). 

See above.

> differently? Will it behave differently after the "final" schedule has

I think so.  I think changes will be batched for posting to the community.

> and conversions. Please point to the "floating" ical capability.

Good suggestion.

> should be to _always_ show conflicts on any given proposed schedule.

I like this idea.

> consider including a screenshot from it as an example.

I like this idea too, but we need to allow flexibility as with other aspects of the UI.

> be cast as a "what other custom reporting capabilities can you support?"

I think we can add a list of examples, but I would like this to be as flexible as possible within reasonable cost.

> Section 8.2 calls out [WG Acronym] but not all session requests are for
> working group meetings.

Right.  This needs to be fixed.

> Section 8.3 mentions the "draft agenda phase". Where are the phases

Yes, the timeline should be added to the introduction.

> and excel. It should instead focus on clear markup of the actual final agenda

We need to make sure the needs of the Secretariat are met.  Are these delivered to the venue?

> Please consider moving definitions early in the document.

Agree.

Russ

---

From: Robert Sparks <[email-masked]>
Date: Fri, 11 Nov 2011 17:33:27 +0800

Trimming to only those places I've responded.

> This is actually current policy, I believe, even if some ADs are approving a third slot for very busy WGs.
I wasn't aware of it.

Is it really current policy to restrict the 1st tier conflicts to 10 groups?
That won't work for area meetings or for groups like DISPATCH.

> I do not understand this comment.  Robert, please try again.

The document calls out the "Do not schedule on <a particular day, like Friday>" in
several places, including making sure the secretariat takes those requests into account
at 3.1 step f, and having the option to ignore them in section 4.3.

What's missing is a requirement to collect any justification when this constraint is added
to a meeting request in the first place.

> I think this is about side-meetings, not breakout rooms.

It's currently in section 3.1 h) which is about working group meetings.

Should it be moved to 3.2? (That section already notes that fees may apply for the room, but not necessarily for stuff in the room).

> I'm not sure.  This could lead to a very significant amount of storage.  Also, it will not be useful after a few days.
I don't think each save would be that large, and we could cause it to have a finite lifetime.
But if I'm wrong about size, we could alternatively look at making it something that could 
be returned to the user as a blob instead of storing it in the database.

---

From: Russ Housley <[email-masked]>
Date: Fri, 11 Nov 2011 17:53:19 -0500

> That won't work for area meetings or for groups like DISPATCH.

I think you can say all of RAI and 10 others.

> to a meeting request in the first place.

This is usually done to accommodate travel restrictions of WG chairs.  I recall one time that a WG chair had to be at a wedding, so they left on Thursday at Noon.

> Should it be moved to 3.2? (That section already notes that fees may apply for the room, but not necessarily for stuff in the room).

We should let the Secretariat tell us what they need.

> be returned to the user as a blob instead of storing it in the database.

That is a good thought.

Russ

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 14 Nov 2011 18:48:53 +0800

Trimming to only one particular point where I have input:

> be returned to the user as a blob instead of storing it in the database.

The actual data constituting a scheduled set is not particularly large --
unless this is done in an incredibly inept manner, I think the overhead of
storing instances of scheduling solutions is negligible.

Best regards,

	Henrik