Subject: Re: [RAI] RAI processes for handling work effectively

From: Harald Alvestrand <[email-masked]>
Date: Thu, 18 Jul 2013 13:57:14 +0200

Gonzalo,

let me make a radical proposition....

Standing working groups are useless. All of them.

What we have been doing in the IETF for 10+ years is to create things 
that we call working groups, which sometimes are used as places to do 
work, but really are places where people go to get a review and a stamp 
of approval.

This isn't the role of a working group. It's a review panel.

Working groups need a task they can complete. They need chairs that are 
willing and able to get the hard questions asked, get a decision out, 
and PUBLISH.
Review panels need to have competent people on them, and they need to be 
able to say NO at need.

The drivers for these 2 kinds of activity are orthogonal-to-opposed, 
they are not aligned.
And by using WG procedures for both kinds, we are doing a disservice to 
ourselves.

I'd suggest that among the requirements for a working group to be formed 
- AND for it to continue to exist - there should be:

- A definition of the problem that needs solving
- A definition of the work that needs to be done
- A sign that one can look at in order to tell that it's finished

(Picking on MMUSIC not because it's the only one, but because I'm 
working in it at the moment)

In contrast, the MMUSIC working group includes statements like "Various 
extensions to SDP will be pursued to remedy the most urgent of SDP's 
shortcomings". How do we tell that we'll achieve that?

To my mind, SCTP-in-SDP could have been a working group. Once there are 
two working implementations that can successfully negotiate SCTP, its 
work is done and can shut down.

Trickle ICE could have been another. The day we have 2 implementations 
interworking - done.

Both need to have competent review and input from people who are aware 
of the architectural issues of SDP, and the people who provide that 
input need to talk to each other so that each reviewer is in sync with 
the others.

But that isn't the role of a WG. It's a review panel.

         Harald

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 18 Jul 2013 09:55:50 -0400

Harald,

What you say makes some sense to me.

But if we were to act on it then I think we need to replace some WGs 
with review panels. (We have an SDP directorate, but I don't think that 
is a sufficient replacement for mmusic.) And so we will need some clear 
processes for review panels. How would we process drafts that would 
otherwise have been handled through the standanding wg? Would they be AD 
sponsored? And we may need to schedule meeting sessions for review 
panels too.

This seems like it would need to be an ietf-wide change, not just RAI. 
(But maybe we could pilot it in RAI.)

	Thanks,
	Paul

On 7/18/13 7:57 AM, Harald Alvestrand wrote:
> _______________________________________________
> RAI mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rai
>

---

From: Spencer Dawkins <[email-masked]>
Date: Thu, 18 Jul 2013 09:47:40 -0500

On 7/18/2013 8:55 AM, Paul Kyzivat wrote:
> with review panels. (We have an SDP directorate, but I don't think 
> that is a sufficient replacement for mmusic.) And so we will need some 
> clear processes for review panels. How would we process drafts that 
> would otherwise have been handled through the standanding wg? Would 
> they be AD sponsored? And we may need to schedule meeting sessions for 
> review panels too.
>

I could be in the rough on this, but if RAI wants to head in that 
direction, I'd suggest doing so in RAI alone as an 
http://tools.ietf.org/html/rfc3933 process experiment. That would bypass 
two frequent sources of failure:

  * Getting everyone in the entire IETF to agree (at least roughly) to
    that this is where we all want to go, and
  * Getting everyone in the entire IETF to agree (at least roughly) to
    change BCP text for the entire IETF, with no operational experience.

 From RFC 3933:

    The IESG is explicitly authorized to use this mechanism (based on
    Experimental RFCs) to gain experience with proposed changes to BCP
    specifications.  There is no requirement to approve a BCP
    specification for the experiment until the experiment is found to
    have value.

and

    In each case, the IESG's decision to go forward (or not) with a
    procedural experiment, or the steps leading up to one, is expected to
    reflect their judgment of the existence of rough consensus in the
    community.  That judgment may be appealed using the usual procedures,
    but the IESG and the community are reminded that an experimental
    attempt to try something for a fixed period is typically a better
    engineering approach than extended philosophical discussion without
    any experience to back it up.

No one understands the last sentence better than Harald, of course ...

Spencer

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 18 Jul 2013 10:58:07 -0400

In many ways you're right of course - at least for the "long-term" Working Groups.  I find that many of the newly-created ones do perform real work at the beginning, though.

But I strongly disagree with what's really driving your angst: the problems in MMUSIC with making SDP changes.

The real problem you have is you'd like to have *your* changes "rubber-stamped" quickly.  And I don't mean that in a negative way - I mean the problem is you'd like WebRTC usage for SDP to follow a faster get-to-publication process, because WebRTC has a greenfield deployment model and few vendor implementations involved and is in an early phase of life.  So you see the problem as: "we've got two vendors wanting to do X, and if X isn't broken for our needs then publish the damn thing".

But SDP isn't just WebRTC's playground to try new stuff in.  It's not even simply a client-server encoding schema.  Instead, it's a *protocol*, with a complicated multi-point offer/answer model and state.  One that's decades-old, widely-deployed, multi-usage/market, and widely-implemented by a crap-load of vendors.  Sadly, it's also used, implemented and deployed in ways that are not well documented in RFCs, but are known by the participants in MMUSIC.  Because of all that, it's arguably actually *inappropriate* to make changes to it quickly, or for specific use-cases/markets.  Having two vendors implement something and agree on it simply isn't a sufficient bar for publication to RFC, for SDP uses.

Part of that slow-change-process also has to do with what the expectations are for published RFCs.  Despite the distinctions made in RFC 2026/6410, the fact of the matter is once a document is published as an RFC, it's essentially set in stone and treated like gospel - for RFCs updating/extending mature protocols, which SDP is.  At least from the perspective of how the rest of the world treats it.  It's not like just saying: "here's version 1.0, please keep checking back for new versions which might change this thing completely".

That's one of the reasons I argued so much a couple years ago that RTCWEB/WebRTC should *NOT* use SDP for its API.  I tried to warn you guys this would happen.  It was #1 on my "reasons not to use SDP" argument list:
http://www.ietf.org/mail-archive/web/rtcweb/current/msg01256.html

Tying WebRTC to SDP is going to hurt your ability to make changes quickly... FOREVER.

-hadriel

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Thu, 18 Jul 2013 15:35:09 +0000

As a process experiment, that already exists - what do you think dispatch has been doing for the last few years.

It seems more to me that Harald is raising the question of whether that applies to MMUSIC.

In terms of the current thorny problem that Harald wants to resolve - no it would not have helped at all - he would still have a thorny problem, and it would probably have been shot down at the IETF review stage rather than having the discussion in the working groups.

Keith

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Spencer Dawkins
Sent: 18 July 2013 15:48
To: Paul Kyzivat
Cc: [email-masked]
Subject: Re: [RAI] RAI processes for handling work effectively

But if we were to act on it then I think we need to replace some WGs with review panels. (We have an SDP directorate, but I don't think that is a sufficient replacement for mmusic.) And so we will need some clear processes for review panels. How would we process drafts that would otherwise have been handled through the standanding wg? Would they be AD sponsored? And we may need to schedule meeting sessions for review panels too.

This seems like it would need to be an ietf-wide change, not just RAI. (But maybe we could pilot it in RAI.)

I could be in the rough on this, but if RAI wants to head in that direction, I'd suggest doing so in RAI alone as an http://tools.ietf.org/html/rfc3933 process experiment. That would bypass two frequent sources of failure:

  *   Getting everyone in the entire IETF to agree (at least roughly) to that this is where we all want to go, and
  *   Getting everyone in the entire IETF to agree (at least roughly) to change BCP text for the entire IETF, with no operational experience.

   have value.

and

Spencer

---

From: SM <[email-masked]>
Date: Thu, 18 Jul 2013 12:18:12 -0700

Hi Harald,
>let me make a radical proposition....

http://www.alvestrand.no/ietf/ietf55/Revolutionizing%20the%20IETF.pdf :-)

>What we have been doing in the IETF for 10+ years is to create 
>things that we call working groups, which sometimes are used as 
>places to do work, but really are places where people go to get a 
>review and a stamp of approval.

Yes.

>Working groups need a task they can complete. They need chairs that 
>are willing and able to get the hard questions asked, get a decision 
>out, and PUBLISH.
>Review panels need to have competent people on them, and they need 
>to be able to say NO at need.

It is not easy for Chairs to get the hard questions asked.  It's not 
easy for a review panel to say no unless they have some authority.  I 
don't think it is a matter of the people not being competent; there 
are also social issues.

>And by using WG procedures for both kinds, we are doing a disservice 
>to ourselves.
>
>I'd suggest that among the requirements for a working group to be 
>formed - AND for it to continue to exist - there should be:
>

Yes.

>In contrast, the MMUSIC working group includes statements like 
>"Various extensions to SDP will be pursued to remedy the most urgent 
>of SDP's shortcomings". How do we tell that we'll achieve that?

The working group could be recharted instead of having open-ended 
statements in its charter.

Regards,
-sm

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Thu, 25 Jul 2013 18:42:47 +0000

On Jul 18, 2013, at 5:57 AM, Harald Alvestrand <[email-masked]> wrote:

> To my mind, SCTP-in-SDP could have been a working group. Once there are two working implementations that can successfully negotiate SCTP, its work is done and can shut down.
> 
> Trickle ICE could have been another. The day we have 2 implementations interworking - done.

These are interesting examples to consider.

I think the existing dispatch process would have easily allowed creation of mini WG for either of these topics. I seem to recall that at the time the authors of the drafts were in favor of going to existing WGs.

---

From: Harald Alvestrand <[email-masked]>
Date: Sat, 27 Jul 2013 20:49:25 +0200

> But I strongly disagree with what's really driving your angst: the problems in MMUSIC with making SDP changes.
Hadriel, what's driving this is a lot older than my involvement with SDP
(this round).
SDP is just my current whipping horse for a persistent, pernicious, and
prevalent problem.

> But SDP isn't just WebRTC's playground to try new stuff in.  It's not even simply a client-server encoding schema.  Instead, it's a *protocol*, with a complicated multi-point offer/answer model and…

All of this points to a need for careful review. What it does not point
out is any need for a *slow*, *inconsistent* and *unpredictable* review
process - which is what I feel the "standing working groups" are giving
us now.

My flagship this week is the reviews of BUNDLE in MMUSIC: What I have
heard over the last six IETFs (2 years, since my August 2011 -one-rtp
draft) is advice wavering back and forth between "same port", "different
ports", "MMT", "an M-line is a channel", "an M-line is a stream", "an
M-line is a decorated 5-tuple" and variations thereof - sometimes the
same person giving opposite advice at successive IETF meetings. Every
time, new information comes up at meetings.

In between, the list discusses clock rates, RTSP, alt-ports, loopback,
and a ton of other stuff that *I* don't care much about, bu the people
discussing them certainly do.

With BUNDLE, Christer has been doing yeoman's work to sort through the
issues and figure out what matters, what doesn't, and what we just have
to live with - but it should not be *this* hard.

It's possible that the problem could be solved by a different management
style, without changing the current formal structure. But I doubt it;
I've seen this in too many contexts.
> Part of that slow-change-process also has to do with what the expectations are for published RFCs.  Despite the distinctions made in RFC 2026/6410, the fact of the matter is once a document is publ…

That may be a reason why we need to publish more bad ideas as RFCs :-)

One of our fairy tale stories that we tell ourselves is that our
protocols are tools, can be adopted into different contexts, and
profiled appropriately for that context.

> Tying WebRTC to SDP is going to hurt your ability to make changes quickly... FOREVER.

At this time, I can't say that you're wrong.

But it is an orthogonal issue to my opinion that standing working groups
Just Don't Work.

Harald

---

From: Martin Thomson <[email-masked]>
Date: Sun, 28 Jul 2013 01:48:54 -0700

> process - which is what I feel the "standing working groups" are giving

Isn't that just a natural by-product of using SDP, with all that
baggage Hadriel describes?

I don't think that you get any better review if you make new working
groups.  In fact, you get even slower, less consistent and possibly no
review at all.  One virtue of the long-standing working group is that
over time it accumulates a following of people who know stuff about
the work.  I saw this with the DHC WG, which was actually invaluable
when it came to bringing work there.

Again, not an indictment of your thesis about more focused working
groups, just an observation about this particular problem.

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 28 Jul 2013 12:31:50 +0200

> baggage Hadriel describes?

Well - I could have made a lot of the same comments about AVTCORE and
the RTP issues we are facing, but MMUSIC just felt like a more current
example :-)

> when it came to bringing work there.
That's where my thinking about review teams, directorates and other
possible constructs comes in. I think one core point here is that for
any question asked, there needs to be some identified "someone" who's
charged with coming up with "the answer" (or "the information needed to
make sense of the question") - and is able to do so on a relatively
short timeframe.

Quite hard to achieve consistently in a volunteer organization, but I
don't think it's impossible to try something that might work better than
the WG format.

---

From: Martin Thomson <[email-masked]>
Date: Sun, 28 Jul 2013 03:51:13 -0700

> don't think it's impossible to try something that might work better than

I don't think that you have the human-factors aspects of your argument
quite right here.

The desire to have some mechanism for aggregating expertise is
understandable, but I just can't get around the volunteer organization
problems.  Personally, I am quite reluctant to volunteer for review
teams or directorate for a number of reasons.  Those reasons include
the continuing, unwelcome imposition on my time (I review for gen-art
and appsdir, and on balance am happy doing those occasional reviews,
but I still dread the emails that include assignments).  I am also
unlikely to volunteer for something like an SDP directorate, simply
because I know that my knowledge is lacking.  However, if that were
the criterion used to gate entry into any directorate, then no
directorate would ever have members.

On the other hand, working groups accrue members as people want to do
work.  That produces groups of people who might only know a small
amount about the given topic, but often those people are still capable
of detecting bugs.  That said, you aren't thinking much like an
engineer if that is your goal.  Surely you can think of better ways to
test hypotheses than using an imperfect system like the IETF?

--Martin

p.s. I have little faith in the value of reorganization or other such
nonsense in light of the discussion thus far.  Is there actually a
clear, unambiguous problem here to solve?

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 30 Jul 2013 10:27:18 -0400 (EDT)

> From: Harald Alvestrand <[email-masked]>
> 
> I think one core point here is that for

That's hard to do if you don't have (1) a budget for maintaining a
staff of experts, either full-time or on a consulting basis, and (2)
granting the "someone" the authority to have their answer accepted by
all parties.

Dale