Subject: Re: [rrg] [IRSG] IRSG Review: draft-irtf-rrg-recommendation-12.txt

From: "Stephen D. Strowes" <[email-masked]>
Date: Wed, 01 Sep 2010 11:02:54 +0100

Hi,

Looking at this with fresh eyes based on Paul's feedback...

On Wed, 2010-09-01 at 07:43 +0100, Tony Li wrote:
> Any suggestions on how we could wordsmith this?

What's missing here is not in section 17, but in the abstract and the
introduction.

The emphasis in the current text is on the review of alternative
architectures, and not on what the title of the document should imply,
i.e.:
1: What the RRG was chartered to deliver
2: What the RRG is delivering in this document

For example: The abstract mentions nothing of a recommendation. It
requires a second sentence of the form "This document presents, as
recommendation of future directions for the IETF, solutions which will
aid future scalability of the Internet. To this end, this document
surveys [...]"

Likewise, section 1 must discuss the purpose of the recommendation.
Possibly achieved by inserting a second paragraph, with the following
structure based on what Tony wrote:

- Our directive was to use an open process to determine architecture(s)
which meets these challenges, forming the basis of a recommendation for
the IETF to pursue as a solution to Internet scalability.
- We chose to seek consensus on which architectures best meet the
requirements, but did not manage to do so.
- Thus, this document presents a recommendation decided by the co-chairs
and based on much discussion on the RRG mailing list.

Without mention of a recommendation in the abstract or section 1, the
reader arrives at section 1.1 with a discussion of a recommendation to
the IETF but no context of why the RRG is recommending anything. I think
this was the crux of Paul's issue?

Cheers,
-S.

---

From: Robin Whittle <[email-masked]>
Date: Wed, 01 Sep 2010 21:44:48 +1000

Short version:  I think it would make no sense to do more work on the
                RRG Design Goals draft now, since it resulted from
                minimal discussion in early 2007 and has not been
                discussed or revised since then.

                Likewise, I think it would make no sense to use the
                current draft as a normative reference in the Report,
                or to pretend that the RRG completed a "list of
                prioritized design goals", as required by our Charter.

Hi Tony and Paul,

Tony, you wrote, in part:

> The expired normative reference is the design goals document.
> We'll be updating that and passing that up for IRSG review as
> well.

This is the first time I am aware of you expressing interest in
working on:

  http://tools.ietf.org/html/draft-irtf-rrg-design-goals-01

since 8 July 2007 when you created this version 01.

As I wrote in March this year:

  http://www.ietf.org/mail-archive/web/rrg/current/msg06160.html

this version 01 resulted from very few list messages.  Your
announcement of it was:

  http://www.ietf.org/mail-archive/web/rrg/current/msg00183.html

The scalable routing phase of the RRG began on 2007-03-17, with
message 29.  We are now up to message 7287.  There have been 40.5
months since then, averaging 179 messages a month.

The Design Goals version 01 resulted from about 36 list messages,
between about 11 people - though I guess there were off-list
discussions too.  Most of these messages were no more than a few
sentences.  By message count, this is half a percent of our list
discussions to date.

A week after you released version 01, I wrote some detailed suggestions:

  http://www.ietf.org/mail-archive/web/rrg/current/msg00203.html

thinking there would be further discussions and a new version within
weeks.  However, nothing happened.  I wrote more suggestions in
December 2007:

  http://www.ietf.org/mail-archive/web/rrg/current/msg00733.html
  http://www.ietf.org/mail-archive/web/rrg/current/msg00786.html

and there was one brief response from Xiaohu Xu.

Neither you, nor (I think) anyone else has discussed the Design Goals
on the mailing list in any substantial fashion since July 2007.

We should have worked much harder on this.

I think the proposals in the Report are not based to any substantial
degree on the Design Goals version 01, since that was a very early
document, rarely discussed or mentioned on the list.  All the actual
material is in Section 3 - just 1047 words.

A week ago, 18 people voted (all Yes) on publication of the Report:

  Tony Li, Joel Halpern, Ran Atkinson, Mohamed Boucadair, Fred Templin
  Wes George, Klaas Wierenga, K. Sriram, Robin Whittle, Ruediger Volk,
  Hannu Flinck, David Williamson, Patrick Frejborg, Stephen Strowes,
  Shane Amante, Bruce Curtis and DY (Dae Young Kim).

These people wrote about 39% of the list messages to date.  These 18
people do not include anyone from the LISP team, or proponents of
about half the 14 proposals.

Even if these 18 people, and whoever else is still engaged in the RRG,
did agree on a revised Design Goals document, it would not be a
document which was used in the preparation of the proposals.

Each proposal should, I think, contain an explicit discussion of its
goals and non-goals.  Ivip has 18 pages of goals and non-goals:

  http://tools.ietf.org/html/draft-whittle-ivip-arch-04

Due to the 500 word limit on the Summary for each proposal, the RRG
Report can't be expected to convey the full set of goals and non-goals
for each proposal.

Rather than pretend that all the proposals referred to the Design
Goals draft, it would be better to let the reader delve into each
proposal's documentation regarding goals and non-goals.

Another requirement of the Charter was to produce a taxonomy.   Bill
Herrin worked towards this and I tried to refine it in 2009:

  http://www.firstpr.com.au/ip/ivip/rrgarch/

A second attempt at a taxonomy was the Core Edge Separation (CES) vs.
Core Edge Elimination (CEE) distinction:

  http://www.ietf.org/mail-archive/web/rrg/current/msg06219.html
  (17.3.3)

  http://www.ietf.org/mail-archive/web/rrg/current/msg06110.html

Though at least one person supported writing up the CES/CEE
distinction material as an RFC, I have not had time to do it, and the
distinction and terminology has been rejected by some people,
including Ran Atkinson.

  - Robin

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 31 Aug 2010 20:50:35 -0700

Greetings. I have reviewed draft-irtf-rrg-recommendation-10.txt as part of the IRSG review process, described at <http://trac.tools.ietf.org/group/irtf/trac/wiki/IRTF-RFCs>. That page says "The purpose of the IRSG review is to ensure consistent editorial and technical quality for IRTF publications."

This document definitely meets a high standard for editorial and technical quality. There are a few issue that I (as a definite outsider to routing in general, much less next-generation routing in specific) think could be usefully addressed before publication. Please take these suggestions as proposals for improvements, not requirements.

There is no description of who wrote the rebuttals and counterpoints. I don't think that names need to be used, but a description in the introduction of the process used to collect rebuttals and counterpoints would give useful context to the reader.

Two of the proposals, NOL (Section 6) and 2-phased mapping (Section 9), have no references, which means that the entire technical description of the proposal is contained in this document. However, neither of those were particularly clear and concise, as compared to most of the rest of the proposals. Having an explanation of why they discussed here without further reference would be useful to understand where they stand in the pantheon of routing proposals.

Many of the Internet Drafts in the references are long-expired, including one of the two normative references. This will make for an interesting conversation with the RFC Editor.

This last recommendation is obviously the trickiest one, and it may be a touchy subject. From reading this document without reading the mailing list, I have absolutely zero idea why Section 17, the recommendations, exists. Section 1.1 says:
   The group did not reach consensus on this
   recommendation, thus the recommendation reflects the decision of the
   co-chairs.  The group did reach consensus that the overall document
   should be published.
This can be (mis)interpreted many ways: 
  - The RG could not come to consensus on recommendations, so the co-chairs are reporting the strongest agreement they could find.
  - The RG could not come to consensus on recommendations, so the co-chairs are taking advantage of this and saying what the two of them like.
  - The RG only wanted the overall document without recommendations published, but the co-chairs thought that didn't go far enough to be useful, so they added some recommendations almost as a placeholder.
  - The IETF demands recommendations, and we couldn't come to consensus, so we put some in, but you can ignore them because they aren't really RG consensus.
  - One of the three recommendations listed is co-authored by one of the co-chairs, so she must have insisted that it be listed and consensus be damned.
  - It's all dumb IETF politics! Run away!
Many of those interpretations are unhelpful and do a disservice to the obviously large amount of work in the RG that went into this document. There hopefully would be a clear explanation of why the co-chairs get to give their recommendations when there was no RG consensus, and hopefully also why there was not consensus in the RG. Short of being able to add that, please strongly consider replacing the section with a few paragraphs about the lack of consensus on recommendations and an apology for not being able to give recommendations.

As I said, I recognize that this last bit is tricky, and there are probably good reasons for the document being the way it is now. However, that doesn't help the reader who is not part of the RG, much less the reader 20 years from now who wants to know why the IETF chose the new routing architectures that it did. If the document is published without context about the recommendations, the document will lose a fair amount of its long-term usefulness.

Please let me know if you have any questions.

--Paul Hoffman, Director
--VPN Consortium

---

From: Tony Li <[email-masked]>
Date: Wed, 1 Sep 2010 10:26:28 -0700

Hi Robin,

> since 8 July 2007 when you created this version 01.

Please read:

http://www.ietf.org/mail-archive/web/rrg/current/msg07262.html

> this version 01 resulted from very few list messages.

This is correct.  There was a considerable amount of discussion offline and during meetings.

> We should have worked much harder on this.

Actually, we did work really hard on this.  I realize that you didn't see it because you didn't attend, but we spent many hours discussing these issues.  It is grossly unfair of you to ignore that.  

> document, rarely discussed or mentioned on the list.  

That was the choice of the designers of the proposals.  The goals were there so that the group understood the baseline.  They were also there for participants to use in critiquing the proposals.

> about half the 14 proposals.

Consensus is only taken across those who choose to participate. 

> document which was used in the preparation of the proposals.

Again, you're inferring from your own personal experience.  It is grossly unreasonable for you to state what the remainder of the group was or was not thinking about.  You simply do not know.

Tony

---

From: Tony Li <[email-masked]>
Date: Wed, 1 Sep 2010 10:32:46 -0700

Hi Paul,

> Let me drill down a bit on that. Was your process going into this "get consensus; if success, publish the results of that consensus; if fail, don't publish anything" or "get consensus; if success, publish the results of that consensus; if fail, let the co-chairs write up their own views".
> 
> OK, that wasn't really fair. I will assume that the process going into this was "get consensus and publish the results of that consensus" with no thought of failure.

You are correct, the process did not clearly articulate the details in the case of a lack of consensus.  The chairs then proceeded to make lemonade.

>> We've been trying to be as clear as possible about this and it took several passes to get to the current wording.  Any suggestions on how we could wordsmith this?
> 
> That is really between the RG and Aaron, I think. Determine if the process was open, and whether you followed it. If so, simply add a paragraph about the process in the introduction and/or Section 17. If not, maybe try again, even if process fatigue has set in.

Thanks, that sounds good, will add.

Regards,
Tony

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 02 Sep 2010 13:25:35 +1200

> Hi Paul,

...
>> Many of the Internet Drafts in the references are
>> long-expired, including one of the two normative
>> references. This will make for an interesting conversation
>> with the RFC Editor.
> 
> 

Being a member of the RSAG (RFC Series Advisory Group), I can't
resist the urge to comment, with no authority whatever. The
concept of 'normative' references was invented, iirc, to
identify references that an implementor of an IETF Standard
necessarily must read in order to produce a correct
implementation. I don't see the value of splitting the
references for an IRTF Infomational document; if I'd noticed
that during RRG Last Call I would have said so then. So this
issue is easily fixed by simply listing 'References' rather than
splitting them into two heaps.

> Note that many of the other proposals are not opting to
> publish as RFC status, so many of the informational
> references will time out.  I'm not particularly happy about
> this, but I don't see a good solution either.  We can't
> exactly force people to publish.

Strangely enough, the RSAG's sub-committee on citation formats
(yes, there is really such a sub-committee) is today discussing
whether long expired drafts should continue to be cited as "work
in progress". Indeed, Dave Crocker just came up with several
possible alternatives:

>>    WNLIP  - Work No Longer In Progress
>> 
>>    WA     - Work Abandoned.
>> 
>>    WRIP   - Work Resting In Peace 

I rather like WRIP. For now, though, the convention is to label
them as "work in progress" even when it's a lie. There's one
level below that, "private communication", but Google makes that
a pretty rare case.

    Brian

---

From: Robin Whittle <[email-masked]>
Date: Thu, 02 Sep 2010 12:25:43 +1000

Hi Tony,

> http://www.ietf.org/mail-archive/web/rrg/current/msg07262.html

OK - I missed this message last week.  My point remains that there has
been no revision to this document for over 3 years, and that my three
attempts to discuss improvements on it, in July and December 2007, did
not result in any significant discussion from you or anyone else.

> offline and during meetings.

OK.

> Actually, we did work really hard on this.  I realize that you
> didn't see it because you didn't attend, but we spent many hours
> discussing these issues.  It is grossly unfair of you to ignore
> that.

I didn't mean that you and others didn't work hard on the Design Goals
leading up to version 01.  I meant that in the three years since, we
should have worked harder on it.  I do not regard the version 01,
followed by three years of no further discussion, as fulfilling one of
the requirements of the Charter:

  Milestones

      It is hard to forecast exactly how the consensus process
      will play out, and therefore the precise timing to achieve
      various milestones. However, as a starting point, the RRG
      will produce a document enumerating the design goals for a
      new routing architecture.

I did my bit, starting a week after you released version 01 - and
there was no interest.  After Iljitsch van Beijnum chided me for the
lack of goals and non-goals in the first Ivip I-D, I did a proper job
of it in Ivip-arch-04, earlier this year.  Anyone could have used this
as a basis for discussion, or as input for revising the Design Goals.

My attempt at writing a Recommendation (2010-03-09):

contained 1830 words on design goals - about 80% more than version 01
of the RRG Design Goals I-D.  I don't recall anyone debating these or
suggesting they be used to revise the RRG Design Goals I-D.

If you, Lixia, and other RRG participants had continued to work on the
Design Goals from mid-2007 then I think we would have had a much more
comprehensive document as a basis for the rest of the RRG work.

This is all the more important now, since the RRG Report does not
contain any proper discussion of goals and non-goals, and the 500 word
limit on the Summaries makes it impossible for each proposal to have
its goals and non-goals clearly described.

I would support the publication of version 01 of the Design Goals I-D
as in informational RFC, provided it carries a disclaimer that it was
developed in the early months of the 3+ years of the RRG's scalable
routing work, and that it was not widely discussed, cited, or revised
since then.

If you want to work on a significant revision, my suggestions are
still probably relevant:

  http://www.ietf.org/mail-archive/web/rrg/current/msg06219.html 17.2

That would be a good project, albeit several years late.  I sense that
most people are keen to wind up their RRG involvement, so I can't
imagine that there will be significant discussion and revisions to the
RRG Design Goals.  If this does occur, and result in a better
document, then that would be great.  However, I think its publication
as an RFC should involve a clear statement of how significant
revisions were made after the proposals and Report was finalised and
so do not reflect a shared understanding between the designers of the
various proposed architectures.

>> I think the proposals in the Report are not based to any
>> substantial degree on the Design Goals version 01, since that was
>> a very early document, rarely discussed or mentioned on the list.
> 
> That was the choice of the designers of the proposals.  The goals
> were there so that the group understood the baseline.  They were
> also there for participants to use in critiquing the proposals.

Yes, but that was version 01, from July 2007.  Since then, more people
became involved and many ideas and proposals have been developed.

>> These people wrote about 39% of the list messages to date.  These
>> 18 people do not include anyone from the LISP team, or proponents
>> of about half the 14 proposals.
> 

I understand this.  My point is that the people who seem to be
involved in the RRG now only represent a subset of those who have been
involved in the last few years, and who contributed to the proposals
in the Report.

>> Even if these 18 people, and whoever else is still engaged in the
>> RRG, did agree on a revised Design Goals document, it would not
>> be a document which was used in the preparation of the
>> proposals.
> 
> Again, you're inferring from your own personal experience.  It is
> grossly unreasonable for you to state what the remainder of the
> group was or was not thinking about.  You simply do not know.

OK - here is a clearer version:

It appears that at this late stage of the RRG, there are too small a
subset of the RRG contributors (as measured over 3 years by mailing
list messages, and by their proposed architectures in the Report) to
arrive at a document which could be properly regarded as any kind of
reference for the proposals in the Report.

The version 01 is so old, and has so rarely been discussed or cited,
that it cannot properly be regarded as a basis for these proposals, in
the absence of the proposals themselves explicitly citing it.  Any
substantial revisions to it now would have been made well after the
proposals and the Report was written.

  - Robin

---

From: Tony Li <[email-masked]>
Date: Wed, 1 Sep 2010 20:16:34 -0700

[Audience restricted back to the RG, as this is no longer relevant to the IRSG.]

Robin,

> not result in any significant discussion from you or anyone else.

Correct.  And it's now time that we discuss improvements on it.

> should have worked harder on it.

Your criticism has been duly noted.

> of the RRG Design Goals I-D.

Volume is not indicative of quality or value.

> I don't recall anyone debating these or

Because we decided to put the design goals document down until we made further progress elsewhere.  

> its goals and non-goals clearly described.

Criticism of history seems irrelevant, not important.

> routing work, and that it was not widely discussed, cited, or revised

That would be incorrect.  It was widely discussed.  Just not by you.

>  http://www.ietf.org/mail-archive/web/rrg/current/msg06219.html 17.2

Thank you for the reminders.

> became involved and many ideas and proposals have been developed.

Which is exactly why we wish to revise the goals.  We have learned more about what is important and what isn't.

> involved in the last few years, and who contributed to the proposals

Wholly irrelevant.  The active members of the group at any given time _are_ the group.

> proposals and the Report was written.

Thank you for your opinion.  However, we will be moving the document forward.

Regards,
Tony

---

From: Robin Whittle <[email-masked]>
Date: Fri, 03 Sep 2010 00:12:45 +1000

Short version:   I support improving the Design Goals and publishing
                 it as an RFC.

                 I think it is important to mention on the list the
                 difficulties the RRG has had, since enquiring minds
                 in the future will want to know how the Report and
                 Design Goals RFCs came into being.

Was:

  Re: [rrg] [IRSG] IRSG Review: draft-irtf-rrg-recommendation-12.txt

Hi Tony,

I wrote my first attempt (msg00203) to improve the Design Goals I-D,
three days after you announced it on the list.  I-Ds are drafts and
involve an implicit request for improvements.  As far as I can see,
there's nothing you wrote to the list, or which you mentioned in the
2007-07-29 meeting (according to the minutes), about not wanting any
input on this I-D.

Scanning the list archives to the end of 2007, I see no mention from
you or Lixia that you were not interested in improving the Design
Goals I-D.  On 2007-12-03, you indicate to Russ White that you are
interested in input on this I-D:

   The design goals document captures some technical issues that we
   deem worthy of discussion. If that seems insufficient, we're
   certainly open to proposals.

So I had no reason to believe that my msg00203, and two others in
December 2007 (msg00733 and msg00786) were not appropriate to
furthering the work of the RRG.

> Correct.  And it's now time that we discuss improvements on it.

I think it has always been time.

> Your criticism has been duly noted.

Thanks.

> Volume is not indicative of quality or value.

Sure, but if you think what I wrote was sub-standard, irrelevant or
had any other failings, it would have been good - and still would be
good - if you provided constructive criticism.

I think you sometimes take your preference for brevity too far in your
own communications, at least in respect of the mailing list.  I don't
recall you writing anything about why you are uninterested in, or
disagree with, what I wrote in msg06219 or in the three 2007
contributions.  Nor can I recall any message you or Lixia wrote about
why the Design Goals should not have been discussed from mid-2007
until now.

> Because we decided to put the design goals document down until we made further progress elsewhere.  

I follow this mailing list pretty closely, and I don't recall you
stating such a decision before.

> Criticism of history seems irrelevant, not important.

Design goals are crucially important.

The proposals were made with only the quickly developed, and not
further discussed, Design Goals 01.  Proponents didn't necessarily
accept those goals.  They didn't have space in your 500 word format to
explain each proposal's design goals.

I think this should be known to anyone who reads the Report.  Some
people who read it will make important decisions, based in part on the
Report - so they need to know about any difficulties in the
preparation of the Report.

These include:

  Only a quickly developed set of Design Goals, which were not
  the subject of any on-list two or more way discussion since July
  2007.

  Your and Lixia's attempts to stop discussion of "proposals"
  (candidate architectures) for much of 2008 and 2009, in favour
  of "architectural" discussion (as distinguished from "engineering"
  or "proposals").  Yet I don't recall you leading by example in the
  type of discussion you wanted us to have.

  Your lack of interest in discussing the constraints imposed by the
  need for widespread voluntary adoption:

    http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

  What I believe is your lack of interest in the last year or two in
  discussing in detail any proposal other than ILNP.

  Your and Lixia's apparent misunderstanding of some proposals:

    The co-chairs do not understand Ivip and some other architectures
    http://www.ietf.org/mail-archive/web/rrg/current/msg06373.html
    (2010-03-27).

  The 500 word limit on Summaries, Critiques etc. - and the inability
  for multiple critiques to be included in the Report.  For instance,
  the 500 word version of my critique of LISP is in the Report,
  probably because I wrote it before Noel Chiappa wrote his
  (msg05747).

  The general pattern of proponents of proposed architectures
  failing to critique - or show they have read and understand - other
  architectures (msg06285).

  Failure to correct or acknowledge problems in the way ILNP is
  represented in the Report (msg07254 & msg07185).

So I think the history of the RRG is important, and worth mentioning
on the list for the benefit of people today and in the future.

> That would be incorrect.  It was widely discussed.  Just not by you.

It was not revised in 3 years, and I don't recall it being widely
discussed on the mailing list.

> about what is important and what isn't.

We have been learning continually - and I think the Design Goals I-D
could and should have been improved on a continual basis.

> Wholly irrelevant.  The active members of the group at any given time _are_ the group.

Sure, but let's not pretend that the Report, most of which was
finalised in early 2010, was based substantially on Design Goals 01,
or on any later revision we make in late 2010.

Enquiring minds in the future will want to know why development of the
Design Goals was halted so early.

> Thank you for your opinion.  However, we will be moving the document forward.

I fully support improving the Design Goals and publishing it as an RFC
- as long as the history of its development, and how out of sync it
has been with the main body of RRG work, is clearly stated in the RFC.

  - Robin

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 1 Sep 2010 09:40:12 -0700

At 11:43 PM -0700 8/31/10, Tony Li wrote:
>The expired normative reference is the design goals document.  We'll be updating that and passing that up for IRSG review as well.

Great. Note that I don't need to be Cc'd on the ensuing discussion of this topic; I just wanted to point out that the current state of a long-expired draft as a normative reference *might* cause problems with the RFC Editor; it also might not.

>Note that many of the other proposals are not opting to publish as RFC status, so many of the informational references will time out.  I'm not particularly happy about this, but I don't see a good solution either.  We can't exactly force people to publish.

Exactly right. Thus, a note in the introduction about this could be useful, given that the research value of an academic paper published a few years ago and an Internet Draft that expired a few years ago are approximately the same.

>You're right, this is tricky.  We've been over the wording quite a bit, but apparently it's still not sufficiently clear. 

The recommendations themselves are clear; the motivations for why they are in this document are what is not.

>Our directive from Aaron was very simple: whatever process we chose, it had to be open.  We could have flipped a coin, rolled dice, or thrown darts.  Instead, we tried for consensus.  When that didn't materialize we added our own two cents. 

--Paul Hoffman, Director
--VPN Consortium