Subject: [INDEP] Queue time?

From: Jeroen Massar <[email-masked]>
Date: Thu, 08 Jun 2006 13:53:46 +0200

[Lets see if I can formulate this with nice wordings...]

How long is the average queue time, otherwise said, how long does it
take for a submission to be looked at? The status codes say that
documents in ISR are under review, but quite a number of them have been
in that state already for over a year (+1 day in my case).
There are even items going back to 2002 and 2003, which is three years
ago as listed on http://www.rfc-editor.org/queue.html

Is it for the same reason that port numbers assignements took a long
time to be processed, aka too little man&woman power or is there
something else that authors should do to speed up the process.

A related question is the copyright/expiry statement as these are set to
6 months, while the document is in the queue it can expire. How is this
handled?

Greets,
 Jeroen

_______________________________________________
INDEPENDENT mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/independent

---

From: Keith Moore <[email-masked]>
Date: Thu, 08 Jun 2006 08:55:16 -0400

> take for a submission to be looked at? 

To me this seems like the wrong question.

Authors should not assume that they have some right to timely review 
and/or publication by the RFC Editor.  Or to put it another way, the RFC 
Editor should not be expected to commit arbitrary amounts of resources 
to handle an arbitrary volume of submissions.  Independent submissions 
might be valuable, but (IMHO) they are not the RFC Editor's primary job.

Also from my experience with IESG, the worst documents were the ones 
that required the most review time, because we weren't allowed to simply 
say "no".  RFC Editor must be able to say "no" in various ways including 
not just "no, for this reason" but also "no, and we don't see how to fix 
this document to the point where we would publish it", or "no, and we 
don't have time to explain to you what it takes to fix this".  I mention 
this here because it has a significant impact on workload.

I find myself thinking that the RFC Editor needs to be able to ask 
authors to supply additional information to support their documents, 
including: endorsement from recognized community members or well-known 
authorities, detailed analysis of one or more aspects of the protocol, 
etc.  The point here is to get the RFC Editor out of the business of 
trying to decide whether a protocol is technically sound.  The author 
should provide a convincing argument (if not outright evidence) that the 
protocol is sound.

I also find myself thinking that the RFC Editor should avoid trying to 
maintain a complicated state machine for documents.   Maybe there should 
be a limit to the number of round-trips (say 3) between author and RFC 
Editor before a document is either rejected or approved for publication 
(there could be additional round-trips post-approval to clean up 
editorial issues).

Keith

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jun 2006 09:12:40 -0400

--On Thursday, 08 June, 2006 08:55 -0400 Keith Moore
<[email-masked]> wrote:

>> How long is the average queue time, otherwise said, how long
>> does it take for a submission to be looked at? 
> 
> 
> Authors should not assume that they have some right to timely
> review and/or publication by the RFC Editor.  Or to put it
> another way, the RFC Editor should not be expected to commit
> arbitrary amounts of resources to handle an arbitrary volume
> of submissions.  Independent submissions might be valuable,
> but (IMHO) they are not the RFC Editor's primary job.
> 
> Also from my experience with IESG, the worst documents were
> the ones that required the most review time, because we
> weren't allowed to simply say "no".  RFC Editor must be able
> to say "no" in various ways including not just "no, for this
> reason" but also "no, and we don't see how to fix this
> document to the point where we would publish it", or "no, and
> we don't have time to explain to you what it takes to fix
> this".  I mention this here because it has a significant
> impact on workload.

Please read draft-klensin-rfc-independent... It covers this
issue and the RFC Editor's right to say "no" at any time and for
any reason.  There are safeguards, but the author basically has
to convince others that the cause is worthwhile.  One could add
a more specific list of reasons, but I'm not sure they are
helpful.

> I find myself thinking that the RFC Editor needs to be able to
> ask authors to supply additional information to support their
> documents, including: endorsement from recognized community
> members or well-known authorities, detailed analysis of one or
> more aspects of the protocol, etc.  The point here is to get
> the RFC Editor out of the business of trying to decide whether
> a protocol is technically sound.  The author should provide a
> convincing argument (if not outright evidence) that the
> protocol is sound.

See above.

> I also find myself thinking that the RFC Editor should avoid
> trying to maintain a complicated state machine for documents.
> Maybe there should be a limit to the number of round-trips
> (say 3) between author and RFC Editor before a document is
> either rejected or approved for publication (there could be
> additional round-trips post-approval to clean up editorial
> issues).

Given that there are always special cases, do you think this is
worth turning into a rule that might be incorporated into a
contract.   I would think that merely encouraging the RFC Editor
to use good sense and discretion would be sufficient (and, in
the domain of maximum number of round trips, I think the
evidence is that good sense and discretion are already being
applied).

    john

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 08 Jun 2006 15:19:44 +0200

On Thu, 2006-06-08 at 08:55 -0400, Keith Moore wrote:
> and/or publication by the RFC Editor.

I am not asking for that much, to put it bluntly:
 "am I wasting my time when there is nothing being done with it" ?

>   Or to put it another way, the RFC 

I agree on this and I rather have indeed that they say "no because ..."
then "no". On the other side I also rather see "no this is not something
what we will ever publish" instead of silence so that I can move on.

> should provide a convincing argument (if not outright evidence) that the 

I fully agree with this and I am very willing to provide input, but when
none is asked for then there is none to give...

> (there could be additional round-trips post-approval to clean up 

Well the reason it is in "ISR" now is simply as that is the first state
it entered into. I submitted the document, it was acked that it was
received and it got into the queue. After that zilch....
Which is why I asked if there was a workload issue as there has not been
any further communication.

As for the reason why I submitted draft-massar-v6ops-heartbeat-01.txt as
an individual submission, which is the same as the reason why the tsp
draft, draft-blanchet-v6ops-tunnelbroker-tsp-03.txt in this queue: v6ops
stopped working on tunneling related protocols and thus these could not
find a home there and not in other places. They are used though heavily,
there are multiple implementations of both and as such as (I think I can
say this for the TSP one too) would like to have it documented in the
IETF so that interoperable implementations can be, and have been, build
based on these drafts. And that was afaik why the whole RFC series idea
was there.

Greets,
 Jeroen

---

From: Michael Richardson <[email-masked]>
Date: Thu, 08 Jun 2006 10:42:59 -0400

>>>>> "Jeroen" == Jeroen Massar <[email-masked]> writes:
    Jeroen> How long is the average queue time, otherwise said, how long
    Jeroen> does it take for a submission to be looked at? The status
    Jeroen> codes say that documents in ISR are under review, but quite
    Jeroen> a number of them have been in that state already for over a
    Jeroen> year (+1 day in my case).  There are even items going back
    Jeroen> to 2002 and 2003, which is three years ago as listed on
    Jeroen> http://www.rfc-editor.org/queue.html

  I was in the queue for a long time. Over a year. I'd have to look to
figure out exactly how long.

  The biggest problem is that one goes to the back of the queue if the
RFC editor asks for changes. Those changes can take 1 day, or 1
month.  If the author can turn around in 1 day, then there are
advantages to having the RFC-editor work on it immediately, because it
will be fresh for them.

    Jeroen> A related question is the copyright/expiry statement as
    Jeroen> these are set to 6 months, while the document is in the
    Jeroen> queue it can expire. How is this handled?

  They are supposed to be held -- not expired.
  In the past, the system wasn't perfect, and things did expire. I think
that the ID tracker solved that problem.

- -- 
]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] [email-masked]      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [

    "The Microsoft _Get the Facts CD_ does not work on Linux." - orospakr

---

From: Michael Richardson <[email-masked]>
Date: Thu, 08 Jun 2006 10:45:03 -0400

>>>>> "Keith" == Keith Moore <[email-masked]> writes:
    Keith> I find myself thinking that the RFC Editor needs to be able
    Keith> to ask authors to supply additional information to support
    Keith> their documents, including: endorsement from recognized
    Keith> community members or well-known authorities, detailed
    Keith> analysis of one or more aspects of the protocol, etc.  The
    Keith> point here is to get the RFC Editor out of the business of
    Keith> trying to decide whether a protocol is technically sound.
    Keith> The author should provide a convincing argument (if not
    Keith> outright evidence) that the protocol is sound.

  You are asking for a review process. 
  I don't have a problem with that.

    Keith> I also find myself thinking that the RFC Editor should avoid
    Keith> trying to maintain a complicated state machine for documents.

  When do they maintain it? 

  It would be nice if we had a standard way to present state machines.

- --

---

From: Joe Touch <[email-masked]>
Date: Thu, 08 Jun 2006 08:06:07 -0700

>     Keith> outright evidence) that the protocol is sound.

I wasn't under the impression that technical soundness was a requirement.

I had thought it was more about:

	- not obviously unsound (which is quite different)

	- not obviously incomplete

	- not obviously dangerous

	- not obviously interfering with existing standards

Examples include protocols that ask for a handful of ports up front,
rather than demuxing within the protocol; protocols that have 8 current
variants and have only a 3-bit field (or worse, a 2-bit one), etc. They
also (IMO) should include protocols that lack a version number field
early in the control channel data unit.

Then there is the IETF cross-review, which is triggered by the check:

	- not obviously interfering with IETF process

That's NOT a real review, endorsement by recognized community members,
detailed analysis, or soundness test.

It is a _cursory_ review. If we could distribute the load for that,
that'd be fine. Or perhaps provide a template or checklist for protocols
or protocol mods. But this is not the venue for 'peer reviewed only'
publications; one of the benefits of the Individual track is the lack of
that level of peer review, IMO.

Joe

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 8 Jun 2006 10:38:35 -0700

>take for a submission to be looked at?

draft-klensin-rfc-independent could give an average expected time, 
but it might be better to state maximum suggested times. For example, 
it would be reasonable to say that the step in section 4.3 should 
happen within a month of the step in section 4.2. The step in section 
4.5 maybe should start within a month of the step in section 4.3, and 
should last no longer than six months.

>6 months, while the document is in the queue it can expire. How is this

That is a good topic for the document to cover, particularly given 
that the entire time could easily take more than six months. Maybe 
add a second paragraph to section 4.3:

    At the time that the initial review is begun, the RFC Editor will
    ask the Internet-Drafts Administrator to prevent the Internet
    Draft from expiring.

--Paul Hoffman, Director
--VPN Consortium

---

From: Keith Moore <[email-masked]>
Date: Thu, 8 Jun 2006 13:44:38 -0400

> I wasn't under the impression that technical soundness was a requirement.

IMHO, it is.  I won't claim that it's always possible to objectively determine whether
an idea is technically sound, but I do believe that to publish unsound ideas dilutes
the value of the RFC series and harms IETF's reputation.  Even if strictly speaking,
those RFCs were not produced by IETF and should not reflect on IETF, the public
has a hard time making that distinction.

So I think the bar needs to be higher than "not obviously unsound".

Keith

---

From: Craig Partridge <[email-masked]>
Date: Thu, 08 Jun 2006 13:44:49 -0400

My experience watching the RFC Editorial Board reviews go by is that
the bar is much higher than "not obviously unsound."

Craig

---

From: Leslie Daigle <[email-masked]>
Date: Thu, 08 Jun 2006 13:51:18 -0400

So -- this list is starting from the premise that Independent
Submissions are a real part of the RFC Editor job, and therefore
will be handled properly by the RFC Editor.  WRT queue experiences --
I'll observe the RFC Editor knew it had queue problems and
has been working with the IASA to get support and get that
cleared -- things *are* improving.

Therefore, I believe this list should get back to discussing
the policy for managing the approval of independent submissions,
in a way that respects the intention of that document stream [1]
and yet fits in with the others.

As previously noted -- draft-klensin-rfc-editor has had a lot
of thought put into it, so let's start from that.

Leslie.
[1]  draft-iab-rfc-editor talks about the "streams" and how
they fit together;  probably necessary reading for this
discussion.

Jeroen Massar wrote:
> ------------------------------------------------------------------------
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 08 Jun 2006 11:02:03 -0700

> So I think the bar needs to be higher than "not obviously unsound".

1. I don't think it has been in the past. I.e., how 'sound' does a
proposal need to be? How well developed? How much consensus. And don't
we have a process for that already - the IETF and IRTF WGs?

2. I don't agree that's appropriate for the future. IMO, that's what the
standards-track is for.

Individual submissions have always been available as a path for the
expression of alternate - and sometimes unpopular - ideas. Shutting down
that path shuts out ideas.

I'd prefer to see us put more effort into soundness of WG-based
documents instead.

Joe

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 08 Jun 2006 14:26:02 -0400

I think I probably agree with Joe, although I find the wording confusing.
Soundness (whether "not obviously unsound", or "clearly sound" ought 
to be orthogonal to popularity.  it is indeed the point of the 
independent submissions track to enable publication of alternative, 
and often unpopular, ideas or approaches.

At the same time, those publications do have to pass a reasonable 
soundness test.  If it does not seem to make technical sense 
(internally at least) then it ought not be published as an RFC.

Yours,
Joel

---

From: Craig Partridge <[email-masked]>
Date: Thu, 08 Jun 2006 14:30:56 -0400

Hi Leslie:

I guess I'm a bit puzzled.

Why was this list created?  A document that explains an (existing) process
should not (I hope) automatically cause a mailing list to be created.
Yet that seems to be the tone of the attached note.

So what caused this list to appear?  Are there issues that need
fruitful discussion/airing?

Thanks!

Craig

---

From: Keith Moore <[email-masked]>
Date: Thu, 08 Jun 2006 14:42:43 -0400

> we have a process for that already - the IETF and IRTF WGs?

the RFC editor shouldn't require community consensus for ISDs - if the 
proposal is sound and can get consensus it should be publishable by 
IETF.  (IRTF doesn't try to get community consensus either)

I don't think something published as an ISD necessarily needs to be 
complete, but whatever degree of completeness is there needs support or 
justification.

> that path shuts out ideas.

unpopular but well-thought-out ideas are fine.  stupidity isn't.

> I'd prefer to see us put more effort into soundness of WG-based

all well and good, but not within the scope of this discussion.

Keith

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jun 2006 14:57:37 -0400

Comment as editor:

	I am willing to put anything into this document the
	community decides is necessary.

The comments below are strictly as an individual who has some
experience with independent submissions and as a member of the
current RFC Editorial Board.

--On Thursday, 08 June, 2006 10:38 -0700 Paul Hoffman
<[email-masked]> wrote:

> draft-klensin-rfc-independent could give an average expected
> time, but it might be better to state maximum suggested times.
> For example, it would be reasonable to say that the step in
> section 4.3 should happen within a month of the step in
> section 4.2. The step in section 4.5 maybe should start within
> a month of the step in section 4.3, and should last no longer
> than six months.

I don't know how many of you as readers have participated in the
review process for professional journals, either as reviewers or
as authors, but, when timing depends critically on volunteer
reviews, targets are fine, but large variances are inevitable,
especially when documents may differ significantly in length and
complexity.  So words would be welcome, but we should be sure
that we don't specify something that cannot be realized within
available resources lest we force the RFC Editor to reject
anything that cannot be handled within guidelines.

>> A related question is the copyright/expiry statement as these
>> are set to 6 months, while the document is in the queue it
>> can expire. How is this handled?
> 
> That is a good topic for the document to cover, particularly
> given that the entire time could easily take more than six
> months. Maybe add a second paragraph to section 4.3:
> 

Of course, that is what happens now (or is supposed to -- in the
last year or two, there have been screwups, but my impression is
that most of them have been on the secretariat side which is
something this draft can't fix).  

More important and to the point, I'm very concerned about
getting into a situation in which we are micromanaging by
specification and anticipation.  At some level, it would be
plausible to put a statement in the document that said "the RFC
Editor will work with the IAD to develop procedures such that
silly states are avoided" and leave it at that.  Then one has to
question whether that statement is needed or just reflects an
expectation of good sense.

     john

---

From: Michael Richardson <[email-masked]>
Date: Thu, 08 Jun 2006 15:35:10 -0400

I was also a bit confused by Leslie's note.

Queue experiences are important --- because not all hangups are the RFC
editor's fault.   Some of them are due to the RFC editor asking for
other review, and not getting it.

I also think that there is a stream which we think exists, but maybe
does not:
	- Bert's bad ideas stream
	i.e: in some cases, documents which a WG has decided are out
	of scope, wrong, etc. but that should be published.

- --

---

From: Michael Richardson <[email-masked]>
Date: Thu, 08 Jun 2006 15:41:09 -0400

>>>>> "John" == John C Klensin <[email-masked]> writes:
    John> Please read draft-klensin-rfc-independent... It covers this
    John> issue and the RFC Editor's right to say "no" at any time and
    John> for any reason.  There are safeguards, but the author

  I'm concerned about section 4.8 --- where the IESG reviews.

  Specifically --- workload on IESG, and how that can affect the
queue-period. 

  I'm concerned because it seems at odds with the concept that these are
independant submissions.  I don't have a problem if the RFC-editor
decides that some document might be need expert review, and that the
appropriate area director is the right person. That's already provided
for. 

  I know that this kind of what happens now. How many documents do the
IESG review in this way? All of them? None of them? 10%? 80%?

- --

---

From: Leslie Daigle <[email-masked]>
Date: Thu, 08 Jun 2006 15:56:16 -0400

Well, hopefully everyone here has had some exposure to
the larger discussion of which this is part -- I'll include
references to some of the relevant mail messages at the
end, against the possibility people have not.

Then, the purpose of discussion at this time is to ensure
we have a community consensus document describing the independent
submissions (approval) process.  Since "community" here is
broader than "IETF community", yes, a new list was needed.

WRT draft-klensin-rfc-independent -- that's a document describing
something *close* to the existing process. Are people happy
with its representation of current reality & proposals for
modifications?

Leslie.

References
----------

Latest version of "timeline" for "RFC Editor & 2006", which
includes overall context:

http://www1.ietf.org/mail-archive/web/ietf-announce/current/msg02524.html

General announcement of this list:

http://www1.ietf.org/mail-archive/web/ietf-announce/current/msg02583.html

IAB umbrella document:

http://www.ietf.org/internet-drafts/draft-iab-rfc-editor-00.txt

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 8 Jun 2006 12:59:22 -0700

>anything that cannot be handled within guidelines.

Completely agree. "should start" and "should last" should not be 
taken to mean "if it doesn't happen within this period of time, the 
document is automatically published". But I don't think anyone could 
take it that way.

>something this draft can't fix).

Then the action should be documented in this document.

>expectation of good sense.

The reason it should be in this document is so that it is clear what 
the RFC Editor is supposed to do, and when. If the IAD wants to 
juggle who the RFC Editor talks to, that's fine, and will not require 
a change to this document.

--Paul Hoffman, Director
--VPN Consortium

---

From: Joe Touch <[email-masked]>
Date: Thu, 08 Jun 2006 13:27:13 -0700

Keith Moore wrote:
...
> unpopular but well-thought-out ideas are fine.  stupidity isn't.

Agreed; that's why I said the negative 'not clearly X' rather than the
positive. That ought to catch silly/stupid/dangerous, but be as
inclusive as possible (be generous in what we receive ;-)

> all well and good, but not within the scope of this discussion.

IMO, it is - albeit on the border. It impacts how much effort we can
expect from competent reviewers, and whether they should be distracted
from IETF-level reviews and cross-area reviews to participate in vetting
ISDs.

Joe

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jun 2006 15:55:56 -0400

--On Thursday, 08 June, 2006 15:41 -0400 Michael Richardson
<[email-masked]> wrote:

>     John> Please read draft-klensin-rfc-independent... It
> covers this     John> issue and the RFC Editor's right to say
> "no" at any time and     John> for any reason.  There are
> safeguards, but the author
> 
>   Specifically --- workload on IESG, and how that can affect
> the queue-period. 
> 
>   I'm concerned because it seems at odds with the concept that
> these are independant submissions.  I don't have a problem if
> the RFC-editor decides that some document might be need expert
> review, and that the appropriate area director is the right
> person. That's already provided for. 
> 
>   I know that this kind of what happens now. How many
> documents do the IESG review in this way? All of them? None of
> them? 10%? 80%?

All of them.  The IESG concluded some years ago (circa RFC 2026
or earlier) that it needed to be able to check every independent
submission as a potential end-run of IETF activities and that
only the IESG had the knowledge and perspective needed to make
that check.  The current model is more or less as described in
RFC 3932, which is probably required reading for anyone
participating in this discussion.

And, yes, sometimes there are IESG-induced delays in reviewing
that slow down RFC Editor functions.

The other thing that is beyond the RFC Editor's control is that
sometimes documents come along that it is very hard to review or
get reviewed in a timely fashion.  As I indicated in another
note, one does not want to create a situation in which the RFC
Editor has an incentive to reject documents for that reason.

Some statistics on turnaround time at the various stages of the
process, frequency with which the IESG does not produce an
answer within the agreed-upon timeout period, etc., can be
teased out of the data in the RFC Editor reports to the IAB
plenaries.  Probably more data would be of help to this
discussion, but I don't know if it is readily available.

        john

---

From: Keith Moore <[email-masked]>
Date: Thu, 8 Jun 2006 16:55:31 -0400

> inclusive as possible (be generous in what we receive ;-)

I don't want the evaluation to be based on the orthodoxy of the idea,
I want it to be based on how well the document is written and how well
the idea is supported/defended.   I'd much prefer that the criteria
be defined in positive terms (e.g. "well supported") rather than negative
terms ("not clearly brain-damaged").   Even orthodox ideas need to be
well-supported.