Subject: Proposed IESG Statement on the Conclusion of Experiments

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 19 Apr 2012 21:31:14 +0100

All,

The IESG has been discussing how to tidy up after Experimental RFCs.

We have developed the following draft IESG statement. This does not 
represent a change in process, and continues to value Experimental RFCs
as an important part of the IETF process. It does, however, seek to 
encourage documentation of the conclusion of experiments.

We are aware that there may be other discussion points around 
Experimental RFCs, and we would like to discuss these, but we also
believe that there is merit in making small, incremental improvements.

The IESG would welcome your thoughts on this draft before they approve
the final text on April 26th.

Thanks,
Adrian

=============

IESG Statement on Conclusion of IETF Experiments

Experiments are an established and valuable part of the IETF process.
A number of core Internet protocols were first published as Experimental
RFCs while the community gathered experience and carefully investigated
the consequences of deploying new mechanisms within the Internet.

In the case where an experiment leads on to the development of a      
Standards Track RFC documenting a protocol, the new RFC obsoletes the 
old Experimental RFC and there is a clear conclusion to the experiment.

However, many experiments do not lead to the development of Standards
Track RFCs. Instead, the work may be abandoned through lack of interest
or because important lessons have been learned.

It is currently hard to distinguish between an experiment that is still
being investigated, and an old experiment that has ceased to be of
interest to the community. In both cases an Experimental RFC exists in
the repository and newcomers might easily be misled into thinking that
it would be helpful to conduct more research into an abandoned
experiment.

In view of this, the original proponents of experiments (that is, 
authors of Experimental RFCs, and Working Groups that requested the
publication of Experimental RFCs) are strongly encouraged to document
the termination of experiments that do not result in subsequent
Standards Track work by publishing an Informational RFC that:

- very briefly describes the results of the experiment

- obsoletes the Experimental RFC

- if appropriate, deprecate any IANA code points allocated for the 
  experiment

- may request that the Experimental RFC is moved to Historic status.

If there is no energy in the community for the producing such an
Informational RFC, if the authors have moved on to other things, or if
the Working Group has been closed down, Area Directors should author or
seek volunteers to author such an Informational RFC.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 19 Apr 2012 22:38:38 +0200

Hi Adrian,

I do not support such a view, and it is not supported in a plain reading
of RFC 2026.  What's more, it's not how researchers work.  Researchers
naturally move on.  If we are looking to further push researchers away
from the IRTF, this is a good way to do it.

Whether or not an experiment is active is also not how research works. 
Sometimes work is taken up after years of gaps.  ILNP is a good example
of this.  Had 8+8 been published in a timely fashion as an experimental
RFC, we would have had obsoleted it, only to then unobsolete it?

Eliot

---

From: Scott O Bradner <[email-masked]>
Date: Thu, 19 Apr 2012 16:48:27 -0400

encouraging a report is fine

retracting the code points seems to add more confusion than it is worth unless the
code space is very tight

and I see no reason to obsolete the experimental rfc or move it to historic status unless the report is
that some bad thing happens when you try it out - updating the old rfc is fine

and I agree with Elliot about the nature of research - it is very common to not
reach a conclusion that something is bad (as in bad for the net) - and that is the
only case where I think that an experiment should be flagged as a don't go there situation

Scott

---

From: Dave Crocker <[email-masked]>
Date: Thu, 19 Apr 2012 14:00:06 -0700 (PDT)

+1

/d

--
Dave Crocker
bbiw.net

via mobile

-----Original Message-----
From: Scott O Bradner <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked]
Sent: Thu, 19 Apr 2012 1:48 PM
Subject: Re: Proposed IESG Statement on the Conclusion of Experiments

Scott

---

From: Yoav Nir <[email-masked]>
Date: Fri, 20 Apr 2012 00:17:46 +0300

RFC 2026 says this about Experimental RFCs:

   The "Experimental" designation typically denotes a specification that
   is part of some research or development effort.

However, I do not believe that this is still typical. Authors come up with ideas that they think are useful. If when the documents are ready for publication it is still not clear whether there are enough implementers convinced by the use case (some definition of running code), they are encouraged to publish them as experimental rather than proposed standard.

I am the author of two Experimental RFCs. In both cases I was going for standards track, and in both cased an AD was the document shepherd and he told me to go for Experimental. 

The first of these (4478) has three independent, interoperable implementations in shipping code, although I don't know if it's used much. I am not aware of any implementations of the other one (6023), although there may be some. When I asked an AD about promoting 4478 to standards track (seeing as we have all this running code) his response was "why bother?" (these are three different ADs in this story...)

Neither of these can really be called an "experiment" except in the sense of "let's see if the Internet needs this specification". They may be used more later than they are now, but they're in no way stopping. I don't see why we should waste precious cycles on maintaining old Experimental RFCs. Either they're in use or they're not, and a search engine can usually reveal this.

It may be a good idea to have former authors or ADs or volunteers write a paragraph that can be displayed on, say, the tools page for the RFC, along the lines of "In seven years, this RFC has seen three independent implementations, but it is not in wide use", or "the protocol in this RFC is used everywhere on the web, and this could be progressed to PS if we had the energy."

I don't think we should waste our energy by creating an RFC for each of those experimental RFCs.

Yoav

On Apr 19, 2012, at 11:31 PM, Adrian Farrel wrote:

> Scanned by Check Point Total Security Gateway.

---

From: David Conrad <[email-masked]>
Date: Thu, 19 Apr 2012 14:17:44 -0700

Hi,

> encouraging a report is fine 

Agreed.

> retracting the code points seems to add more confusion than it is worth unless the code space is very tight

Disagree.  From my experience at IANA, trying to figure out who to contact to remove a code point gets harder the longer the code points are not being used.  Unless the code space is unlimited, I'd argue that you want to deallocate as soon as an experiment is over.  I'd even go so far as to say that the original proposal for experimental code points should have explicit revocation dates (which can, of course, be refreshed similarly to IDs).

> and I see no reason to obsolete the experimental rfc or move it to historic status unless the report is that some bad thing happens when you try it out - updating the old rfc is fine

Having been involved with RFC 6563, I think in general it is quite useful to signal "hey, you really don't want to implement this".  If this can be done by updating the old RFC, that's fine.

> and I agree with Elliot about the nature of research - it is very common to not reach a conclusion that something is bad (as in bad for the net) - and that is the only case where I think that an experiment should be flagged as a don't go there situation

Agreed, with the proviso that limited resources (whether they are scarce or not) should be reclaimed.

Regards,
-drc

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 20 Apr 2012 00:19:57 +0200

On Apr 19, 2012, at 23:17, David Conrad wrote:

> RFC 6563

But that's the point.  RFC 6563 is about documenting a failed experiment.
That's where the proposed approach actually helps.
(Retracting the A6 code point, maybe not so much :-)

Many other experimental RFCs neither fail nor succeed.

Between November 2004 and November 2009, I could not have said whether RFC 3940 "failed" or "succeeded".  It just was there, and people played with it.  It didn't take over the world, and it didn't destroy the Internet either.

When would the IESG have cut in under the new policy?

(RFC3940 finally was superseded by standards-track RFC 5740 in November 2009.)

Now imagine an over-eager AD somewhere in these 5 years, wanting to clean up their backyard.

Grüße, Carsten

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 19 Apr 2012 18:27:45 -0400

Hi Scott,

Comments inline.....

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Scott O Bradner
> Sent: Thursday, April 19, 2012 4:48 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]

We are talking about deprecating, not reusing the code point. There is currently a draft in the works that deprecates a few ICMPv4 code points. This draft illustrates the benefits of deprecation. Namely,

- operators have a smaller decision to make when deciding whether to filter the deprecated ICMP message
- if anybody is still writing ICMPv4 software, they don't have to fiddle with the deprecated messages

Conversely, do you see any benefit in not deprecating ICMPv4 message #31. (RFC 1475 reserves this code for IPv7.)

> 
> and I see no reason to obsolete the experimental rfc or move it to
> historic status unless the report is that some bad thing happens when
> you try it out - updating the old rfc is fine
>

I think that this is a case-by-case judgment call. In some cases (e.g., RFC 1475), the experiment is clearly over. IMO, allowing RFC 1475 to retain EXPERIMENTAL status detracts from the credibility of current experiments that share the label.

                                                   Ron

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 19 Apr 2012 22:40:40 -0400

> However, I do not believe that this is still typical. Authors come up with ideas that they think are useful. If when the documents are ready for publication it is still not clear whether there are…

Yes, and surely what we want is that we continue to mislabel as many
RFCs as possible in this way.  We want to make our own contribution to
the gradual reduction of all words to mean everything else, so that we
can all communicate exclusively by mumbling, "Whatever."  Now that the
AP Stylebook has given up on "hopefully" and Oxford is playing with
the comma, anything goes.
 
> Neither of these can really be called an "experiment" except in the
> sense of "let's see if the Internet needs this specification". 

Yes, and that's a stupid meaning of the term in this case, because
quite frankly all the rest of the RFCs are also experiments in this
sense (and many of them have delivered the clarion answer, "Don't need
it at all!"  When was the last time you used BEEP?)

I don't know what I think about the proposal overall, but I will say
that one thing definitely in its favour is that it would discourage
those who think that "Experimental" is the Standards Track Consolation
Prize Track.  I'd be wholeheartedly in its favour, except that the old
"2 year review" standards track process didn't work, so we ditched
it.  I don't see how recycling the approach with Experimental
documents is any more likely to succeed.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Apr 2012 08:19:53 +0100

On 2012-04-19 23:27, Ronald Bonica wrote:
...
> I think that this is a case-by-case judgment call. In some cases (e.g., RFC 1475), the experiment is clearly over. IMO, allowing RFC 1475 to retain EXPERIMENTAL status detracts from the credibility…

I agree that it is case by case, so I don't really see the value in the
IESG statement. If it's appropriate to write an experiment-terminating
RFC, do so; if it's inappropriate, don't bother. That doesn't need
any new legislation.

    Brian

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 20 Apr 2012 07:31:20 +0000

Hi,

> encourage documentation of the conclusion of experiments.

any IESG statement would only cover Experimental RFCs on the IETF Stream, right?

Lars

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 20 Apr 2012 08:24:09 +0000

Hi,

> RFC, we would have had obsoleted it, only to then unobsolete it?

I agree with Eliot. I think the IESG is interpreting things into 2026 that aren't there. Not all Experimental RFCs describe what - with my research hat on - I would call a scientific experiment. Actually, I can't come up with single one that does.

Experimental RFCs specify protocols and protocol extensions *for experimentation*, just as Standards Track RFCs specify protocols and protocol extensions for widespread implementation and deployment. But they do *not* (usually) describe the actual experiments that are to be performed, and I know of no one that would go so far as discuss a timeline for any such experiments. The assumption that an Experimental RFC always describes an actual scientific experiment is simply wrong, and since this assumption underlies the proposed IESG statement, it renders it ineffectual at best and actually harmful at worst.

Experimental RFCs get published for all kinds of reasons. On the IETF side, this may range from actual proposals for protocol evolution for the community to play with, to things that failed to get Standards Track consensus but that a WG didn't feel like abandoning outright, and some WGs use them as a pre-PS step. The proposed IESG statement completely misses that reality.

And obviously, Experimental RFCs exist on other Streams. In terms of whether any IESG statement would even apply to IRTF and Independent Stream RFCs, that is not at all clear to me. In any event, it would have been good to CC the IRSG, since they are not included on the wgchairs list. If the IESG believes that their statement would extend to RFCs on other Streams, they should REALLY have engaged us much earlier, rather than with a six-day deadline for comments.

>> represent a change in process,

Actually, this *does* represent a change in process. It implies that any Experimental RFC must only describe an actual scientific experiment, so that later on it can be judged whether that experiment has succeeded or failed, and the outcome recorded. This is a much more limited view of what Experimental RFCs are than I believe 2026 states.

It also requires the proponents of an Experimental RFC to also sign up for writing another future document describing the "results." (Which per above, I believe doesn't make sense for the cast majority of Experimental RFCs, because they do not describe scientific experiments.)

>> and continues to value Experimental RFCs

This is simply much too soon.

>> Experiments are an established and valuable part of the IETF process.

Actually, I would claim they are not. Experimental *RFCs* are, and IETF participants (and others) may perform experiments with IETF protocols and proposed extensions (whether they are described in IDs, or RFCs of whatever status), but the IETF itself does not do experiments.

>> old Experimental RFC and there is a clear conclusion to the experiment.

It's not so simple. Consider for example TCP F-RTO. The Experimental RFC is 4138. It defines F-RTO for both TCP and SCTP. TCP F-RTO was moved to PS in 5682, but 4138 was not obsoleted, because the SCTP F-RTO is still considered an experiment.

4128 was published in 2005. 5682 was published in 2009. So clearly SCTP F-RTO has failed the "experiment"? Not really. With RTCWEB strongly considering SCTP as a transport framework for the data channel, it's actually very likely that SCTP F-RTO will move to PS in a few years too. So if we had declared the experiment closed, would we have done good or harm?

>> or because important lessons have been learned.

Or, nobody feels a great need to do the work to republish an Experimental RFC as PS, although it is widely implemented. Again, 4138 was Experimental for four years, although all major stacks implemented and enabled it by default. Why took it so long? Because the community had other things to work on and the original document was fine.

>> interest to the community.

Very true. That is why it is very hard to come up with any generic rules here, as this IESG statement is attempting to do. (And never mind that many Experimental RFCs don't actually describe experiments.)

>> In both cases an Experimental RFC exists in

An abandoned experiment is not a failed experiment. Old ideas aren't necessarily bad ideas, and many IETF technologies that lie dormant for a long time all of a sudden gain traction. (Example: SCTP in RTCWEB.)

>> - may request that the Experimental RFC is moved to Historic status.

So is the intent here RFC database and IANA registry housekeeping? In that case, focusing on Experimental RFCs is simply misguided. There are many more standards-track RFCs that have "failed" and would also need to be obsoleted and have their codepoints reclaimed. 

And unlike for Experimental RFCs, we actually do have 2026 text that talks about standards-track maturity, etc. 

>> seek volunteers to author such an Informational RFC.

If I am counting things correctly, almost 1500 Experimental RFCs have been published so far, many by working groups that no longer exist. That's a lot of RFCs to write by volunteers (ADs really should have better things to do...) for little gain.

Lars

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 20 Apr 2012 09:24:15 +0100

Hi Lars,

I was just typing a similar response to Eliot and Scott.

Clearly the IESG does not have authority over the IRTF stream.

I had thought this context ant the limitation to the IETF stream was clear in
the initial blurb ("Experiments are an established and valuable part of the IETF
process.") but obviously I was wrong :-)

We will add clarification of this point to the statement.

Thanks,
Adrian

> From: Eggert, Lars [mailto:[email-masked]]
> Sent: 20 April 2012 08:31
> To: <[email-masked]> 
> Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 20 Apr 2012 10:03:46 +0100

Hi, 

I sort of agree that no new legislation is needed, and I don't read the statement as legislation.

But it is clear that, with the exception of promotion of Experimental RFCs onto the Standards Track, this function has not been happening. It makes sense (I think) to set expectations.

Adrian

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: 20 April 2012 08:20
> To: Ronald Bonica
> Cc: Scott O Bradner; [email-masked]; [email-masked]; [email-masked];
> [email-masked]
> > I think that this is a case-by-case judgment call. In some cases (e.g., RFC 1475),
> the experiment is clearly over. IMO, allowing RFC 1475 to retain EXPERIMENTAL
> status detracts from the credibility of current experiments that share the label.
>

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 20 Apr 2012 13:36:48 +0000

What about the idea of requiring new Experimental documents to include text that indicates when the experiment is to be considered completed absent new work on it?  Essentially, the document declares a date by which the experiment is considered concluded, and code points automatically deprecated, and the document itself goes to Historic status, unless some other document action updates the deadline or moves the work to the Standards Track.

Keeping track of such dates to trigger these actions by default becomes a new burden for the Secretariat, I imagine.  I don't know what that cost would be compared to the "cost" of experiments currently languishing in this unresolved state.

To Eliot's point, work that is resumed much later could always restore document and code point status and declare new drop-dead dates when it becomes interesting again.

-MSK

---

From: Ned Freed <[email-masked]>
Date: Fri, 20 Apr 2012 00:33:08 -0700 (PDT)

> RFC, do so; if it's inappropriate, don't bother. That doesn't need

While I agree 100% that this needs to be handled on a case-by-case basis, I
also think that it's an issue that always needs to be addressed when an
experiment ends. So some sort of statement saying that the handling of any
allocations associated with the experiment must be resolved when the experiment
ends might be appropriate. Or not - this could just be something to add to some
checklist somewhere. (We do have checklists for this sort of thing, right?)

				Ned

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 20 Apr 2012 09:42:31 +0200

Hi Brian,

My personal observation is that some folks are just too afraid about 
success of some 12 experimental RFCs to be published soon and are 
already actively seeking a formal way to kill them in the not too 
distant future.

If the goal here is really about clean-up of dead experiments - I would 
highly recommend to add a clear definition to the mentioned IESG 
Statement (if at all needed) that this is applicable only to those 
experiments which are no longer being actively tested in any way.

If anyone on the respective list would state that he is still testing 
given experimental RFC for as long as it takes I am against moving such 
RFC to historic status. If no one would care to respond within min 4-6 
weeks period I see no reason to block corresponding allocations.

Best regards,
R.

> On 2012-04-19 23:27, Ronald Bonica wrote: ...
>> I think that this is a case-by-case judgment call. In some cases
>> (e.g., RFC 1475), the experiment is clearly over. IMO, allowing RFC
>> 1475 to retain EXPERIMENTAL status detracts from the credibility of
>> current experiments that share the label.
>
> I agree that it is case by case, so I don't really see the value in
> the IESG statement. If it's appropriate to write an
> experiment-terminating RFC, do so; if it's inappropriate, don't
> bother. That doesn't need any new legislation.
>
> Brian

---

From: John Leslie <[email-masked]>
Date: Fri, 20 Apr 2012 07:38:56 -0400

Adrian Farrel <[email-masked]> wrote:
> [Lars Eggert wrote:]

   The IESG (IMHO) feels it has a responsibility to manage Experiments
it starts. This is evidenced by the care they take to craft a path by
which Experimental RFCs can be backed out. "Tidy up after" may not quite
capture their intent here. They want results of experiments to be
documented, and they want experiments they start to conclude.

>>> encourage documentation of the conclusion of experiments.

   This statement captures the IESG intent better, though it omits to
state "experiments the IESG is responsible for".

> Clearly the IESG does not have authority over the IRTF stream.

   Hopefully this clarification remedies that omission.

====

   I don't really have an opinion on how the IESG should manage
IETF-stream experiments.

   Suggestions on how to manage IESG-stream experiments would be helpful
here. I would hope that opinions on how IRTF-stream experiments should
be managed will find a different thread, ideally in a different list.

--
John Leslie <[email-masked]>

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 20 Apr 2012 16:12:33 +0100

> What about the idea of requiring new Experimental documents to include text that indicates when the experiment is to be considered completed absent new work on it?  Essentially, the document declar…
If you factor in the historic success rate that engineers typically have 
in predicting s/w development schedules, I would expect that the overrun 
rate on predicted end exp dates would be  close to 100%, even after 
several extensions.

- Stewart

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Apr 2012 16:23:48 +0100

On 2012-04-20 16:12, Stewart Bryant wrote:
>> What about the idea of requiring new Experimental documents to include
>> text that indicates when the experiment is to be considered completed
>> absent new work on it?  Essentially, the document declares a date by
>> which the experiment is considered concluded, and code points
>> automatically deprecated, and the document itself goes to Historic
>> status, unless some other document action updates the deadline or
>> moves the work to the Standards Track.

Exactly. This whole discussion seems to be about over-engineering a
small corner of the IETF process that isn't a particular source
of practical problems anyway, afaik.

So, the standard question: what's the problem that needs solving here?

   Brian

---

From: Dave Crocker <[email-masked]>
Date: Fri, 20 Apr 2012 09:15:44 -0700

On 4/20/2012 6:36 AM, Murray S. Kucherawy wrote:
> What about the idea of requiring new Experimental documents to
> include text that indicates when the experiment is to be considered
> completed absent new work on it?  Essentially, the document declares
> a date by which the experiment is considered concluded,
...
> To Eliot's point, work that is resumed much later could always
> restore document and code point status and declare new drop-dead
> dates when it becomes interesting again.

As Brian notes, "dates" are probably unlikely to be useful while also 
imposing additional administrative tracking overhead.

On the other hand, objective performance/achievement criteria --
statements that describe what experiential information is being sought
-- would be useful to encourage.

d/

-- 

-- 
  Dave Crocker
  bbiw.net

---

From: Marc Blanchet <[email-masked]>
Date: Fri, 20 Apr 2012 15:06:02 -0400

I agree with everything Scott wrote. I don't see a good reason for removing code points from a registry, unless very exceptional cases (range is almost full need more space), which could be processed as one-off, not as a regular process.

Marc.

---

From: Melinda Shore <[email-masked]>
Date: Fri, 20 Apr 2012 11:27:25 -0800

> of practical problems anyway, afaik.

Reading the back-and-forth on this I think at this point I feel pretty
strongly that I don't support the proposal.  If someone wants to take
it upon themselves to badger authors of drafts that might be published
as experimental protocols, fine, but even while (heaven knows) I'm a
fan of "process" stuff this really seems overly legalistic and not a
solution for anything that's clearly a problem.

So, no.

Melinda

---

From: Yoav Nir <[email-masked]>
Date: Fri, 20 Apr 2012 23:49:13 +0300

On Apr 20, 2012, at 5:40 AM, Andrew Sullivan wrote:

> RFCs as possible in this way.  

No, I don't believe they are mislabeled, they just don't fit the statement in RFC 2026, and they're not scientific experiments. It gives fair warning to would-be implementers regarding the maturity of the specification. Other implementations may not have implemented this RFC (if it's an extension), or they may solve the same problem with some other mechanism. Or they may even have implemented the same RFC, but we're not sure yet that it's unambiguous enough and error-free enough to produce interoperable implementations.

> We want to make our own contribution to

True. Perhaps if successful standards were advanced form Proposed to something else, there would be less pushback against publishing ideas with little running code as PS. Surely your little idea with no running code can't be in the same category as TLS, right?

---

From: Ronald Bonica <[email-masked]>
Date: Fri, 20 Apr 2012 16:56:54 -0400

Folks,

I would like to clarify a few points regarding this statement. Specifically, 

- It applies only to documents that were published on the IETF stream.
- It does not apply to documents that were published on the IAB, IRTF or Independent streams.
- Reuse of deprecated code points is not a goal of this exercise.
- It does not recommend any change in IETF process.

Today, experiments can be concluded through either of the following actions:

- Obsoleting the EXPERIMENTAL RFC with a Standards Track document
- Publishing an INFORMATIONAL RFC that terminates the experiment and possibly moves the EXPERIMENTAL RFC to HISTORIC

Today, both of these actions require community consensus. Today, both of these actions can be undertaken by anyone. If this IESG statement is published, none of that changes.

                                             Ron

> Adrian Farrel
> Sent: Thursday, April 19, 2012 4:31 PM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: Proposed IESG Statement on the Conclusion of Experiments
> 
> All,
> 
> A number of core Internet protocols were first published as
> Experimental RFCs while the community gathered experience and carefully
> investigated the consequences of deploying new mechanisms within the
> Internet.
>

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 20 Apr 2012 16:05:33 -0700

Hi -

> From: "Ronald Bonica" <[email-masked]>
> To: <[email-masked]>; <[email-masked]>; <[email-masked]>
> Sent: Friday, April 20, 2012 1:56 PM
> Subject: RE: Proposed IESG Statement on the Conclusion of Experiments
....
> If this IESG statement is published, none of that changes.

It might be helpful to say what *would* change upon publication
of this statement, and why the IESG believes it is important to make
that change, because we're obviously not unerstanding the
proposal in the same way.

Randy

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Sat, 21 Apr 2012 02:28:52 +0200

Maybe I am the worst person to respond here but I agree with Brian.

I've always had a problem with the large number of different document types.

My overall view is that essentially market forces decide how a document is taken and implemented, and the IETF process has never had enough responsiveness to follow that in a timely fashion that has any real meaning. This applies to proposed standard --> draft standard --> standard, and it will not be different here.

Changing something from experimental to proposed standard in a process that will probably take 12 months will be unlikely change the number of people implementing and deploying an RFC.

Regards

Keith

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Brian E Carpenter
> Sent: 20 April 2012 16:24
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]

---

From: Melinda Shore <[email-masked]>
Date: Fri, 20 Apr 2012 16:37:30 -0800

On 4/20/12 4:28 PM, DRAGE, Keith (Keith) wrote:
> Changing something from experimental to proposed standard in a
> process that will probably take 12 months will be unlikely change the
> number of people implementing and deploying an RFC.

I'm going to take the liberty of mentioning that I spoke with Ron
earlier today about this.  Basically what he's asking is that there
be no process changes, and, I think, no policy changes, just
that IESG members should be mindful about how to phase experimental
stuff out when it's flopped.

Personally, I think he's correct about both cruft and mindfulness
and suspect that probably nearly everybody agrees with what he's
saying, anyway, but unfortunately it was presented in a form that
made it look like More Process.

Melinda

---

From: Aaron Stone <[email-masked]>
Date: Fri, 20 Apr 2012 18:00:41 -0700

> the final text on April 26th.

If no concluding document has been written, then why wouldn't it be
useful to conduct further research?

It sounds like the main issue is negative results that aren't getting
published. We should encourage people to publish them. If they don't,
then of course someone may come along and revisit the experiment.
How's that any different if the experiment is moved to historical or
not? Old ideas that came before their time are often revisited to
useful outcomes.

If the result of this discussion is the statement below that
officially encourages publishing results, positive or negative, great.
If it's added process or any kind of automatic document expiration,
not great.

Cheers,
Aaron

---

From: Ronald Bonica <[email-masked]>
Date: Sat, 21 Apr 2012 10:44:09 -0400

Hi Randy,

Both you and Brian have rightly noted that the proposed IESG statement changes neither process nor policy. It is intended to direct the community's attention to a problem that is small today, but promises to grow with time.

Please review the list of experimental RFCs at the following URL:

- http://www.rfc-editor.org/categories/rfc-experimental.html

I am willing to bet that you laughed at least once while scanning this list. I got a chuckle from RFC 1475, the IPv7 experiment. But seriously, I identified nearly a dozen RFCs that fell into one of the following categories:

- should have been obsolete by subsequent Standards Track documents, but weren't
- are more than 10 years old and have not gained traction, generally because an alternative approach has gained traction and become widely deployed (e.g., RFC 1393 - Traceroute Using an IP Option).

The proposed IESG statement encourages community members to terminate their own experiments when they have clearly ended. In many cases, the experimenter moves on before the experiment is terminated. In that case, the cleanup chore is left to others.

The proposed IESG statement *does not* seek to expedite the termination of any experiment. Its only goal is to identify experiments that clearly have terminated.

IMO, it is important that we label terminated experiments as such. Otherwise, in years to come, it will become even more difficult to distinguish between active experiments that deserve community attention and old cruft that can safely be ignored.

                                                Ron

> Randy Presuhn
> Sent: Friday, April 20, 2012 7:06 PM
> It might be helpful to say what *would* change upon publication of this
> statement, and why the IESG believes it is important to make that
> change, because we're obviously not unerstanding the proposal in the
> same way.
> 
> Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 21 Apr 2012 08:42:28 -0700

Hi -

> To: "Randy Presuhn" <[email-masked]>; <[email-masked]>; <[email-masked]>; "Brian E Carpenter"
<[email-masked]>
> Sent: Saturday, April 21, 2012 7:44 AM
...
> The proposed IESG statement encourages community members to terminate
> their own experiments when they have clearly ended. In many cases, the
> experimenter moves on before the experiment is terminated. In that case,
> the cleanup chore is left to others.

I think there are some  fundamental problems in this framing of the question.
The use of the phrase "their own" and "clearly" are the ones that concern me.
The problems with "clearly" are obvious, particularly for protocols that have
a long legacy of "just working", regardless of their theoretical shortcomings.
"Their own" assumes a level of "ownership" and, more importantly, *control*
which is rarely true, and in some cases the "owners" may actually have a
conflict of interest leading them to favor premature termination.

> The proposed IESG statement *does not* seek to expedite the termination
> of any experiment. Its only goal is to identify experiments that clearly have terminated.

I do not see any way in which this improves the criteria or process for determining
"clearly", or even the subtler problem of "terminated."  Sometimes the result of
the experiment may be "we don't want to standardize this", but also "in the
aftermath of the experiment, there are a lot of implementations of this, and
insufficient motivation for those who have deployed it to move to something else."

Randy

---

From: Joel jaeggli <[email-masked]>
Date: Sat, 21 Apr 2012 10:01:29 -0700

> retracting the code points seems to add more confusion than it is worth unless the

unless retracting  involves marking it do not use, I too do no see value
in doing that.

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Sun, 22 Apr 2012 13:08:46 +0200

Hi,

Part of the problems with Experimental RFCs which were prompted up also
in this discussion derive from the fact that many of the RFCs labeled as
Experimental do not describe in clear terms the goals of the experiment
that is being proposed. It is hard for somebody to describe the results
of the experiment (even very briefly) if the goals were not specified.
If the IESG decides to go ahead with this statement I suggest that a
strong recommendation is added for documents that aim Experimental as
their Proposed Status to include clear and explicit text about the goals
of the experiment and wherever relevant conditions, restrictions, and
timeframe. 

Regards,

Dan

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Adrian Farrel
> Sent: Thursday, April 19, 2012 11:31 PM
> Experimental

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 22 Apr 2012 14:17:32 +0200

On Apr 22, 2012, at 13:08, Romascanu, Dan (Dan) wrote:

> that is being proposed. 

I never knew that was an objective for an experimental RFC.

(But I have only been contributing to the IETF for 19 years.)

The way that I understood *experimental* RFCs all this time was that they provide interoperability specifications that are *experimental*, i.e., we need further *experience* to validate them (e.g., we don't fully understand yet whether they work as well as we think).

The design of formal *experiments* around these specifications never was a subject, and I would be surprised to do these anywhere in the IETF (maybe in the IRTF, but even that is a long shot for most of these).  More importantly, the *experience* needed for validation may not come out of *experiments* at all.

Please read these two sentences from RFC 2026 again and tell me what I have missed these 19 years.

   The "Experimental" designation typically denotes a specification that
   is part of some research or development effort.  Such a specification
   is published for the general information of the Internet technical
   community and as an archival record of the work, subject only to
   editorial considerations and to verification that there has been
   adequate coordination with the standards process (see below).  

I wouldn't mind if an experimental RFC were more vocal about what kind of experience is missing, motivating *why* it is still experimental.  As was already said in this thread, "experimental" status is often threatened as a cop-out out of a standardization process that some party does not want to complete.  Curbing *that* would be worth some effort.

Grüße, Carsten

PS.: Since I'm not a native speaker, I just looked up again what experimental means.  Yes, experiments may be involved (in submeaning 2), but that's not the main semantics of this word.  Deriving the *need* for an "experiment" from this word strikes me as confused.

experimental |ɪkˈˌspɛrəˈˌmɛn(t)l|
adjective
(of a new invention or product) based on untested ideas or techniques and not yet established or finalized: an experimental drug.
• (of a work of art or an artistic technique) involving a radically new and innovative style: experimental music.
• of or relating to scientific experiments: experimental results.
• archaic based on experience as opposed to authority or conjecture: an experimental knowledge of God.

---

From: Fred Baker <[email-masked]>
Date: Sun, 22 Apr 2012 15:44:26 +0200

On Apr 19, 2012, at 11:17 PM, David Conrad wrote:

> I'd argue that you want to deallocate as soon as an experiment is over.

But the point is that experiments don't necessarily become "over". Some of them in effect become standards, and it would probably be good if there were an easy way to reclassify the experimental RFC as a Proposed Standard. Saying that "nobody" is using a given code point or RFC of any category is very difficult.

---

From: "John G. Scudder" <[email-masked]>
Date: Sun, 22 Apr 2012 07:54:28 -0600

On Apr 22, 2012, at 7:44 AM, Fred Baker wrote:

> Saying that "nobody" is using a given code point or RFC of any category is very difficult.

True enough, although since you didn't give context I'm not sure who you're making that point to, or why. The proposal Adrian originally sent talks about optionally deprecating code points, which while generally related to your comment, isn't the same.

--John

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Mon, 23 Apr 2012 01:12:38 +0200

I wasn't discussing the point about whether there should be a process change or not. Rather, I don't think any process, existing or new, of changing the document type can be done in less than 12 months, and I don't think that such change in status will match any market needs for extra identification of active RFCs worthy of implementation.

Therefore essentially in many respects this discussion will achieve nothing.

Regards

Keith

> From: Melinda Shore [mailto:[email-masked]]
> Sent: 21 April 2012 01:38
> To: DRAGE, Keith (Keith)
> Cc: Brian E Carpenter; [email-masked]; [email-masked]

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 23 Apr 2012 12:28:35 +0200

Hi Carsten,

We are discussing a proposal for an IESG statement that includes the following: 

> In view of this, the original proponents of experiments (that is, authors of Experimental RFCs, and Working Groups that requested the publication of Experimental RFCs) are strongly encouraged to document the termination of experiments that do not result in subsequent Standards Track work by publishing an Informational RFC that:

So the 'experiments' seem to be considered by the IESG as related to Experimental RFCs. 

My point is that if we discuss about 'results of the experiment' and the IESG plans to strongly encourage authors of Experimental RFCs to briefly describe the results, they should also recommend to include in the RFCs in an explicit manner the goals which the results of the experiments are to be measured against. 

Also, if you are quoting RFC 2026 you must have also encountered the following paragraph in section 3.3: 

> (d)  Limited Use:  The TS is considered to be appropriate for use
      only in limited or unique circumstances.  For example, the usage
      of a protocol with the "Experimental" designation should generally
      be limited to those actively involved with the experiment.

So the term 'experiment' already appears in 2026. 

Regards,

Dan

> From: Carsten Bormann [mailto:[email-masked]]
> Sent: Sunday, April 22, 2012 3:18 PM
> To: Romascanu, Dan (Dan)
> The way that I understood *experimental* RFCs all this time was that
> they provide interoperability specifications that are *experimental*,
> i.e., we need further *experience* to validate them (e.g., we don't
> fully understand yet whether they work as well as we think).
> 
> The design of formal *experiments* around these specifications never
> was a subject, and I would be surprised to do these anywhere in the
> IETF (maybe in the IRTF, but even that is a long shot for most of
> these).  More importantly, the *experience* needed for validation may
> not come out of *experiments* at all.
> 
> I wouldn't mind if an experimental RFC were more vocal about what kind
> of experience is missing, motivating *why* it is still experimental.
> As was already said in this thread, "experimental" status is often
> threatened as a cop-out out of a standardization process that some
> party does not want to complete.  Curbing *that* would be worth some
> effort.
> 
> Grüße, Carsten
> 
> PS.: Since I'm not a native speaker, I just looked up again what
> experimental means.  Yes, experiments may be involved (in submeaning
> 2), but that's not the main semantics of this word.  Deriving the
> *need* for an "experiment" from this word strikes me as confused.
> 
> (of a new invention or product) based on untested ideas or techniques
> and not yet established or finalized: an experimental drug.
> • (of a work of art or an artistic technique) involving a radically new
> and innovative style: experimental music.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 23 Apr 2012 14:27:47 +0200

> My point is that if we discuss about 'results of the experiment' and the IESG plans to strongly encourage authors of Experimental RFCs to briefly describe the results, they should also recommend to…

This makes sense if you believe the Experimental RFCs are actually (with respect to time, with respect to the people involved) coupled to any experiments.
Otherwise it's just another layer of red tape, another hoop to jump through.

> So the term 'experiment' already appears in 2026. 

Yes (but note that this is about the "limited use" designation, not about "experimental").  I've wondered about this, too, a long time ago when I started to learn about the IETF and RFC 1310.  More than once this has been explained to me as a euphemism for "stuff should be used only by people that know the stuff, know what they are doing and know what will break and how to fix this".  But of course I didn't record those hallway conversations and don't remember who said that at the time.

Honestly, I would have no idea what I would have written into RFC 3940/3941, except for "it is believed we haven't had enough deployment experience to make this a PS, so please join us in the experiment of deploying this for the applications where it is needed."  

If that sentence is fine with the new procedure, I don't have a problem.

Of course, it would be easy to describe the result of the "experiment": 3940/3941 were upgraded to PS 5 and 4 years later, with some fixes learned from the wider use that the Experimental status (and more time) brought with it.  But, again, I'm not sure what we'd have written if people had wanted us to write up "the experiment" before going to PS.  Maybe a list of fixes (section 10 of RFC 5740)?

I understand there may be grander things going on (like attempting to change the way we use addresses in the Internet) that do justify jumping through those hoops.  But as a general rule, this procedure makes me unhappy.  How about "The IESG may ask for..."?

Grüße, Carsten

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 25 Apr 2012 10:27:09 -0400

I see no value in deallocating code point spaces and a huge amount of
potential harm.

Except at the very lowest levels of the protocol stack (IP and BGP)
there is really no technical need for a namespace that is limited. We
do have some protocols that will come to a crisis some day but there
are plenty of ways that the code space for DNS, TLS etc can be
expanded if we ever need to.

The Internet is driven by innovation and experiment. There are
certainly people who think that the role of this organization is to
protect the Internet from the meddling of people who might not get it
but they are wrong.

Even more wrong is the idea that IANA can actually act as suggested.
IANA only exists by mutual consent. I am happy to register a code
point if there is a reasonable procedure to do so. But if the idea is
to force me to kiss someone's ring then I'll pick a number and ship
the code and let other folk work out the problems later.

This already happens on a significant scale in the wild. SRV code
points being a prime example. There are far more unofficial code
points than recognized ones. Some of them are in standards I wrote at
W3C and OASIS. It would be best if IANA tracked these but I think it
rather unlikely anyone is going to accidentally overwrite the SAML or
the XKMS SRV code points.

It has happened with other registries too. Back in the day there was
an attempt to stop the storage manufacturers using ethernet IDs on
their drives. So the drive manufacturers simply declared a block of
space (Cxxxxxx) as theirs and the IEEE has since been forced to accept
this as a fait accompli. It has happened in UPC codes as well, when
the Europeans decided that the US authority was charging them
ridiculous fees they just extended the code by an octet and set
themselves up as a meta-registry.

The only role of IANA is to help people avoid treading on each other
by accident. If it starts issuing code points that have been 'lightly
used', the value of IANA is degraded. I certainly don't want my
protocol to have to deal with issues that are caused by someone's
'experiment' that has not completely shut down. The only value of
going to IANA rather than choosing one myself is to avoid
contamination with earlier efforts.

The only area where I can see the benefits of re-allocation
outweighing the risks is in IP port assignments but even there I think
the real solution has to be to simply ditch the whole notion of port
numbers and use SRV type approaches for new protocols.

IANA is not a control point for the Internet. A fact that people need
to keep in mind when the ITU attempts to grab control in Dubai later
on this year.

Weakness is strength: The registries are not the control points people
imagine because they only have authority as long as people consent.

-- 
Website: http://hallambaker.com/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 25 Apr 2012 14:24:34 -0400

+1

Deprecating a code point is very different from deallocating it which
implies that it is going to be given out again in the future. Last
thing I want is a used code point.

I agree that avoiding multiple allocations is also a function of IANA,
but this is also something that argues for not being overly strict. If
it is too hard to get an allocation then there will be multiple
unofficial versions. Eventually there may be an attempt to pick one of
them as 'official'.

On Wed, Apr 25, 2012 at 1:46 PM, Ned Freed <[email-masked]> wrote:
> It depends on the size of the space. I completely agree that if the space is
> large - and that's almost always the case - then deallocating is going to be
> somewhere between silly and very damaging.
>
> Deprecacting the use of a code point, OTOH, may make sense even if the space is
> large.
>
> The takeaway here, I think, is that if you're going to "conclude" experiments,
> you need to examine these allocations and do something sensible with them,
> where "sensible" is rarely going to mean "deallocate".
>
> There may not be any technical need, but there are a number of legacy designs
> that were done ... poorly.
>
> Media types are an excellent example of this. The original registration
> procedures were too restrictive so people simply picked names to use. We fixed
> that for "vendor" assignments (fill in a web form) and the registrations
> starting rolling in. (We're now trying to do the same for "standard"
> assignments.) But of course we now have a legacy of unassigned material
> to deal with.
>
> No, not quite. The other important role is (hopefully) to keep people from
> having multiple allocations for the same thing.
>
> Good point.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 25 Apr 2012 14:26:50 -0400

Not arguable in the fashion that you do. You seem to want to signal
disagreement without needing to actually argue a contrary case.

Cutting pieces out of someone's argument to make it look stupid is
itself a stupid trick.

On Wed, Apr 25, 2012 at 12:55 PM, David Conrad <[email-masked]> wrote:
> On Apr 25, 2012, at 7:27 AM, Phillip Hallam-Baker wrote:
>> there is really no technical need for a namespace that is limited.
>
> Arguable, but irrelevant since the reality is that historically many (most?) protocols defined by the IETF to date used fixed length fields implying limitations in the number of identifiers in those namespaces.
>
>> We
> Unfortunately, experience has demonstrated that most implementations of protocols do not handle potential expansion.
>
> You seem to have an odd idea of what is being suggested. However, experience has shown arguing with you is a waste of time so I'll let others engage if they care.
>
>> Weakness is strength
>
>
> And we've always been at war with Eastasia.
>
> Regards,
> -drc
>