Subject: Re: Running code [was Re: [Sipping] Re: [Sip] Thoughts on the Principle of DividingandConquering]

From: Dean Willis <[email-masked]>
Date: Fri, 3 Mar 2006 17:34:37 -0600

On Mar 3, 2006, at 4:21 PM, Henry Sinnreich wrote:

> Marc,
>
>> What about a standardized section in future I-D/RFC that list the  
>> running
>> code that implement this document - like a "Running Code  
>> Considerations",
>> after the "IANA Considerations" and "Security Considerations"  
>> sections.
>
> This is an excellent proposal and I would also add:
>
> Documents that are only proposals or concepts, but are not backed  
> up by
> running code can be very well exchaged as e-mails on the various  
> lists.
> In this way, WG I-D overload would disappear and Dean's problem  
> will be
> solved.
>
> If there is no running code, discussions on e-mail is just fine.
>
> Thanks, Henry
>
>

Lovely idea, but needs to be part of the IETF process, not the SIP  
process. The New Track working group, perhaps?

--
Dean

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 03 Mar 2006 18:49:57 -0500

[Sorry, I couldn't let this go without comment.  I have trimmed the 
CC so the leadership won't receive 3 copies.]

For SIP, maybe the problem space is in a state where that is 
practical (although I doubt it.)

For many other working groups, it would be really strange to require 
running code before accepting a draft for discussion.  For example, 
in ForCES we needed to agree on what the structure of the ForCES 
protocol needed to be.  Requiring people to implement ideas that we 
wanted to discuss would have made it very hard to have clear 
discussion.  Particularly since we had not agreed on the underlying 
transport, so it would have been easy to confuse transport properties 
with application properties.

In general, with new work it is pretty hard to require implementation 
of the early stage ideas.  It can sometimes even be 
counter-productive.  Efforts to provide early implementations of 
alternative management structures seemed to cause those emplementors 
to become wedded to their particularl approach.

I definitely like implementation.  The routing area is in fact 
increasing the emphasis on implementation before 
standardization.  But implementation before I-D seems a mistake.  We 
usually ask folks to write up an I-D on anything they want 
discussed.  And in most working groups, if the topic is on charter 
and the approach looks sane, the document is often made a WG document 
to create a clear trackable state for the work.

Yours,
Joel M. Halpern

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Fri, 03 Mar 2006 16:28:31 -0800

The way I see it, the I-D can start with an empty Running Code section,
like it is often the case with the IANA or Security section. As the
draft evolves, it can keep track of the various implementation of the
I-D in this section. This could be also a good indicator of the interest
in this work.

Joel M. Halpern wrote:
> [Sorry, I couldn't let this go without comment.  I have trimmed the CC
> so the leadership won't receive 3 copies.]
> 
> For SIP, maybe the problem space is in a state where that is practical
> (although I doubt it.)
> 
> running code before accepting a draft for discussion.  For example, in
> ForCES we needed to agree on what the structure of the ForCES protocol
> needed to be.  Requiring people to implement ideas that we wanted to
> discuss would have made it very hard to have clear discussion. 
> Particularly since we had not agreed on the underlying transport, so it
> would have been easy to confuse transport properties with application
> properties.
> 
> In general, with new work it is pretty hard to require implementation of
> the early stage ideas.  It can sometimes even be counter-productive. 
> Efforts to provide early implementations of alternative management
> structures seemed to cause those emplementors to become wedded to their
> particularl approach.
> 
> increasing the emphasis on implementation before standardization.  But
> implementation before I-D seems a mistake.  We usually ask folks to
> write up an I-D on anything they want discussed.  And in most working
> groups, if the topic is on charter and the approach looks sane, the
> document is often made a WG document to create a clear trackable state
> for the work.
> 
> Yours,
> Joel M. Halpern
> 

-- 
Marc Petit-Huguenin
Home: [email-masked]
Work: [email-masked]

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 04 Mar 2006 10:03:50 -0500

Let's try to separate the pieces.
Having implementations as early as possible is clearly a good idea.
It is sometimes practical to have an implementation along with the 
first I-D on a topic.
When it is practical, it is often helpful.

There are many cases where it is not practical.
There are cases where it is even counter productive.
(As far as I can tell, much of the argument over the proposals in the 
OSPF-MANET extensions is because folks are wedded to the proposals 
they have implemented, and they have lovely simulations using their 
code to show why their answer is better.)

Part of my concern here is that for all that we need "bits on the 
wire" to get things to work, it is often counter-productive to argue 
about bits-on-the wire before one agrees on how a protocol ought to behave.
Having an I-D to discuss that describes a particular approach is MUCH 
better than just having an email discussion.
In quite a few working groups I work in, the response to an 
interesting idea is "give us a draft".  That allows the working group 
to understand the idea at a suitable level of detail.  It would be 
really bad if that were only permitted after the person suggesting 
the idea had built an implementation.
It is also the case that implementation can be counter-productive.  I 
can recall a case where a participant in a working group wanted the 
standard to essentially require an all-paths graph 
traversal.  Several of us told him this was a bad idea.  He went and 
implemented and measured it.  He said , roughly ~see, it works and it 
isn't too expensive.~  It was not too expensive in his tests, with 
other limitations, etc.  However, there was no question that it was a bad idea.
Implementation is a good tool.  It is not always the right answer.

I had an implementation of XCAP.  I could not and would not have 
built my implementation if the I-D had not existed.  There was no way 
I could have taken the time of product engineers to build something 
that was not even an I-D, much less an I-D with some traction in the 
working group.  By building it, we learned some things that helped 
shape the work.  I am not arguing that implementation is not a good 
idea.  It is a good idea.  But requiring implementation before the 
first I-D is difficult and often counter-productive.

Let me take this a step further.  I work with a lot of routing area 
activities.  I do not have spare machine sitting around.  I don't 
have a machine sitting around with a router stack with a set of 
routing protocols that I can modify.  Am I supposed to be prohibited 
from writing an I-D on a change to a routing protocol?  I don't know 
if my co-authors on the IDR I-D have implemented yet or not.  Given 
that the first question with regard to the I-D was whether the 
working group thought the problem needed to be addressed, I could 
well understand someone holding off implementation for a bit.

Yours,
Joel M. Halpern

PS: It is not at all clear to me that this proposal would help the 
SIP situation much at all.  Many of the folks have access to stacks, 
and could throw together an implementation of their proposal.  Not a 
commercial grade implementation, but enough to claim 
implementation.  And given the strategic importance of SIP, these 
folks can frequently even get management support for doing the work.

At 09:23 AM 3/4/2006, Henry Sinnreich wrote:
>Joel,
> > of the early stage ideas.
>
>If the authors of an idea do not invest and commit enough into writing code,
>why should they SPAM the WG with I-Ds? Can anyone here stand this flood any
>more?
>
>The idea could still be discussed on e-mail on the list (with a topic title)
>without hijacking the IETF brand name to draw attention to themselves.
>
>The flood of I-Ds that Dean has signaled clearly needs a better control
>mechanism and just legitimizing more I-Ds with more WGs is not the answer.
>
>Maybe there are better self control and stabilization mechanisms to control
>the flood of I-Ds and it would be interesting to discuss various options.
>
>More WGs is not the answer IMHO, a self-regulating preselection mechanism is
>the answer: Auto preselection by obligation and commitment to write code
>first.
>
>Dean writes:
>
> >Lovely idea, but needs to be part of the IETF process, not the SIP process.
>
>Yes, this is the original IETF process, before the _great flood_ swamped us.
>
> >The New Track working group, perhaps?
>Great idea! The New Track will be a good place to experiment for progress.
>
>My two cents. Thanks, Henry
>
>
>
>-----Original Message-----
>From: Marc Petit-Huguenin [mailto:[email-masked]]
>Sent: Friday, March 03, 2006 7:29 PM
>To: Joel M. Halpern
>Cc: [email-masked]; 'sipping'
>Subject: [Sipping] Re: [Sip] Re: Running code
>
>Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
>This list is for NEW development of the application of SIP
>Use [email-masked] for new developments of core SIP

---

From: "Pete Cordell" <[email-masked]>
Date: Sat, 4 Mar 2006 16:04:50 -0000

My 2 cents...

I've been involved in a number of standardisation efforts that have 
attempted to track the standard with running code, and I can't say that 
having running code has helped in any case.

Quite often, initial proposals address a specific issue.  When they go into 
the IETF, the IETF quite rightly generalises the issue, most likely morphing 
it into something completely different and invalidating any code that 
addressed the specific case.

Equally, if you submit a draft, it's quite common for someone to say 'that's 
not the way to do it,' and produce a counter-proposal thus invalidating your 
code.

Writing code also takes a lot of effort.  Testing it in different scenarios 
is even harder.  I personally would rather that effort was put into 
'paper-based' exercises to flesh out (and document) the use-cases and 
identify corner cases etc.

Naturally you want running code the instant the RFC is published.  That's 
not just IETF sense, that's business sense! And for anything non-trivial 
that means code will end up being developed while the proposal is being 
progressed; which is good from the point of view of developing and 
validating the proposal.  Exactly when you start coding, and how much you 
commit to it has to be a judgement call though.

Overall, I feel if someone has identified a useful feature, then them not 
having running code should not be a limitation to its progression.  Equally, 
just because someone has running code shouldn't mean a proposal should be 
progressed.

Also, I don't think drafts progress slowly because their authors are not 
committed to them.  My experience of the problem is that it's hard to get 
debate by third-parties on drafts, which slows progression; possibly because 
they are writing their own drafts!  If the authors (or authors' 
affiliations) are also required to produce running code, that's going to 
reduce the effort that they can put into reviewing third-party drafts even 
more, and I think that will slow things down even more.

Maybe that was 3 cents!

Pete.
--
=============================================
Pete Cordell
Tech-Know-Ware Ltd
                         for XML to C++ data binding visit
                         http://www.tech-know-ware.com/lmx
                         (or http://www.xml2cpp.com)
=============================================

----- Original Message ----- 
From: "Joel M. Halpern" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>; "'Marc Petit-Huguenin'" <[email-masked]>; "'sipping'" 
<[email-masked]>
Sent: Saturday, March 04, 2006 3:03 PM
Subject: RE: [Sipping] Re: [Sip] Re: Running code

> It is sometimes practical to have an implementation along with the first 
> I-D on a topic.
> OSPF-MANET extensions is because folks are wedded to the proposals they 
> have implemented, and they have lovely simulations using their code to 
> show why their answer is better.)
>
> Part of my concern here is that for all that we need "bits on the wire" to 
> get things to work, it is often counter-productive to argue about 
> bits-on-the wire before one agrees on how a protocol ought to behave.
> In quite a few working groups I work in, the response to an interesting 
> idea is "give us a draft".  That allows the working group to understand 
> the idea at a suitable level of detail.  It would be really bad if that 
> were only permitted after the person suggesting the idea had built an 
> implementation.
> It is also the case that implementation can be counter-productive.  I can 
> recall a case where a participant in a working group wanted the standard 
> to essentially require an all-paths graph traversal.  Several of us told 
> him this was a bad idea.  He went and implemented and measured it.  He 
> said , roughly ~see, it works and it isn't too expensive.~  It was not too 
> expensive in his tests, with other limitations, etc.  However, there was 
> no question that it was a bad idea.
> I had an implementation of XCAP.  I could not and would not have built my 
> implementation if the I-D had not existed.  There was no way I could have 
> taken the time of product engineers to build something that was not even 
> an I-D, much less an I-D with some traction in the working group.  By 
> building it, we learned some things that helped shape the work.  I am not 
> arguing that implementation is not a good idea.  It is a good idea.  But 
> requiring implementation before the first I-D is difficult and often 
> counter-productive.
>
> activities.  I do not have spare machine sitting around.  I don't have a 
> machine sitting around with a router stack with a set of routing protocols 
> that I can modify.  Am I supposed to be prohibited from writing an I-D on 
> a change to a routing protocol?  I don't know if my co-authors on the IDR 
> I-D have implemented yet or not.  Given that the first question with 
> regard to the I-D was whether the working group thought the problem needed 
> to be addressed, I could well understand someone holding off 
> implementation for a bit.
>
> Yours,
> Joel M. Halpern
>
> PS: It is not at all clear to me that this proposal would help the SIP 
> situation much at all.  Many of the folks have access to stacks, and could 
> throw together an implementation of their proposal.  Not a commercial 
> grade implementation, but enough to claim implementation.  And given the 
> strategic importance of SIP, these folks can frequently even get 
> management support for doing the work.
>
>> >process.
>> > of