Subject: Re: An IETF repository for working code in our protocols?

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 19 Aug 2020 18:19:21 +0000

Hi Vijay,

In additional to the points others have made, I’ll add that an IETF code repository would need to carefully work out license issues and also how it fits into the IETF’s Intellectual Property set up that is managed by the IETF  Trust.

The various IP issues that the IETF deals with  are captured in the Trust Legal Provisions (TLP) documented here:  https://trustee.ietf.org/license-info/IETF-TLP-4.htm

There’s a couple of points that jump out at first glance:

  1.  We would want the keep the licenses consistent and not allow a free for all of whatever license the authors liked – as you’ll see in the TLP  section 4.c the IETF uses as. Simplified BSD License for code components in IETF Contributions and IETF Documents.

From experience it can be hard to have simple consistent licensing because a lot of contributions have had some other license applied by the authors, the authors companies, or code that the new code is built on.

  2.  If you look at section 8. You’ll also see that different Document Streams are treated differently today, consideration would have to be given to how such different treatment might apply to code in this IETF repository would interest with the different Document Streams.

  1.  Of course there is the big question of who would be affected if some party tool legal action around the code in the repository.  While the license terms applied that the code always have limitations clauses such as clauses to remove any responsibility for function or reliability or indemnification expectations that doesn’t mean that someone with money to throw at it couldn’t still go ahead and sue the IETF around something in the repository – either that it’s in there, or IP that it infringes, or even how well it works or doesn’t work, or around security holes it might have.   The license would help in the defense, but it would cost the IETF money to defend itself.   We’ve had to do this in the past so it’s something we do know how to deal with, but adding a repository of code would open up a whole new set of things someone could decide to sue us about, or sue someone else like the co-contributor and include us as defendant in the lawsuit.

  1.  There would need to be some management of the repository applied to make sure legal issues didn’t pop up in contributions.   Today the RSE plays a bit of a role for documents as does everyone in the AD-WG process where documents get read and reviewed, but there isn’t any equivalent for code.

There may be more that pop up once the topic was looked at deeper.

I’m not saying that any of these are show stoppers, but there’s a lot of legal elements that would be need to worked out before any bits got checked into a repository.

regards
  glenn deen, wearing my IETF Trustee hat

On 8/19/20, 8:12 AM, "WGChairs on behalf of Vijay Gurbani" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

All: I was reviewing a draft [1] as part of Gen-ART.  In the draft, there is a section pertaining to "Implementation Status" as dictated by RFC 7942.  I found this section to be very interesting in that it captures the running code corresponding to the I-D, but RFC 7942 does not appear to provide the means to retain this section upon publication as a RFC.  (To be sure, it does not mandate that the section be removed, just suggests it.  However, most I-D authors simply will end up asking the RFC editor to remove the section, as [1] does.)

I think that is rather counter productive.  After all, we standardize protocols so that others can write programs that implement the protocols, and I see a lot of value in preserving any running code.  In the particular case of the I-D I reviewed, there were two implementations, both from reputable organizations (APNIC and the Italian National Research Council).  By simply deleting the "Implementation Status" section when the I-D was published as an RFC, it seems that good, quality implementations that folks spent time on would be lost, perhaps not irrevocably, but for most practical purposes, the code would be orphaned.

I am wondering whether the IETF should preserve some of this running code as an archive.  The IETF makes no guarantees of the correctness of the code, just provides a repository where code can be linked to a particular RFC to allow implementers to bootstrap themselves, if needed.  This repository will also contain the contact of the original author of the code.  The repository is sort of like an IANA registry of IETF working code corresponding to an RFC.  The authors of the I-D can be the "expert" and suggest which implementations they consider worthy of archival.  The boilerplate of implementations is given in Section 2 of RFC 7942 [2].  We would have to spec out an IANA-registry like process to do so, but I think that the ends will justify the means.

I note that organizations like IEEE have evolved to provide a repository where authors of published papers can park their datasets for reproducibility (see IEEE DataPort [3]).  The 2017 ACM Task Force on Data, Software, and Reproducibility in Publication also recommended that, "The ACM Digital Library will need to interoperate with a number of recommended data repositories and software curation platforms..." [4]..  So this idea of the IETF maintaining a software repository is not entirely unheard of.

I apologize for taking your time if this has been discussed before; if not, it is worth some discussion.

[1] https://tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-15<https://urldefense.com/v3/__https:/tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-15__;!!PIZeeW5wscynRQ!6LRzdhz1qyCdl5E-IAyg-txM3vLh5MRVyCisn5Uy82SHV3dp2HBGrsBcrCs0QPsH$>
[2] https://tools.ietf.org/html/rfc7942<https://urldefense.com/v3/__https:/tools.ietf.org/html/rfc7942__;!!PIZeeW5wscynRQ!6LRzdhz1qyCdl5E-IAyg-txM3vLh5MRVyCisn5Uy82SHV3dp2HBGrsBcrJzEst06$>
[3] https://bigdata.ieee.org/ieee-dataport<https://urldefense.com/v3/__https:/bigdata.ieee.org/ieee-dataport__;!!PIZeeW5wscynRQ!6LRzdhz1qyCdl5E-IAyg-txM3vLh5MRVyCisn5Uy82SHV3dp2HBGrsBcrMKWaHCt$>
[4] https://www.acm.org/publications/task-force-on-data-software-and-reproducibility<https://urldefense.com/v3/__https:/www.acm.org/publications/task-force-on-data-software-and-reproducibility__;!!PIZeeW5wscynRQ!6LRzdhz1qyCdl5E-IAyg-txM3vLh5MRVyCisn5Uy82SHV3dp2HBGrsBcrB5KC76r$>

Thank you,

- vijay

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 13:46:32 -0500

Dear Glenn: Thank you very much for your note.  More inline.

On Wed, Aug 19, 2020 at 1:19 PM Deen, Glenn (NBCUniversal) <
[email-masked]> wrote:

> Hi Vijay,
>
>  In additional to the points others have made, I’ll add that an IETF code
> repository would need to carefully work out license issues and also how it
> fits into the IETF’s Intellectual Property set up that is managed by the
> IETF  Trust.
>
> [...]
>
>
> I’m not saying that any of these are show stoppers, but there’s a lot of
> legal elements that would be need to worked out before any bits got checked
> into a repository.
>

Yes, absolutely.  All of what you listed are important and weighty
concerns.  But none of them appear to be insurmountable if we decided to do
this.  Clearly, the precedent set by IEEE and ACM in similar areas seems to
point to the possible existence of a happy medium, should we go this route.

Cheers,

- vijay

---

From: Erik Kline <[email-masked]>
Date: Wed, 19 Aug 2020 13:09:10 -0700

I tend to think that the value of example implementation code varies over
time, in no small part because people often need to redo an implementation
because of development language requirements, or they're building a
component that subsumes/merges several protocols, etc.

The things I find most useful are the those like the NIST and crfg RFC
style of test vectors and expected outputs (and even sometimes expected
internal state!).

I would love to see some kind of standard approach for documenting what are
essentially unit tests: given these inputs/this state/this HTTP method
call/... -> expect these outputs/state changes/lists of observable
behaviour.

Given the centrality of interoperability to the IETF, this seems like
something we might be able to explore.

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 15:21:39 -0500

Dear Erik: Thank you for your thoughts and your time.  Please see inline.

> component that subsumes/merges several protocols, etc.

Yes, absolutely.  The code archive should not require that the code be the
most optimized code with the smallest runtime and space complexity
measures.  And because the code will be archived upon at the end, when an
I-D becomes an RFC, hopefully it reflects the merges of several protocols.
(For those protocols where this is important, of course.  Not all protocols
are subsumes/merges of others.)

> I would love to see some kind of standard approach for documenting what
> are essentially unit tests: given these inputs/this state/this HTTP method
> call/... -> expect these outputs/state changes/lists of observable behaviour
>

Those constructs you mention --- protocol style unit tests --- are
extremely valuable.  The SIP working group, when it was active, produced a
few such documents that contained complete protocol flows that aided in
testing and understanding of how the protocol works.  Many of these
documents were made into Informational RFCs.  The ALTO working group had
also produced a HTTP style RESTful API document that outlines the state of
HTTP headers and payload; alas, it did not progress to an RFC, but it was
very helpful when we had interoperability events for ALTO.

> something we might be able to explore.

Yes, absolutely.  However, the working code (good working code) being
available is a bit of an orthogonal, though equally important, discussion
as is the one for establishing test vectors.  So at least I would not like
to conflate these two at the current time.

Thanks much!

- vijay

---

From: Erik Kline <[email-masked]>
Date: Wed, 19 Aug 2020 13:52:32 -0700

> very helpful when we had interoperability events for ALTO.

Brilliant!  Yes, this was exactly the kind of thing of which I was thinking.

> to conflate these two at the current time.
But surely we'd want the code to path the protocol unit tests, yes?  ;-)

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Aug 2020 18:22:57 -0400

One of the advantages of pointers to implementations is that it can 
include equally pointers to open source and pointers to closed source of 
various forms.  The IETF doesn't take a stance among implementations.

Yours,
Joel

On 8/19/2020 2:46 PM, Vijay Gurbani wrote:
> On Wed, Aug 19, 2020 at 1:19 PM Deen, Glenn (NBCUniversal) 
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Hi Vijay,
> 
>       In additional to the points others have made, I’ll add that an
>     IETF code repository would need to carefully work out license issues
>     and also how it fits into the IETF’s Intellectual Property set up
>     that is managed by the IETF  Trust.
> 
>     [...]
> 
>     I’m not saying that any of these are show stoppers, but there’s a
>     lot of legal elements that would be need to worked out before any
>     bits got checked into a repository.
> 
> 
> concerns.  But none of them appear to be insurmountable if we decided to 
> do this.  Clearly, the precedent set by IEEE and ACM in similar areas 
> seems to point to the possible existence of a happy medium, should we go 
> this route.
> 
> Cheers,
> 
> - vijay
>

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 19 Aug 2020 23:33:56 +0100

Hiya,

> various forms.  The IETF doesn't take a stance among implementations.

Entirely surmountable problem IMO. Add caveats to avoid
misperceptions wrt stance.

That said, open source implementations are plainly more
useful for people who want to implement RFCs, compared to
equivalent closed-source implementations. It'd be folly
to ignore that reality.

I just now discovered that what I thought ought be a
compressed-point representation of a DH shared secret is
supposed to use the uncompressed format thanks to being
able to add more trace lines to someone else's code. That's
the kind of thing could suck up many more hours of effort,
or that might lead to non-interop, (as it only affects some
groups), if no open-source code were available.

S.

>> concerns.  But none of them appear to be insurmountable if we decided
>> to do this.  Clearly, the precedent set by IEEE and ACM in similar
>> areas seems to point to the possible existence of a happy medium,
>> should we go this route.
>>
>> Cheers,
>>
>> - vijay
>>
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Aug 2020 18:51:41 -0400

There is different value in open and closed source implementations.  But 
bother are valuable.
There is as far as I can tell no benefit in the IETF actually storing 
the source for these projects.  Either they are being maintained or they 
aren't.  If they aren't, the source has much less value.  If they are, 
the maintained one is what people should look at.

So again, having an easily findable and easily maintained set of 
pointers to any implementations that folks want to list, with whatever 
terms those folks like (whether that be creative commons open source, 
GPL, or descriptions of proprietary implementations) seems very useful. 
The problem of findability is real, but is not different for an archive 
compared with a set of pointers.

A set of input / output vectors along the lines Erik suggested also 
sounds useful.  For some protocols.  (I don't quite know how one would 
do that for something like OSPF.)

Yours,
Joel

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 20 Aug 2020 00:09:42 +0100

Hiya,

> There is different value in open and closed source implementations.  But

I maintain that, for implementers of RFCs, the existence of
open-source code is valuable in a way that closed-source
can never match. ISTM, that means OSS is inherently better
for the purpose we're discussing here.

> the source for these projects.

I disagree. Finding the commit that matches the time at
which the RFC was written can be non-trivial and will
sometimes be useful/needed.

>  Either they are being maintained or they
> aren't.  If they aren't, the source has much less value. 

I disagree. For an implementer of an RFC, there is still a
lot of value in looking at someone else's code that matches
the RFC (or nearly does).

>  If they are,

Not necessarily. The not-very-good but easy to read code
that existed at the time the RFC was written might well be
the most useful thing for an implementer.

> compared with a set of pointers.

Given the above, I fail to see how you can think closed-
source can be as useful there, regardless of the mechanism
for de-referencing a pointer.

Cheers,
S.

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Wed, 19 Aug 2020 23:12:30 +0000

I am thrilled to see this thread and all the great thoughts shared on it.
The IETF Hackathons have thankfully resulted in the creation of a great deal of code, much of which is open source and almost all of which is related to existing and evolving IETF standards. Some of this code does not have much value beyond a specific IETF Hackathon, whereas some has the potential to serves as a valuable resource for developers looking to better understand or implement what is specified in a given IETF draft or RFC. 

Each IETF Hackathon wiki provides a links to much of the code used in the context of that Hackathon (e.g.   https://trac.ietf.org/trac/ietf/meeting/wiki/108hackathon.)
We also have a the results presentations that typically provide links to code (e.g. https://github.com/IETF-Hackathon/ietf108-project-presentations). 
We also have some repos created in the IETF-Hackathon GitHub org that host Hackathon projects, https://github.com/IETF-Hackathon. 

All of this is good; however, I have been thinking for a while that we really should do something better. I would be interested in putting a bit more thought, structure, and consistency behind how and where to document the existence of code related to IETF draft and RFCs. I am would be happy to work with others to define some conventions and try to use them for the Hackathon and for code related to IETF work in general. A link in an "Additional URL" in the datatracker to one or more GitHub repos or GitHub orgs, or to a README file that contains links to corresponding code repos is an example of something simple we could try. I am sure others have better idea.

Cheers,
Charles 

On 8/19/20, 3:34 PM, "WGChairs on behalf of Stephen Farrell" <[email-masked] on behalf of [email-masked]> wrote:

    Hiya,

    S.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Aug 2020 19:42:06 -0400

For IETF purposes, the existence of implementations is very important.
For some purposes, open source code is useful.
Depending upon the license, for many purposes looking at open source 
code can actually be harmful to the implementation work.

If we want to make available pointers to specific versions of open 
source, as well as pointers to the information about the implementation, 
that is fine.  For many uses, it is as or more important to be able to 
look at the implementors summary of what they actually support than it 
is to look at the code.  For other purposes, looking at the code is 
clearly useful.

I am not asking that the IETF not point to such available code.
I am asking that if we are going to identify implementations that we 
identify implementations regardless of their status.

I presume even for open source we will not attempt to verify the code 
correctness, operability, or quality.  We are not talking about 
reference implementations.  For different consumers of this information, 
different implementations will have different value.  We do not 
currently second-guess implementation or interoperability reports, and I 
do not suggest we start.

As just one example, if I am looking for things to interoeprate with, I 
may well be just as happy with commercial implementations as with open 
source.  Maybe happier as I do not have to figure out how to build 
someone else's structures.

Archiving the soruce has many complications.  And very little value I 
can understand.  After all, if it is open soruce, then it is available 
somewhere else.  if the people making it available do not want to 
continue to do so, it is not the IETF's job to take care of their code 
for them.  Conversely, if they are interested in their work being 
pointed to by the IETF, I am happy to see us do so.

Yours,
Joel

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 20 Aug 2020 00:51:39 +0100

I won't keep repeating myself (promise:-) but...

> code can actually be harmful to the implementation work.

The "many" above seems to me to be a severe exaggeration.
At least my reality is otherwise.

And anyway, if looking at someone's code hurts someone's
eyeballs or profits, they can just avert their eyes:-)

S.

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Thu, 20 Aug 2020 00:01:22 +0000

Hi Stephen, et al, 

I agree totally with Joel. 

On 8/19/20, 7:52 PM, "WGChairs on behalf of Stephen Farrell" <[email-masked] on behalf of [email-masked]> wrote:

What is the advantage of the IETF maintaining the repository as opposed to pointing to a GitHub repository/repositories maintained the open source implementor(s)? I see none. 

Thanks,
Acee

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 20 Aug 2020 01:22:01 +0100

On 20/08/2020 01:01, Acee Lindem (acee) wrote:
> What is the advantage of the IETF maintaining the repository as
> opposed to pointing to a GitHub repository/repositories maintained
> the open source implementor(s)? I see none.
I do. Linking the RFC to the version of the code that
claims to implement that is useful. Not depending on
the kindness of, or lack of bitrot at, git<blah>.<tld>
is good.

Doesn't need to be hard, could just be a snapshot
of a git repo that's tied to the RFC (or I-D) via
the tracker. (I don't much care exactly how.)

But claiming that's useless is not something I can
grok TBH.

S.

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 19:38:20 -0500

On Wed, Aug 19, 2020 at 6:42 PM Joel M. Halpern <[email-masked]> wrote:

> code can actually be harmful to the implementation work. [...]
>

Dear Joel: Thank you for your comments, much appreciated.

I will note two things that are important to keep in mind as we develop our
protocols.

One, you do not need to look very far to ascertain that there is a dearth
of reproducibility in the sciences today, be it in academia or industrial
research.  This has, in part, lead to the stance of IEEE and ACM that I
pointed to in my original email where they have made repositories available
to researchers for storing datasets and software artifcacts.  Making the
source code available mitigates the reproducibility problem, not in whole,
but in substantial part.

Two, for an organization that prides itself on open and transparent
development of protocols, it seems rather incongruous that a key component
of a protocol --- the source code that implements a protocol --- is kept
secret.  Companies and individuals that do not want to share code are free
not to do so.  However, that need should not impinge upon the need for
others who can benefit from sharing for the purpose of reproducibility.
Individuals who would rather not deal with compilation and other problems
associated with open source are just as welcome not to worry about it, and
let others who revel in it move the implementation and the protocol ahead.
In most of the cases, the source code is not going to be a production ready
version, so I really see no harm in making these available on a "best
effort" scale.

In closing, I don't think that expounding the virtues or vices of open
source versus closed source is necessarily the issue here.  All of us have
strong viewpoints on this, and for the record, I am more in alignment with
Stephen in the need for the code to be available.  It can be made so under
some license just as we handle IPR issues.  Bringing in open versus closed
source code detracts from the larger aim of the discussion here, with all
due respects.

Thank you.

- vijay

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Aug 2020 20:50:52 -0400

I am missing something Stephen.  We can establish exactly the 
relationship you describe without the IETF archiving the source code.

So what is the advantage you see in the IETF making a snapshot of a 
project and storing it separately from that project?  For example, I 
work with fd.io.  That implements a lot of IETF RFCs.  Very effectively. 
  In open source.  I can not see any benefit in the IETF making one (or 
worse yet several) copies of fd.io.  I can see benefi, for cases where 
the fd.io developers want it, for the IETF to have pointers to the fd.io 
work with indications of what RFCs it implements (a lot).  If someone 
wants to do the work, that could include pointers to specific versions 
(which will fall out of maintenance), pointers to the most current 
versions, pointers to the whole project so people can get context, and 
pointers to specific sub-projects concerned with specific RFCs.  How 
elaborate it gets probably depends upon both what we want and how much 
work the fd.io developers care to put in.

But if we want to point to OSPF implementations, it would seem very 
strange not to point to the widely used implementations from Cisco and 
Juniper (multiple from each, and ones from a number of other vendors). 
Yes, we should point to the several open source ones (if the 
implementors want us to.)

We do regularly get asked / ask ourselves "who has implemented this". 
Improving and maintaining our ability to answer that question seems very 
useful.  Pretending that the only implementations that matter are the 
open source ones seems counter-productive.

Yours,
Joel

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 20 Aug 2020 02:08:09 +0100

Hiya,

> relationship you describe without the IETF archiving the source code.

I don't believe I made any suggestion as to who would
claim/establish that some code maps to an RFC/I-D. I'd
imagine authors/chairs are far more likely to do that
than "the IETF."

> project and storing it separately from that project? 

I believe I already said that. See my example wrt EC points. I could wax
lyrical but not sure that'd help
more;-)

> For example, I

That all sounds good. I also already said why I do think
there's value in an IETF-archived copy. As another point,
fd.io (hadn't heard of 'em btw), was registered in 2013.
I've seen various excellent industry efforts come and go
over the years and would be more confident that ietf.org
might have the same function in 10 years than such well
meaning but likely more short-lived domains.

> implementors want us to.)

I'd have no objection to that, assuming all the same
caveats as ought apply to a pointer to any implementation.
All I claimed was that having the overall set of pointers
include some reasonable OSS adds value that is inherently
missing for closed-source.

> open source ones seems counter-productive.

I don't believe I advocated for "pretending" of any kind.

Cheers,
S.

---

From: Paul Wouters <[email-masked]>
Date: Wed, 19 Aug 2020 22:10:37 -0400 (EDT)

> sometimes be useful/needed.

As a Fedora and RHEL package maintainer, and as a person who has
forked an IKE implementation that still hears about the pre-fork
code, i can tell you that unmaintained code _very_ quickly loses
value and relevance. I do not think there is any value in the
IETF keeping a copy of unmaintained code.

> the RFC (or nearly does).

There is hardly any point in looking at 5 year old code. For one,
it won't compile with current C compilers. Or to pick another
example, 5 year old code using openssl (0.9) will be useless
now with openssl 1.1.

Also, the amount of RFC relevant code compared to the rest of
the code base is likely very small. I can see value in a
pure simple reference implementation for something that is
not meant to be produciton code. But nothing much more.

I also share concerns with people who mentioned that leaving
this in RFCs quickly becomes advertising.

Also, any bugs in "IETF references code" would become the
new defacto RFC.

I think the current method of Implementation Status is exactly
how we should want to keep it.

Paul

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 20 Aug 2020 13:16:09 +0000

There are organizations that scrupulously avoid viral licenses.  We need to make sure that any IETF references, or archival thereof, of open source programs don't slide into the "just read the code" state.  And so on.

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 20 Aug 2020 13:18:07 +0000

Perhaps time to move this to the general IETF discussion list?

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 20 Aug 2020 14:10:05 +0000

Actually we just started an IAB program on Evolvability, Development, and Maintenance (EDM) and this would be fully in scope for the  Maintenance part. Mailing list is [email-masked]

On 20.08.20, 15:18, "irsg on behalf of Salz, Rich" <[email-masked] on behalf of rsalz=[email-masked]> wrote:

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Thu, 20 Aug 2020 14:30:52 +0000

Hi Joel,

Please see some thoughts inline [cue].

On 8/19/20, 4:42 PM, "WGChairs on behalf of Joel M. Halpern" <[email-masked] on behalf of [email-masked]> wrote:

[cue] I do not understand how this can be the case. Can you elaborate?

[cue] For open source projects, this is traditionally done using a README file. As with any code, the quality of the documentation, and in this case the documentation typically starts with a README file, varies great. However, the place to but the status, quality, prerequisites, limitations, known issues, etc., all belong in the README, or in documentation pointed to from the README.

[cue] My hunch is we should not archive code in general There may be exceptions, but most of the code I have in mind is not something we would want to archive. Rather, we want to make it easy to discover and access for those who might find it helpful. Perhaps some code is separate from the related RFCs and worth archiving. YANG models are an example. For the sake of argument and example, let's suppose we were to move YANG models out of RFCs and replace them ptrs to standalone code/YANG models that are archived. I can see this being helpful. There are of course pros and cons, and I am using this only as an example to hopefully explain my thoughts a bit more clearly.

Cheers,
Charles

    Yours,
    Joel

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Thu, 20 Aug 2020 14:38:15 +0000

Thanks, Mirja. I subscribed and look forward to continuing the conversation there. I hope others who are interested do so as well.

Cheers,
Charles

On 8/20/20, 7:10 AM, "WGChairs on behalf of Mirja Kuehlewind" <[email-masked] on behalf of mirja.kuehlewind=[email-masked]> wrote:

---

From: Eliot Lear <[email-masked]>
Date: Thu, 20 Aug 2020 16:45:28 +0200

> On 20 Aug 2020, at 01:09, Stephen Farrell <[email-masked]> wrote:
> 
> 
> Hiya,
> 

When we speak in generalities, some finer points get lost.  I can see value in open source in that people can take it and use it.  And this is good to a point.  One perversity is that if the standard is complex and there is source available, then that might be the ONLY implementation, or one of a very small number.  This is precisely what happened in 2002 with SNMP and the ASN.1 interpreter bug that hit the entire industry, and it has since happened with openssl, although there are now a good number of libraries out there that cover its ground.

If the point is to test interoperability, then OSS and closed source can be of similar value if someone is running a test harness of some sort through the closed source, and we see that fairly often as well in the form of bakeoffs and test portals.  We do both OSS and closed now with MUD for instance.

> sometimes be useful/needed.

There is some value of a pointer.  First, very little code only implements one standard.  The code has a purpose in life that may extend beyond a POC.  And this also means that the only thing that the IETF has to worry about is how to update a list of pointers, and not who gets to commit on active projects.  And yeah, it might also keep the IPR just a bit simpler.

Eliot

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 10:17:04 -0500

Dear Paul: Good morning, and thanks for your comments.  Please see inline.

On Wed, Aug 19, 2020 at 9:10 PM Paul Wouters <[email-masked]> wrote:

> As a Fedora and RHEL package maintainer, and as a person who has forked an
> IKE implementation that still hears about the pre-fork code, i can tell you
> that unmaintained code _very_ quickly loses value and relevance. I do not
> think there is any value in the IETF keeping a copy of unmaintained code.
>

The proposal is not for the IETF to "maintain" code, with all of the fine
points and complexities that entails.  The proposal is simply to provide
starter code that implements a protocol, period.  There is no expectation
that this is production code.  There is no expectation that such code is
mandatory to provide, just that if someone has implemented the protocol and
wants to share their code, well, why not.  It may make it easy for others
to bootstrap themselves.

There is hardly any point in looking at 5 year old code. For one, it won't
> compile with current C compilers. Or to pick another example, 5 year old
> code using openssl (0.9) will be useless now with openssl 1.1.
>

That is a good point, but pragmatically speaking, an implementer looking at
a RFC that is 5 years old and contains a link to some code that was current
5 years ago should reset their expectations appropriately.

But taking an extremist view, I will note that for the most part, I can
still compile 10 years old code with the current version of gcc (10.2.1) on
my system.  Most languages, even as they evolve, tend to keep backward
compatibility.  Yes, there are cases like Python 3.0 not being compatible
with Python 2.x, but those are rare.  One of the reasons that Java Generics
have such a horrible syntax is to preserve backward compatibility at the
bytecode level.  So, at least I am a bit optimistic on the language front.

> I also share concerns with people who mentioned that leaving this in RFCs
> quickly becomes advertising.
>

That is a concern, but earlier emails in the thread touched upon it and
ways to mitigate it.

> Also, any bugs in "IETF references code" would become the new defacto RFC.
>

Respectfully, I don't see why that will be the general case.  I suspect it
will happen in some cases, nothing is ever perfect after all.  What is
under change control of the IETF is the RFC, not the code, which is an
artifact of the RFC.

> I think the current method of Implementation Status is exactly how we
> should want to keep it.
>

Yes, that is certainly one outcome.  Thank you for your thoughts.

- vijay

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 20 Aug 2020 17:18:41 +0200

Good points, Eliot

Random thoughts:

I always thought sendmail was likely the earliest reference example for "single" (dominant)
implementation protocol back in the day (rfc822). Made popular of course by 
https://en.wikipedia.org/wiki/Morris_worm

I wonder if there would be in IETF land ever be value in official "reference software"
I remember this primarily from ISO/MPEG where it was/?is? common, e.g.:

https://standards.iso.org/ittf/PubliclyAvailableStandards/c025029_ISO_IEC_TR_11172-5_1998(E)_Software_Simulation.zip

I guess implicitly we have such reference software implicitly in our tooling chain, e.g.: 
tools written by IETF tooling team for e.g.: rfc7991 (RFC xml). Not sure if
there is anything similar for "technical" area work product (RFCs).

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Eliot Lear <[email-masked]>
Date: Thu, 20 Aug 2020 17:24:25 +0200

> On 20 Aug 2020, at 17:18, Toerless Eckert <[email-masked]> wrote:
> 
> Good points, Eliot
> 
> Random thoughts:
> 

Good point!

> https://standards.iso.org/ittf/PubliclyAvailableStandards/c025029_ISO_IEC_TR_11172-5_1998(E)_Software_Simulation.zip

People have done this in the past in RFCs themselves, of course.  Various crypto algorithms come to mind.

Eliot

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 20 Aug 2020 10:38:06 -0500

Just to add one point to Vijay's incantation of IEEE ...

On Wed, Aug 19, 2020 at 7:38 PM Vijay Gurbani <[email-masked]>
wrote:

> I will note two things that are important to keep in mind as we develop
> our protocols.
>
> research..  This has, in part, lead to the stance of IEEE and ACM that I
> to researchers for storing datasets and software artifcacts.
>

IETF has a leadership coordination team with IEEE 802 (and that team does
occasionally talk about IEEE topics that aren't IEEE 802-specific).

I don't see a members list at
https://www.iab.org/liaisons/iab-ieee-coordination/ (which is fine), but
the most recent minutes at
https://www.iab.org/wp-content/IAB-uploads/2020/06/ietf-ieee802-minutes-2020-02-25.txt
contain a list list of attendees, a fair number of whom are on WGchairs
(either as chairs, or as IESG or IAB members).

Perhaps asking IEEE what legal precautions they took would be instructive?

Best,

Spencer

> Making the source code available mitigates the reproducibility problem,
> not in whole, but in substantial part.
>
> some license just as we handle IPR issues..  Bringing in open versus closed

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 20 Aug 2020 15:39:36 +0000

There is ongoing tussle with one person in the NTP working group who keeps calling their code "the NTP reference implementation." I would be cautious about enabling others to do same.  Yes, for many years TLS was defined as "works with OpenSSL" and I don't think most folks thought was an ideal state either.

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 20 Aug 2020 18:01:29 +0200

> There is ongoing tussle with one person in the NTP working group who keeps calling their code "the NTP reference implementation." I would be cautious about enabling others to do same.  Yes, for man…

I guess there is little that can be done against someone calling their code
reference implementation. Except of course having actually WG endorsed
reference code and calling that "IETF FOOBAR-WG <protocol> reference code".

Eliot mentioned crypto code, but i don't think we have official "endorsement/work-product status"
for any such code... or do we ?

Cheers
    Toerless

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 20 Aug 2020 16:05:38 +0000

For a long time there was only Dave Mills's NTP code, and the current maintainer insists that this is still the reference implementation.  The IETF doesn't have reference implementations, nor should it.

We have several RFC's that contain implementations.

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 20 Aug 2020 18:18:33 +0200

> We have several RFC's that contain implementations. 

Right. I was just compiling the NomCom hash algo RFC code last month. Indeed ;-)

Yeah.... that kinda sucks as a way to represent source code. Maybe if there is sufficient interest this is something that could be overhauled.

Cheers
    Toerless

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 12:07:31 -0500

On Thu, Aug 20, 2020 at 10:40 AM Salz, Rich <rsalz=
[email-masked]> wrote:

> There is ongoing tussle with one person in the NTP working group who keeps
> calling their code "the NTP reference implementation." I would be cautious
> about enabling others to do same.  Yes, for many years TLS was defined as
> "works with OpenSSL" and I don't think most folks thought was an ideal
> state either.
>

Dear Rich: Indeed, I agree.  This was the implicit "advertisement" part
that folks referred to in earlier emails in the thread.  I think it can be
managed (don't know how successful we will be at it, though).

Thanks,

- vijay