Subject: An IETF repository for working code in our protocols?

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 10:10:37 -0500

All: I was reviewing a draft [1] as part of Gen-ART.  In the draft, there
is a section pertaining to "Implementation Status" as dictated by RFC
7942.  I found this section to be very interesting in that it captures the
running code corresponding to the I-D, but RFC 7942 does not appear to
provide the means to retain this section upon publication as a RFC.  (To be
sure, it does not mandate that the section be removed, just suggests it.
However, most I-D authors simply will end up asking the RFC editor to
remove the section, as [1] does.)

I think that is rather counter productive.  After all, we standardize
protocols so that others can write programs that implement the protocols,
and I see a lot of value in preserving any running code.  In the particular
case of the I-D I reviewed, there were two implementations, both from
reputable organizations (APNIC and the Italian National Research Council).
By simply deleting the "Implementation Status" section when the I-D was
published as an RFC, it seems that good, quality implementations that folks
spent time on would be lost, perhaps not irrevocably, but for most
practical purposes, the code would be orphaned.

I am wondering whether the IETF should preserve some of this running code
as an archive.  The IETF makes no guarantees of the correctness of the
code, just provides a repository where code can be linked to a particular
RFC to allow implementers to bootstrap themselves, if needed.  This
repository will also contain the contact of the original author of the
code.  The repository is sort of like an IANA registry of IETF working code
corresponding to an RFC.  The authors of the I-D can be the "expert" and
suggest which implementations they consider worthy of archival.  The
boilerplate of implementations is given in Section 2 of RFC 7942 [2].  We
would have to spec out an IANA-registry like process to do so, but I think
that the ends will justify the means.

I note that organizations like IEEE have evolved to provide a repository
where authors of published papers can park their datasets for
reproducibility (see IEEE DataPort [3]).  The 2017 ACM Task Force on Data,
Software, and Reproducibility in Publication also recommended that, "The
ACM Digital Library will need to interoperate with a number of recommended
data repositories and software curation platforms..." [4].  So this idea of
the IETF maintaining a software repository is not entirely unheard of.

I apologize for taking your time if this has been discussed before; if not,
it is worth some discussion.

[1] https://tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-15
[2] https://tools.ietf.org/html/rfc7942
[3] https://bigdata.ieee.org/ieee-dataport
[4]
https://www.acm.org/publications/task-force-on-data-software-and-reproducibility

Thank you,

- vijay

---

From: "Adrian Farrel" <[email-masked]>
Date: Wed, 19 Aug 2020 16:25:14 +0100

Hi Vijay,

Responding as a co-author of RFC 7942.

When we first wrote the draft that became RFC 6982 and then matured into RFC 7942 we were conscious that any information about implementation held in an RFC is necessarily only true at the time of publication: it may be missing important details of implementations that existed at the time of publication, and obviously cannot be updated for new implementations.

We felt that archiving this information in the RFC may be misleading and also unfair, and so we suggested removing the implementation status section of the document before publication.

However, in Section 3 we note that there are alternative formats, and suggest that one reason for publishing the information elsewhere is:

   -  If the working group decides that the information is still

      valuable (and needs to be kept current) after the I-D is published

      as an RFC, and the Implementation Status section had been removed

      from it.

That seems to fit with the idea in your email.

Although out of scope of our draft, we suggested that working groups might like to use their wiki pages to track implementation and keep it up to date. We also (rather cheekily, IMHO) floated the idea that the datatracker page for a document might also contain a link to the implementation page (like it does for IPR disclosures).

With author hat back off, I’m slightly wary about the IETF becoming a venue for advertisements, so I would encourage people to be quite careful about how this sort of information is collected and maintained.

Best,

Adrian

From: WGChairs <[email-masked]> On Behalf Of Vijay Gurbani
Sent: 19 August 2020 16:11
To: Alissa Cooper <[email-masked]>; Martin Duke <[email-masked]>; [email-masked]
Subject: An IETF repository for working code in our protocols?

All: I was reviewing a draft [1] as part of Gen-ART.  In the draft, there is a section pertaining to "Implementation Status" as dictated by RFC 7942.  I found this section to be very interesting in that it captures the running code corresponding to the I-D, but RFC 7942 does not appear to provide the means to retain this section upon publication as a RFC.  (To be sure, it does not mandate that the section be removed, just suggests it.  However, most I-D authors simply will end up asking the RFC editor to remove the section, as [1] does.)

I think that is rather counter productive.  After all, we standardize protocols so that others can write programs that implement the protocols, and I see a lot of value in preserving any running code.  In the particular case of the I-D I reviewed, there were two implementations, both from reputable organizations (APNIC and the Italian National Research Council).  By simply deleting the "Implementation Status" section when the I-D was published as an RFC, it seems that good, quality implementations that folks spent time on would be lost, perhaps not irrevocably, but for most practical purposes, the code would be orphaned.

I am wondering whether the IETF should preserve some of this running code as an archive.  The IETF makes no guarantees of the correctness of the code, just provides a repository where code can be linked to a particular RFC to allow implementers to bootstrap themselves, if needed.  This repository will also contain the contact of the original author of the code.  The repository is sort of like an IANA registry of IETF working code corresponding to an RFC.  The authors of the I-D can be the "expert" and suggest which implementations they consider worthy of archival.  The boilerplate of implementations is given in Section 2 of RFC 7942 [2].  We would have to spec out an IANA-registry like process to do so, but I think that the ends will justify the means.

I note that organizations like IEEE have evolved to provide a repository where authors of published papers can park their datasets for reproducibility (see IEEE DataPort [3]).  The 2017 ACM Task Force on Data, Software, and Reproducibility in Publication also recommended that, "The ACM Digital Library will need to interoperate with a number of recommended data repositories and software curation platforms..." [4]..  So this idea of the IETF maintaining a software repository is not entirely unheard of.

I apologize for taking your time if this has been discussed before; if not, it is worth some discussion.

[4] https://www.acm.org/publications/task-force-on-data-software-and-reproducibility

Thank you,

- vijay

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 19 Aug 2020 15:30:17 +0000

*   We also (rather cheekily, IMHO) floated the idea that the datatracker page for a document might also contain a link to the implementation page (like it does for IPR disclosures).

Note that WG pages and document pages now have flexible “additional resources” section for creating just these kinds of links.

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 11:04:20 -0500

Dear Adrian: Thank you for your engagement as a co-author of RFC 7942 as
well as your individual comment.  Please see inline.

On Wed, Aug 19, 2020 at 10:25 AM Adrian Farrel <[email-masked]> wrote:

> When we first wrote the draft that became RFC 6982 and then matured into
> RFC 7942 we were conscious that any information about implementation held
> in an RFC is necessarily only true at the time of publication: it may be
> missing important details of implementations that existed at the time of
> publication, and obviously cannot be updated for new implementations.
>
>
>
> We felt that archiving this information in the RFC may be misleading and
> also unfair, and so we suggested removing the implementation status section
> of the document before publication.
>

Those are valid concerns, indeed.  However, I note that the entire RFC
process is an iterative process itself.  When we release an RFC as a PS, we
release it with the knowledge that there may be some design choices that
could be revisited at a later stage.  Ergo, if an implementation tracks the
RFC closely, the underlying assumption would be that the implementation may
well be updated at some later time as the RFC moves through its states from
Proposed to Draft to Internet Standard.  To be sure, I am not saying that
we should simply archive any half-baked or incomplete implementation,
rather rely on the knowledge of the authors of the I-D and the chairs, and
perhaps even the IETF interoperability process for the RFC, to only archive
those implementations that are worthy of being archived.

> However, in Section 3 we note that there are alternative formats, and
> suggest that one reason for publishing the information elsewhere is:
>
> Although out of scope of our draft, we suggested that working groups might
> like to use their wiki pages to track implementation and keep it up to
> date. We also (rather cheekily, IMHO) floated the idea that the datatracker
> page for a document might also contain a link to the implementation page
> (like it does for IPR disclosures).
>

I think the above choices are sound, but I also note that: (a) Wikis have a
history of starting strong, but subsequent updates wane; (b) the notion of
checking a datatracker will not even occur to the average implementer who
does not participate in the IETF as closely as we do, in fact, I would go
as far as to say that the average implementer who looks at a RFC for
implementation only knows the RFC and the information contained there.  I
doubt that they would even be aware of what the datatracker is.  This
observation is borne out by my individual observations of talking to
company R&D staff who know at a very high level what IETF is, but beyond
knowing that the protocol is specified in a RFC, they do not know the
accoutrements associated with IETF that we take for granted (working
groups, area directors, areas, datatracker, etc.).

Thus, just like we have a link to an errata on top of the RFC, we can
maintain a link to archived programs that implement a certain RFC under
strict assumptions laid out in some document.  (These would include no
guarantee of fitness, etc.)

> With author hat back off, I’m slightly wary about the IETF becoming a
> venue for advertisements, so I would encourage people to be quite careful
> about how this sort of information is collected and maintained.
>

I suspect that you imply advertisements as indicated by the domain name in
an email address of the source code's author.  Indeed, that is a concern,
and we would not like to have half-baked or incomplete implementations
archived simply for perpetuating advertising.  I believe that this can be
mitigated by due diligence of the I-D authors, chair knowledge of the
protocol, and any bakeoffs that the WG has overseen.  (I recall that when
we were developing ALTO, there were multiple implementations of the
protocols and various extensions that could have been useful for
individuals and projects like the Open Daylight SDN controller and its
support for ALTO.)

The law of unintended consequences may yet still strike, I don't claim to
foresee all problems.  But insofar as we seek "rough consensus and running
code", having "running code" from multiple strains of company (or
individual) DNAs seems like a reasonable goal.

Thanks!

- vijay

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 11:06:28 -0500

On Wed, Aug 19, 2020 at 10:30 AM Salz, Rich <[email-masked]> wrote:

>
>    - We also (rather cheekily, IMHO) floated the idea that the
>    datatracker page for a document might also contain a link to the
>    implementation page (like it does for IPR disclosures).
>
> Note that WG pages and document pages now have flexible “additional
> resources” section for creating just these kinds of links.
>

Dear Rich: Thank you for your feedback.  Indeed, yes, we do.  But as I
replied in my earlier email, these additional resources are all but hidden
to those who know the IETF way.  Query your average developer in a company
and chances are good that they will not know how to navigate through the
IETF ecosystem to get to what they would like to have.

Cheers,

- vijay

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Aug 2020 12:19:05 -0400

I would be very concerned about the IETF appearing to endorse particular 
implementations by archiving the source.
I would be much more comfortable with Adrian's suggestion of pointers to 
implementations that wish to be listed (open source or otherwise).

In terms of your concern about wiki maintenance, archives of source are 
in my opinion worse, since they will not have any fixes made after the 
archive.

In terms of visibility, I do not see why an archive is better than a 
wiki page.  Neither should be called out in the RFC, since they are 
transient.

Yours,
Joel

On 8/19/2020 12:04 PM, Vijay Gurbani wrote:
> On Wed, Aug 19, 2020 at 10:25 AM Adrian Farrel <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Hi Vijay,____
> 
>     __ __
> 
>     Responding as a co-author of RFC 7942.____
> 
>     __ __
> 
>     When we first wrote the draft that became RFC 6982 and then matured
>     into RFC 7942 we were conscious that any information about
>     implementation held in an RFC is necessarily only true at the time
>     of publication: it may be missing important details of
>     implementations that existed at the time of publication, and
>     obviously cannot be updated for new implementations.____
> 
>     __ __
> 
>     We felt that archiving this information in the RFC may be misleading
>     and also unfair, and so we suggested removing the implementation
>     status section of the document before publication.
> 
> 
> process is an iterative process itself.  When we release an RFC as a PS, 
> we release it with the knowledge that there may be some design choices 
> that could be revisited at a later stage.  Ergo, if an implementation 
> tracks the RFC closely, the underlying assumption would be that the 
> implementation may well be updated at some later time as the RFC moves 
> through its states from Proposed to Draft to Internet Standard.  To be 
> sure, I am not saying that we should simply archive any half-baked or 
> incomplete implementation, rather rely on the knowledge of the authors 
> of the I-D and the chairs, and perhaps even the IETF interoperability 
> process for the RFC, to only archive those implementations that are 
> worthy of being archived.
> 
>     However, in Section 3 we note that there are alternative formats,
>     and suggest that one reason for publishing the information elsewhere
>     is:____
> 
>         -  If the working group decides that the information is still____
> 
>            valuable (and needs to be kept current) after the I-D is
>     published____
> 
>            as an RFC, and the Implementation Status section had been
>     removed____
> 
>            from it.____
> 
>     That seems to fit with the idea in your email.____
> 
>     __ __
> 
>     Although out of scope of our draft, we suggested that working groups
>     might like to use their wiki pages to track implementation and keep
>     it up to date. We also (rather cheekily, IMHO) floated the idea that
>     the datatracker page for a document might also contain a link to the
> I think the above choices are sound, but I also note that: (a) Wikis 
> have a history of starting strong, but subsequent updates wane; (b) the 
> notion of checking a datatracker will not even occur to the average 
> implementer who does not participate in the IETF as closely as we do, in 
> fact, I would go as far as to say that the average implementer who looks 
> at a RFC for implementation only knows the RFC and the information 
> contained there.  I doubt that they would even be aware of what the 
> datatracker is.  This observation is borne out by my individual 
> observations of talking to company R&D staff who know at a very high 
> level what IETF is, but beyond knowing that the protocol is specified in 
> a RFC, they do not know the accoutrements associated with IETF that we 
> take for granted (working groups, area directors, areas, datatracker, etc.).
> 
>     With author hat back off, I’m slightly wary about the IETF becoming
>     a venue for advertisements, so I would encourage people to be quite
>     careful about how this sort of information is collected and maintained.
> 
> 
> I suspect that you imply advertisements as indicated by the domain name 
> in an email address of the source code's author.  Indeed, that is a 
> concern, and we would not like to have half-baked or incomplete 
> implementations archived simply for perpetuating advertising.  I believe 
> that this can be mitigated by due diligence of the I-D authors, chair 
> knowledge of the protocol, and any bakeoffs that the WG has overseen.  
> (I recall that when we were developing ALTO, there were multiple 
> implementations of the protocols and various extensions that could have 
> been useful for individuals and projects like the Open Daylight SDN 
> controller and its support for ALTO.)
> 
> The law of unintended consequences may yet still strike, I don't claim 
> to foresee all problems.  But insofar as we seek "rough consensus and 
> running code", having "running code" from multiple strains of company 
> (or individual) DNAs seems like a reasonable goal.
> 
> Thanks!
> 
> - vijay

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 11:42:51 -0500

Dear Joel: Thank you for your time and your thoughts.  Please see inline.

On Wed, Aug 19, 2020 at 11:19 AM Joel M. Halpern <[email-masked]>
wrote:

> implementations by archiving the source.  I would be much more comfortable
> with Adrian's suggestion of pointers to implementations that wish to be
> listed (open source or otherwise).
>

I believe that there is an implicit assumption that by archiving the source
code, the IETF is endorsing a particular implementation.  I can see reasons
why it may be construed as such, and perhaps this was what Adrian meant by "the
IETF becoming a venue for advertisements".  This is definitely not the
intended consequence of this process.  I can also foresee that if there is
a de-facto state-of-art archived implementation of a certain protocol, then
derivative implementations may become genetic variations of the de-facto
implementation, with all of the problems that this will entail.  So, yes,
this is a concern, and the question is whether we can mitigate it in some
manner?

> In terms of your concern about wiki maintenance, archives of source are in
> my opinion worse, since they will not have any fixes made after the archive.
>

Right; we cannot mandate the fixes, of course.  But the purpose of such a
repository would be to provide bootstrap code, and allow implementers to
find efficient ways around what they started off as.

In terms of visibility, I do not see why an archive is better than a  wiki
> page.  Neither should be called out in the RFC, since they are transient.
>

I understand and respect that point of view.  But I will note that RFCs
themselves are transient.  Even an Internet Standard can be made obsolete
at some time by a new RFC-to-be.  So I don't see the above as problematic,
and instead see that it benefits implementers who are not part of the IETF
lore and just look at the RFC to figure things out.  (And I suspect these
implementers are far more numerous than those of us who attend the IETF and
write the protocols.  My interest is in reaching out to these implementers.)

Thanks,

- vijay

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 19 Aug 2020 10:07:30 -0700

Hi Vijay,

As a note, we’re starting up a new IAB program that may be relevant. As we look at deployability and maintainability, we’re interested in looking at how working groups can host information about implementations and interop during protocol development, as well as after protocols have shipped and groups have closed.

https://www.iab.org/activities/programs/evolvability-deployability-maintainability-edm-program/ <https://www.iab.org/activities/programs/evolvability-deployability-maintainability-edm-program/>

I’d invite people to join the list if they want to discuss!

Thanks,
Tommy

> On Aug 19, 2020, at 8:10 AM, Vijay Gurbani <[email-masked]> wrote:
> 
> [1] https://tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-15 <https://tools.ietf.org/html/draft-ietf-regext-rdap-sorting-and-paging-15>
> [2] https://tools.ietf.org/html/rfc7942 <https://tools.ietf.org/html/rfc7942>
> [3] https://bigdata.ieee.org/ieee-dataport <https://bigdata.ieee.org/ieee-dataport>
> [4] https://www.acm.org/publications/task-force-on-data-software-and-reproducibility <https://www.acm.org/publications/task-force-on-data-software-and-reproducibility>
> 
> Thank you,
> 
> - vijay

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 12:18:42 -0500

Dear Tommy: Thank you for your note below.  Looks like what you are
attempting to do in IAB is very pertinent to the topic of discussion I
started.  I will subscribe to it now.  Cheers.

On Wed, Aug 19, 2020 at 12:07 PM Tommy Pauly <[email-masked]> wrote:

> Hi Vijay,
>
> As a note, we’re starting up a new IAB program that may be relevant. As we
> look at deployability and maintainability, we’re interested in looking at
> how working groups can host information about implementations and interop
> during protocol development, as well as after protocols have shipped and
> groups have closed.
>
>
> https://www.iab.org/activities/programs/evolvability-deployability-maintainability-edm-program/
>

---

From: "Adrian Farrel" <[email-masked]>
Date: Wed, 19 Aug 2020 18:24:02 +0100

Hi again Vijay (and noting Tommy’s important intervention)

[snip]

Those are valid concerns, indeed.  However, I note that the entire RFC process is an iterative process itself.  When we release an RFC as a PS, we release it with the knowledge that there may be some design choices that could be revisited at a later stage.  Ergo, if an implementation tracks the RFC closely, the underlying assumption would be that the implementation may well be updated at some later time as the RFC moves through its states from Proposed to Draft to Internet Standard.  To be sure, I am not saying that we should simply archive any half-baked or incomplete implementation, rather rely on the knowledge of the authors of the I-D and the chairs, and perhaps even the IETF interoperability process for the RFC, to only archive those implementations that are worthy of being archived.

AF> You capture an important aspect, but I was concerned about the other side of it. It one company has implemented at the time the RFC is published, but others have not yet then the RFC stands forever (or at least until it is replaced or updated) without mentioning all of the other companies that implemented just a week to late to be documented. That is not very fair on the new companies and a little misleading for readers of the RFC.

AF> I don’t think we want to be publishing new RFCs just to update this list, so keeping the list out of the RFC seemed better.

[snip]

I suspect that you imply advertisements as indicated by the domain name in an email address of the source code's author.  

AF> Ah, no. I meant (possibly as suggested by Joel) commercial advertisements (as in “promotional material”). In many cases, implementation is described as “Company X’s implementation of this specification” and includes product names and release numbers. We do want that information but we don’t want this to be used as a way of saying that Company X makes splendid boxes and Company Y produces rubbish.

The law of unintended consequences may yet still strike, I don't claim to foresee all problems.  But insofar as we seek "rough consensus and running code", having "running code" from multiple strains of company (or individual) DNAs seems like a reasonable goal.

AF> Absolutely! 

AF> In our RFC, we were looking at how information about running code may influence the standardisation process. You are looking at something slightly different, and that is a good thing. Engaging with Tommy on this sounds like a sound idea.

Best,

Adrian

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 19 Aug 2020 14:47:37 -0500

Dear Adrian: Thank you for the follow up.  More inline.

On Wed, Aug 19, 2020 at 12:24 PM Adrian Farrel <[email-masked]> wrote:

> AF> You capture an important aspect, but I was concerned about the other
> side of it. It one company has implemented at the time the RFC is
> published, but others have not yet then the RFC stands forever (or at least
> until it is replaced or updated) without mentioning all of the other
> companies that implemented just a week to late to be documented. That is
> not very fair on the new companies and a little misleading for readers of
> the RFC.
>

I understand your concern, and it is a valid one.  However, note that the
archival process for the code can be, and indeed should be, limited to the
code that has tracked the I-D.  The intent is to provide some bootstrap
code to implementers from the get go.  If someone chooses to wait until a
RFC has been released before they produce code, that is certainly their
prerogative.  But I hardly think that they can make a case that their
post-RFC implementation is now a candidate for archival.  That would be
stretching the idea too far and in directions it is not intended.  (One way
to tamp down on the unintended consequences.)

> an email address of the source code's author.
>
>
>
> AF> Ah, no. I meant (possibly as suggested by Joel) commercial
> advertisements (as in “promotional material”). In many cases,
> implementation is described as “Company X’s implementation of this
> specification” and includes product names and release numbers. We do want
> that information but we don’t want this to be used as a way of saying that
> Company X makes splendid boxes and Company Y produces rubbish.
>

I agree, we do not need "promotional material" of the sort you outline
above.  And we can include such exhortations in the process we define.
That said, I don't think that the RFCs we produce are touted as "Company
X's implementation of this specification", despite the author's affiliation
on the RFC masthead.  Thus I think that this particular concern may be
mitigated by the normal social mores that have evolved in the IETF to
accord more credence to individuals rather than their particular
affiliations.

> AF> In our RFC, we were looking at how information about running code may
> influence the standardisation process. You are looking at something
> slightly different, and that is a good thing. Engaging with Tommy on this
> sounds like a sound idea.
>

Yes, thank you for that note.  I appreciate it, especially coming from a
constituent who started to even think of defining this process!

Cheers,

- vijay

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 19 Aug 2020 21:55:08 +0100

Hiya,

For many but not all RFCs, it would be valuable to have a
reasonably good implementation that is caveated up the wazoo so I've
always liked that idea and still do.

There are a number of quite surmountable issues to doing
it, yes. The IAB programme might help get that off the
ground, but I don't think that ought be a blocker if a
bunch of people want to try out something on their own.

Cheers,
S.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Aug 2020 11:30:08 +1000

Hi Vijay,

Just my .02 - I'd be wary of an IETF repository of such code, as that would convey too much authority on it. It would also create a fair amount of overhead (especially for long-term maintenance).

I also don't think that it's necessary, once a protocol is successful; after broad deployment, people don't need to go searching for implementations. It *is* useful when a project is starting, to help early implementers find each other, though. 

In the WGs I participate in, we often use wiki pages to track early implementations; e.g.:

* https://github.com/httpwg/wiki/wiki/Structured-Headers
* https://github.com/quicwg/base-drafts/wiki/Implementations

Cheers,

> On 20 Aug 2020, at 1:10 am, Vijay Gurbani <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 20 Aug 2020 11:33:45 +1000

I'm more interested in the IETF maintaining a repository of working test cases and test suites.  Test suites are much more valuable to new people coming along trying to implement the spec, and even just good examples and test cases can help development significantly, but don't necessarily have to be spelled out inside the RFC because they take up lots of space and the RFC formatting is not conducive to things like precise whitespace and line endings!

Bron.

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Wed, 19 Aug 2020 21:34:18 -0400

> practical purposes, the code would be orphaned.

"I find your lack of faith disturbing" :-)

In the days when code lived at ftp.biguniversity.edu, in a weird subdirectory
based upon the grant number under with the code was written... that was
certainly true.
In these days of github, I hardly think the code is lost.

Just the opposite seems to happen in my opinion: middle managers are
reluctant to commit funding to actually do a proper implementation because
they think they can download some proof of concept from github, compile it,
and ship it.

If the code gets orphaned, I think that either says something about the
relevance of the standard, or perhaps more charitably, about the quality of
the implementation.   or maybe the license was just wrong.

But, if the code grows it's own foosystem.org site with a readthedocs link,
and a "fork it on github", then it's probably stable and easy to find with
google.  If one is reluctant (as Joel mentioned) to appear to endorse the
code, then it might be better to write an Acknowledgement,e.g.:

  Much of the ideas and protocols described in this document were validated
  by APNIC[1] foo project team, and the Italian National Research Council[2]
  bar project team.
      [1] https://foosystem.org
      [2] https://barproject.it

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Aug 2020 11:42:36 +1000

> On 20 Aug 2020, at 11:33 am, Bron Gondwana <[email-masked]> wrote:
> 

There are a number of ways that this already happens.

E.g., the QUIC WG uses a Slack for implementers to discuss interop, and a spreadsheet to track status:
  https://docs.google.com/spreadsheets/d/1D0tW89vOoaScs3IY9RGC0UesWGAwE6xyLk0l4JtvTVg/edit#gid=1268516408

The HTTP WG has a list of unofficial testing resources:
  https://github.com/httpwg/wiki/wiki/HTTP-Testing-Resources
... and also hosts some tests in its GitHub org:
  https://github.com/httpwg/structured-field-tests

This can also be done by the community, with help from WG participants and authors:
  https://github.com/json-patch/json-patch-tests

I think the relevant question is how 'official' this needs to be. The big issue that usually comes up is that official tests implies a testing /conformance regime, which is expensive to run properly, and can be legally risky for a standards body. Keeping things a little bit unofficial helps to avoid this, while still encouraging interoperability.

I think all of this is in-scope for the EDM program; if you're interested, see:
  https://www.iab.org/activities/programs/evolvability-deployability-maintainability-edm-program/

Cheers,

>> > All: I was reviewing a draft [1] as part of Gen-ART.  In the draft, there is a section pertaining to "Implementation Status" as dictated by RFC 7942..  I found this section to be very interesting in that it captures the running code corresponding to the I-D, but RFC 7942 does not appear to provide the means to retain this section upon publication as a RFC.  (To be sure, it does not mandate that the section be removed, just suggests it.  However, most I-D authors simply will end up asking the RFC editor to remove the section, as [1] does.)
>> >

---

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 20 Aug 2020 12:09:40 +1000

>   https://www.iab.org/activities/programs/evolvability-deployability-maintainability-edm-program/

Dammit - another thing that I'm interested in and can't dedicate much time to :(

Bron.

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 12:20:50 -0500

Dear Michael: Thank you for your note!  I had to Google to get the "I find
your lack of faith disturbing" reference, and yes, I enjoy Star Wars, but
apparently not enough :-)

More inline.

On Wed, Aug 19, 2020 at 8:34 PM Michael Richardson <[email-masked]>
wrote:

> In the days when code lived at ftp.biguniversity.edu, in a weird
> subdirectory based upon the grant number under with the code was written...
> that was certainly true.  In these days of github, I hardly think the code
> is lost.
>

In the days when code lived at ftp.biguniversity.edu, the Internet had
equivalent tools to find it.  Remember gopher(1), and archie(1), and
veronica(1).  They were the github of their days.

That said, I don't think github is a panacea.  I have established anonymous
github accounts to store datasets and code when I submit papers that are
reviewed double blind.  The conference organizers are not interested in
maintaining links to github once a decision is rendered on a paper.  If the
paper is rejected, the github repository becomes an orphan.  If the paper
is accepted, then the artifacts either get moved to a sanctioned archive,
and the github repository becomes an orphan.  So, no, github is not a
panacea.  A longer term and stable storage is required for certain
scientific output (and RFCs certainly count as scientific output).

> they think they can download some proof of concept from github, compile it,

Just as likely, I have met many managers --- middle or otherwise ---- that
do due diligence and know that what you get from github may still need
scrubbing, and they'd better make sure that the licensing terms allow them
to use the software for their means.  So,  reluctance to funding due to
code being on github is not always the answer.

> the implementation.   or maybe the license was just wrong.

I think that is a value judgment that I would rather not participate in.
Clearly not every RFC that we produce becomes a rousing success.  For the
subset that do, why not provide more support?

Sincerely,

- vijay

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 13:26:38 -0500

Dear Mark: Nice to hear from you, and thanks for your comments.  A couple
of observations inline.

On Wed, Aug 19, 2020 at 8:30 PM Mark Nottingham <[email-masked]> wrote:

> Hi Vijay,
>
> Just my .02 - I'd be wary of an IETF repository of such code, as that
> would convey too much authority on it. It would also create a fair amount
> of overhead (especially for long-term maintenance).
>

Conveying too much authority is a valid concern, no doubt.  As others have
expressed, we do not want to condone a "reference implementation", or
engender some type of advertisements by submitting source code.   This
would have to be worked out through a process, if we get that far.
Regarding overhead for long-term maintenance, the effort of maintenance
could be made similar to the effort we spend on maintaining RFCs, where
most of the effort is in filing and validating errata.  Errata for source
code would mean a patch, and I can see supporting a patch or two for the
code that was submitted when the archive was established by the publication
of the RFC.  However, I do not foresee that we would constantly keep adding
new implementations to the archive, that should be out of bound.

> I also don't think that it's necessary, once a protocol is successful;
> after broad deployment, people don't need to go searching for
> implementations. It *is* useful when a project is starting, to help early
> implementers find each other, though.
>

Yes, agreed!  The idea would be light the fire of widespread implementation
by the kindling (code) that tracked the I-D as it became an RFC.

> * https://github.com/quicwg/base-drafts/wiki/Implementations

The problems with Wiki pages and datatracker hosted additions is that we
(i.e., the IETF community) know what these are and where to find them.  But
I maintain that the average developer who is implementing an RFC will not,
on the average, have the required background to know where in the ietf.org
domain it should go to in order to find all of these additions.  They will
only know one thing, the RFC.  They will know the errata associated with
the RFC because a link to the errata is on the RFC title masthead.  It will
be great if there was a link to some starter code for new protocols.

(I suspect that I have not convinced you, but that is okay.  This is an
exchange of ideas so far to see how much of this is indeed workable.)

Thanks!

- vijay

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 20 Aug 2020 18:31:09 +0000

*   I maintain that the average developer who is implementing an RFC will not, on the average, have the required background to know where in the ietf.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__ietf.org&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=4LM0GbR0h9Fvx86FtsKI-w&m=7OS-Ssh5clxZ5_hRwIAmZPGsyCme5AoxJiHn5hU9ysc&s=UCRGl3jyjE8QIb-sDIRsf66T4SAPkjdN5TecABoYqR8&e=> domain it should go to in order to find all of these additions.  They will only know one thing, the RFC.  They will know the errata associated with the RFC because a link to the errata is on the RFC title masthead.  It will be great if there was a link to some starter code for new protocols.

RFC’s can trivially have a non-normative appendix of links.

---

From: Melinda Shore <[email-masked]>
Date: Thu, 20 Aug 2020 10:50:59 -0800

Hi, Vijay:

I've been following this thread and to be honest I'm not 100%
clear on the problem you're trying to solve, which makes it a
little challenging to figure out whether or not this is the
best solution to that problem.  I do share many of the same
concerns that have been raised, particularly about what would
be implied by having code included in a repository controlled
by the IETF, but I think more clarity around perceived benefits
would be helpful.

Melinda

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 15:08:42 -0500

Dear Rich: Thanks for your time; please see inline.

On Thu, Aug 20, 2020 at 1:31 PM Salz, Rich <[email-masked]> wrote:

>
>    - I maintain that the average developer who is implementing an RFC
>    will not, on the average, have the required background to know where in the
>    ietf.org
>    <https://urldefense.proofpoint.com/v2/url?u=http-3A__ietf.org&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=4LM0GbR0h9Fvx86FtsKI-w&m=7OS-Ssh5clxZ5_hRwIAmZPGsyCme5AoxJiHn5hU9ysc&s=UCRGl3jyjE8QIb-sDIRsf66T4SAPkjdN5TecABoYqR8&e=>

Yes, absolutely.  The problem is the quality of the links external to IETF;
IETF has no control over these.  On the other hand, IETF has control over
everything under the ietf.org domain.  So hosting something under that
domain implies the permanence that is lacking with other approaches.

Thanks,

- vijay

---

From: Michael Richardson <[email-masked]>
Date: Thu, 20 Aug 2020 16:16:41 -0400

> veronica(1).  They were the github of their days.

No, I was there. They sucked for finding out where the latest patches
for the foobar project is.  You couldn't submit patches or bug reports or
anything to those FTP servers.  The code got unknowingly forked multiple times.

    > That said, I don't think github is a panacea.  I have established

Of course not.  But it's way better than before.
github leaves more unwanted things around, I agree.

    > subset that do, why not provide more support?

I'm not really trying to make a value judgement: just observation.
I'm trying to say: if there is a community active around X, then X won't
appear orphaned.  If there is no community around X... then maybe people
voted with their wallets (mindshare) already.
For instance, can you find the latest patch set for Pathalias? (why would you care?)

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 15:28:39 -0500

> by the IETF, but I think more clarity around perceived benefits

Dear Melinda: My apologies on not being clear in my initial email.

So the problem is simple: When we have high quality implementations for
certain protocols that we standardize, can we come up with a place to park
these implementations so they can be used by implementers to get a head
start?

Why is this important?  We want to incentivize the larger RFC
implementation community (i.e., NOT us IETFers, but the legions of
developers working in corporations, academia, and individually) so they can
find these sample implementations easily and use them to bootstrap their
individual implementations.  This means putting a permanent link to the
known implementations in the RFC itself, instead of assuming that the
average developer will want to.  Developers that do not participate in the
IETF to the extent we do have a limited visibility to the organization and
its workings.  To them, the RFC is the main artifact.  Just like the RFC
lists errata, perhaps it could list links to starter code for those who
want to bootstrap themselves.  Reaching out to developers outside of those
of us who are already in the IETF is my biggest incentive.  They are the
ultimate consumers of what we produce.

As always, the devil is in the details, as we have been discussing in the
email thread.  There have been many pertinent and important observations
discussed on the mailing list; I will summarize some that I remember on the
top of my head.

1/ What about licensing issues?
2/ What about maintainability?
3/ Can we stop someone from claiming that "This is *THE* reference
implementation of RFC XXYY"?  And indeed, if someone claims this, what does
that mean?
4/ Can someone suggest that an implementation be added after the RFC has
been published?
5/ Can an implementation be used purely for advertising?
6/ Are the implementations maintained in perpetuity?  Or as long as the RFC
is active (i.e., not obsoleted)?
7/ Does IETF maintaining such a repository convey any authority?
8/ Can we be informed by other organizations (IEEE and ACM)  who have
tackled some of the same problems?
Etc.

Hope this helps to some small degree.  Thanks!

- vijay

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 15:40:50 -0500

Dear Michael: Thanks!  Please see inline.

> anything to those FTP servers.  The code got unknowingly forked multiple

Sure, but those were simpler times, and in their context, the tools held.
I remember when rsync(1) was a wrapper around cvs(1) (the version control
system); collectively it was called rCVS.  It took a bit to get it to work,
but it did work, and provided the illusion of what git(1) and hg(1) provide
seamlessly today.

> I'm not really trying to make a value judgement: just observation.  I'm
> trying to say: if there is a community active around X, then X won't appear
> orphaned.  If there is no community around X... then maybe people voted
> with their wallets (mindshare) already.

Perhaps we don't need to anymore, if Pathalias has been successful.  If I
go back to when we were working on SIP, having some code to bootstrap a SIP
implementation would have been invaluable.  As the protocol became more
important, other high quality  open source and closed source
implementations appeared, obviating the need to have bootstrap code.  But
there is a need for one when a protocol is finding its footing.

Thanks,

- vijay

---

From: Melinda Shore <[email-masked]>
Date: Thu, 20 Aug 2020 12:42:36 -0800

On 8/20/20 12:28 PM, Vijay Gurbani wrote:
> certain protocols that we standardize, can we come up with a place to
> park these implementations so they can be used by implementers to get a
> head start? 

TBH you lost me at "high quality."

Anyway, I'm still unclear on why this would be a sufficient
improvement over existing mechanisms (basically, open source +
decent search engines + WG wiki pages with links to implementations
and other resources) to justify the effort and possible legal
complications.

Melinda

-- 
Melinda Shore
[email-masked]

Software longa, hardware brevis

---

From: "Pete Resnick" <[email-masked]>
Date: Thu, 20 Aug 2020 15:57:22 -0500

On 20 Aug 2020, at 13:26, Vijay Gurbani wrote:

>> I also don't think that it's necessary, once a protocol is 
>> successful; after broad deployment, people don't need to go searching 
>> for implementations. It *is* useful when a project is starting, to 
>> help early implementers find each other, though. 
>
> Yes, agreed!  The idea would be light the fire of widespread 
> implementation by the kindling (code) that tracked the I-D as it 
> became an RFC.  

It might be useful (for you all and for the IAB program) to look at the 
codestand (originally codematch) project, both its successes and its 
failures. The email lists are still in the archives. I followed it at 
the beginning, but don't know much about it other than it was in part 
trying to help match up implementors with standards.

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 16:15:11 -0500

Dear Melinda: Thank you again for your time.  Please see inline.

> TBH you lost me at "high quality."

:-) We can have a gradient measure instead of an absolute measure, sure, no
problem.  "Sufficiently reasonable quality starter code", then :-)

Anyway, I'm still unclear on why this would be a sufficient improvement
> over existing mechanisms (basically, open source + decent search engines +
> WG wiki pages with links to implementations and other resources) to justify
> the effort and possible legal
> complications.
>

Imagine if we told implementers that we know this RFC has an errata, but
with due diligence, please find them yourself.

To my earlier point again, WG pages, WG Wiki pages, datatrackers, all make
sense to you and me.  Not to many people who will like to implement our
protocols without burying themselves deep into IETF lore [1].  When I talk
to developers at companies and students at universities, if they have heard
of IETF at all, it is mostly through knowing that some organization called
IETF produces these RFCs.  That's it.  Perhaps for them that is enough.
And if you buy that argument, then the corollary is that we should do
everything in our power to make sure that they have all of the information
they need to implement the protocol from the RFC itself.

[1] They should, but that is another discussion.  I certainly have
benefited tremendously from my association with the IETF.

Thanks,

- vijay

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 20 Aug 2020 19:05:25 -0400

Vijay, I do not see how the code quality things is too work.
While I do not like the idea, if the community wantede it I could see 
the IETF providing repositories for any code that claims to implement 
specific RFCs.
Even if we were to do that, I can not see how or why we would want to 
get into judging the quality of the code.   that sounds like a recipe 
for disaster.

Yours,
Joel

On 8/20/2020 5:15 PM, Vijay Gurbani wrote:
> On Thu, Aug 20, 2020 at 3:42 PM Melinda Shore <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>      > So the problem is simple: When we have high quality
>     implementations for certain protocols that we standardize, can we
>     come up with a place to park these implementations so they can be
>     used by implementers to get a head start?
> 
> :-) We can have a gradient measure instead of an absolute measure, sure, 
> no problem.  "Sufficiently reasonable quality starter code", then :-)
> 
> To my earlier point again, WG pages, WG Wiki pages, datatrackers, all 
> make sense to you and me.  Not to many people who will like to implement 
> our protocols without burying themselves deep into IETF lore [1].  When 
> I talk to developers at companies and students at universities, if they 
> have heard of IETF at all, it is mostly through knowing that some 
> organization called IETF produces these RFCs.  That's it.  Perhaps for 
> them that is enough.  And if you buy that argument, then the corollary 
> is that we should do everything in our power to make sure that they have 
> all of the information they need to implement the protocol from the RFC 
> itself.
>

---

From: Vijay Gurbani <[email-masked]>
Date: Thu, 20 Aug 2020 20:34:21 -0500

On Thu, Aug 20, 2020 at 6:05 PM Joel M. Halpern <[email-masked]> wrote:

> Vijay, I do not see how the code quality things is too work.  While I do
> not like the idea, if the community wantede it I could see the IETF
> providing repositories for any code that claims to implement  specific RFCs.
> Even if we were to do that, I can not see how or why we would want to  get
> into judging the quality of the code.   that sounds like a recipe  for
> disaster.
>

Dear Joel: You are absolutely right!  We do not want to get into the
business of judging the quality of the code.

My response to Melinda that included the "high quality code" phrase was
mostly driven by the Gen-ART review I did that triggered the thoughts I
have attempted to capture in this email thread.  The code pointers
mentioned in that I-D in the "Implementation Status" section appeared to be
of good quality.

But in general, if we embark on such a code repository, then we definitely
do not want to undertake a code beauty contest of any kind.

Thank you for your time.

Cheers,

- vijay

---

From: Michael Richardson <[email-masked]>
Date: Fri, 21 Aug 2020 12:50:10 -0400

> failures. The email lists are still in the archives. I followed it at the
    > beginning, but don't know much about it other than it was in part trying to
    > help match up implementors with standards.

It was more a mentee/mentor matching around the code/standard situation.
I get the impression that the effort just ran out of oomph.

---

From: Jane Coffin <[email-masked]>
Date: Fri, 21 Aug 2020 16:55:41 +0000

I hope I do not sound too clueless, but is the objective in creating a repository to educate future coders?
If yes, wouldn’t they see that already in running/operational code? Or, is the idea to show how code changed over time/got better?
If no, what would the repository do to educate folks new to the IETF protocols.

I ask these questions sincerely and with apologies if this is super clear and I missed the context.

On 8/21/20, 12:50 PM, "irsg on behalf of Michael Richardson" <[email-masked] on behalf of [email-masked]> wrote:

---

From: Michael Richardson <[email-masked]>
Date: Fri, 21 Aug 2020 18:52:07 -0400

> repository to educate future coders?

That was the proposal.
Some kind of archive.
Beyond the license issue and the appearance of endorsement,
there are many other issues.

It's not just getting a stack of disk space.
The expensive issue is, in my opinion, figuring out how things would be nominated to go in.

I don't think it has value.

The codestand project was about matching WG protocol experts with
implementors (imagined to be mostly students at various maturity levels) who
are trying to do something reseachy with the protocol.
More running code.

---

From: Vijay Gurbani <[email-masked]>
Date: Fri, 21 Aug 2020 18:01:01 -0500

Dear Michael, No, I am afraid that was not the proposal.  It is not about
educating coders.  My apologies, I am somewhat caught up in in other
matters right now, but I will reply later with more detail.

Thanks.

On Fri, Aug 21, 2020, 5:53 PM Michael Richardson <[email-masked]>
wrote:

>     > I hope I do not sound too clueless, but is the objective in creating
> a
>     > repository to educate future coders?
>
> That was the proposal.
> Some kind of archive.
> successful;
> implementation

---

From: Vijay Gurbani <[email-masked]>
Date: Fri, 21 Aug 2020 22:25:26 -0500

On Fri, Aug 21, 2020 at 11:56 AM Jane Coffin <[email-masked]> wrote:

> If yes, wouldn’t they see that already in running/operational code? Or, is
> the idea to show how code changed over time/got better?

Dear Jane: Good evening, and thank you for your note, and your interest.

So, let me start off by saying that the intent of what I proposed is *not*
to educate future coders.  That is the job of universities and bootcamps.

The genesis of what I am trying to affect is simple: We produce standards
that describe protocols.  As a protocol is being defined in an I-D, there
are often implementers implementing the protocol or portions of it.  These
implementations are important, enough that RFC 7942 [1] recognizes the
import of these implementations and exhorts I-D authors to create an
"Implementation Section" to document these implementations.  RFC 7942 also
asks I-D authors to remove this section before the I-D becomes an RFC.  My
question is if there are implementations that are worthy of being mentioned
in an "Implementation Section" of a I-D, why don't we preserve these
implementations in an IETF repository and link the repository in the RFC.
This way, new implementers of the protocol can quickly bootstrap their
implementations, leading to a diversity of implementations, which can only
be good for a protocol that is trying to find its footing.

Why is this important?  Simple: Reaching out to developers outside of those
of us who are already in the IETF is my biggest incentive.  They are the
ultimate consumers of what we produce.

When I talk to developers at companies and students at universities, if
they have heard of IETF at all, it is mostly through knowing that some
organization called IETF produces these things called RFCs.  That's it.
Perhaps for them that is enough.  And if you buy that argument, then the
corollary is that we should do everything in our power to make sure that
they have all of the information they need to implement the protocol from
the RFC itself.   Consider the errata we maintain in an RFC.  Imagine
instead of us providing the errata, we simply told implementers that it is
out there, go find it.  The errata is important, so we list it in an RFC.
When a protocol is finding its footing, early implementations that allow
other developers to understand its behaviour and derive their own
implementations from it is also important.  So why not formalize the
process of archiving implementations that are worthy of being listed as per
RFC 7942?

People have pointed out that we have WG pages, Wikis, datatrackers, and
other such accoutrements for this purpose?  WG pages, WG Wiki pages,
datatrackers, all make sense to you and me.  Not to many people who will
like to implement our protocols without burying themselves deep into IETF
to know the difference between a WG page and datatracker.  Their visibility
into the IETF is through the RFC.  So, put as much information in there
that will make a developer productive.

I don't claim that this will be easy; the length of this thread indicates
the passion of the participants around the issue.  However, I don't think
that the issues raised are insurmountable.

My initial email that started this thread is in [3].  It has more
information and context, including how organizations like IEEE and ACM have
dealt with similar issues.

If you have further questions, I am happy to answer them.

[1] https://tools.ietf.org/html/rfc7942
[2]
https://mailarchive.ietf.org/arch/msg/wgchairs/AGHRTKsDoSF3GJVhB4K2bKYjdWU/
[3]
https://mailarchive.ietf.org/arch/msg/wgchairs/8_Or3U-Wwjw35VmLy_uTPID9sTU/

Thanks,

- vijay