Subject: [newtrk] implementation / interoperability reports vs. ISDs

From: Larry Masinter <[email-masked]>
Date: Thu, 14 Jul 2005 13:06:25 -0700

I think Bruce's hypothetical problems with using
implementation reports are mainly "strawman"
objections. It's easy to set up some possible
failure cases, but I think it's also relatively
easy to route around those cases, too. 

In my thinking, a suite of "interoperability"
information for a "protocol" which covered the
multiple RFCs and features necessary to actually
be interoperable would fill the role that
people are imagining for ISDs: they'd tell
you the big picture.

And instead of putting effort into writing more
text for ISDs, consider the possibility of
putting the same effort into creating the schema
for implementation/interoperability reports.

I say "report", but I really mean something
more like a "database", since it is likely
that individual contributions will be most useful
when they are combined, summarized, or otherwise
reported on together.

For example, if you want interoperability reports
of IPv6, you might select a set of RFCs and a set
of "features" you think are relevant for IPv6,
that are found in interoperable IPv6 implementations.

A first cut on what a "feature list" might be
is to take the MUST/MAY/SHOULD items in the
standards documents, or to take sections or
paragraphs. Creating the feature schema takes
some work, of course, but it's work that is
usually done during interop events, in any case.

The schema itself might need "rough consensus" --
it wouldn't do to allow individual implementors
to create their own set of "features" to report
on. But individual implementors would be responsible
for their own report.

Each IPv6 implementation would give some
precise information about the implementation,
version, identity of person(s) making the report
(and asserting its authenticity) as well as
the precise identity of other independent
implementations against which this one has been
tested. For each "feature", there would be
a separate way of noting that the feature was
"implemented", as well as whether it was
"tested" with successful interoperability.

Since feature/implementation testing is pairwise,
you could cross-check whether implementation
A said it sucessfully tested feature P against
implementation B, but B didn't agree.

You might need some editing to decide whether
implementations were "independent" -- didn't
use the same code base, represented separate
exercises of license agreements when there were
IPR claims. But that information could be
supplied somewhat independently.

The fact that implementations change isn't too
serious a problem -- the user can rule out
old reports or not, depending on their needs.
And implementors can update their reports if
they want.

Larry

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 19 Jul 2005 18:22:22 -0400

On Thu July 14 2005 16:06, Larry Masinter wrote:
> objections.

The problems mentioned are not hypothetical; they are based on
current implementation reports.  You are of course free to
raise substantive issues ad argue to convince me otherwise,
but simply making an assertion and throwing around the term
"strawman" aren't going to convince me.

> people are imagining for ISDs: they'd tell

I doubt it.  Many applications protocols operate as a client/
server pair and provide a number of operational modes or sets
of functionality.  Many existing application implementations
use a small subset of features relevant to a number of protocols.

> when they are combined, summarized, or otherwise

Who is going to do the combining, summarizing, and/or otherwise
reporting?  Using what criteria?
 
> that are found in interoperable IPv6 implementations.

One of the problems that you brushed off with "strawman" is the
fact that different people may select different sets of RFCs.
How precisely do you propose to "route around" that problem?
 
> implementation B, but B didn't agree.

That doesn't work for client/server protocols; it is
possible for product A server to work well with product B
client, while product A might have no client implementation
or if it does, that implementation might not work with product
B's server (if one exists).
  
> old reports or not, depending on their needs.

So you think that each user should do the combining, summarizing,
etc. that you earlier mentioned?

> And implementors can update their reports if

What does any version of aby summary of any such set of reports
usefully say about the quality of protocol design or clarity of
the specification?
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Douglas Otis <[email-masked]>
Date: Tue, 19 Jul 2005 17:10:09 -0700

> use a small subset of features relevant to a number of protocols.

I wish to inject a few comments.  I like the ISD concept, but I think  
there can be a lighter weight version of this approach more readily  
achieved.  Rather than confusing these two approaches, I have called  
this slightly different configuration of virtually the same  
information, SRD.  The concept behind providing durable references in  
the SRD draft depends upon isolating disparate aspects of a specific  
endeavor, and not attempting to defined a set of RFCs needed for an  
application.  The RFCs should be allowed to overlap and not be  
limited to being within a single SRD.  See:

http://www.ietf.org/internet-drafts/draft-otis-newtrk-rfc-set-00.txt

This draft still needs a fair amount of work.  It was just to flesh- 
out the SRD idea.

> reporting?  Using what criteria?

The idea behind the SRD draft is that this information is kept to an  
absolute minimum, and should not represent a significant effort to  
maintain.  A benefit that flows from this small effort is a  
consolidation of any related reporting, discovered using a consistent  
reference.  The reports entered into a Status database should be  
"assigned" to either a current wg, or a consortium supporting  
applications making use of the RFCs referenced in the SRDs.  You will  
notice there is no author information allocated in the SRD, as this  
"assignment" information should be found via the Status database.   
This "assignment" process should be a dynamic function of the IESG,  
and independent of the SRD process itself.

> How precisely do you propose to "route around" that problem?

Set overlap should not become an issue.  The only implication of  
being within an SRD, is that the RFC is useful.  If there are  
different sets of RFCs that relate to different isolated endeavors,  
then overlapping RFCs within SRDs should be created.  This would  
indicate there are mutual endeavors sharing perhaps separate aspects  
within the set of RFCs.  This should be minimized by attempting to  
limit the scope of what is considered an endeavor.  The scope for an  
endeavor should exclude multi-faceted applications, but may consider  
both the client/server of a protocol to be within scope.  The Status  
database could be seen as supplementing or eventually supplanting the  
RFC categories.

> B's server (if one exists).

When there are issues that relate to specific applications,  
considered a superset of a particular endeavor, there can still be  
warnings and guidance offered within the Status database concerning  
these applications.  This type of information is valuable but should  
be reviewed by the IESG prior to posting within the public facing  
Status database.  As this type of implementation information can  
change rapidly, it does not lend itself to extensive review.  It  
should be expected this information will be updated as things change,  
so the review process should be kept to a minimum to encourage a  
higher rate of updating.

> etc. that you earlier mentioned?

My view would be those reporting are assigned by the IESG, and their  
output would be minimally reviewed by the IESG.  This would be  
considerably less formal than that needed for an RFC.  The purpose of  
this Status database is to provide information relevant to the use of  
the RFCs contained within the SRDs.  Only the RFC is able to define a  
protocol and there should be some wording to this effect seen when  
accessing the Status database.

>> And implementors can update their reports if

I would suspect the references used to access the IETF Status  
database would be replicated elsewhere, using less formal and more  
open procedures.  Perhaps more Wiki like.

> What does any version of any summary of any such set of reports

The SRD and Status database would help clarify what portions of any  
particular RFC is being used and how.  It could also serve to alert  
developers what portions are not used and perhaps why.  This could  
also consolidate security related information that could be made  
available via RSS feeds, for example.  This information should also  
prove helpful when generating newer RFCs.  Developers will be less  
likely overwhelmed should the rate of updating RFCs increase, as the  
SRD would provide them a stable reference they can use to keep  
abreast of these related changes.

This SRD draft does not prevent the continued publishing of STD  
indexes and the like.  This should be seen as something done in  
conjunction with these other existing efforts.  If this SRD approach  
proves to be easy to maintain and use, then perhaps, at a later date,  
these other methods of publishing can be reviewed.   I would advocate  
changing as little as possible until this approach is proved useful.

-Doug

.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 19 Jul 2005 20:25:45 -0400

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "Douglas" == Douglas Otis <[email-masked]> writes:
    Douglas> RFCs needed for an application.  The RFCs should be allowed
    Douglas> to overlap and not be limited to being within a single SRD.

  I didn't think that the ISD proposal required that a document be
mentioned in only a single ISD. Or did I miss something?

- -- 
] Michael Richardson          Xelerance Corporation, Ottawa, ON |  firewalls  [
] mcr @ xelerance.com           Now doing IPsec training, see   |net architect[
] http://www.sandelman.ca/mcr/    www.xelerance.com/training/   |device driver[
]                    I'm a dad: http://www.sandelman.ca/lrmr/                 [
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Douglas Otis <[email-masked]>
Date: Tue, 19 Jul 2005 18:12:39 -0700

> mentioned in only a single ISD. Or did I miss something?

These comments were specially about the SRD draft, while I consider  
the ISD to contain virtually the same information.  However, the ISD  
approach imposes almost the same rigidity as publishing an RFC, with  
added levels of complexity dealing with use and category issues.  The  
SRD approach would distill the effort expended to create the document  
to a minimum.  It would not contain information related to RFC  
categories or use.  There is only the SRD title, a simple SRD  
description, references to those RFCs contained within a set, which  
would be classified as:

  - core
  - extensions
  - guidance
  - obsoletes
  - experimental
  - companion SRDs

An SRD document would look like:

  <?xml version='1.0' encoding="UTF-8"?>
  <!DOCTYPE srd SYSTEM "rfcxxxx.dtd" >
  <?xml-stylesheet type='text/xsl' href='rfcxxxx.xslt' ?>
  <?srd sortrefs="yes"?>
  <?srd strict="yes" ?>
  <srd set=srd-xml.1 >
  <title>XML SRD</title>
  <description>XML structure for SRDs.</description>
  <srdDate>2005-07-16T19:20:30+01:00</srdDate>
  <docs>http://www.ietf.org/</docs>
  <core>
    <?srd include="reference.RFC.9876" ?>
    <?srd include="reference.RFC.9875" ?>
  </core>
  <extensions>
    <?srd include="reference.RFC.9912" ?>
  </extensions>
  <guidance>
    <?srd include="reference.RFC.9915" ?>
  </guidance>
  <obsoletes>
    <?srd include="reference.RFC.9811" ?">
  </obsoletes>
  <experimental>
    <?srd include="reference.RFC.9837" ?">
  </experimental>
  <companion>
    <?srd include="reference.SRD.srd-example.1" ?>
  </companion>
  </srd>

The difference between the ISD approach, and the SRD is that the use  
and status information presented within the ISD is relegated to being  
published within a more dynamic and less formal Status database.  The  
intent is also to include all useful RFCs and not just elevate some  
RFCs.  It should be practical to generate much of the SRD documents  
via scripts.  The information within the Status database would  
accumulate over the use of the related RFCs.

-Doug

.

---

From: John C Klensin <[email-masked]>
Date: Thu, 21 Jul 2005 18:13:12 -0400

--On Tuesday, 19 July, 2005 20:25 -0400 Michael Richardson
<[email-masked]> wrote:

>   I didn't think that the ISD proposal required that a
> document be mentioned in only a single ISD. Or did I miss
> something?

You did not.  There is no such restriction.

The key difference, IMO, is the one Doug noted in his response:

ISDs are more formal and more structured than what he proposes,
and their normative content is subject to a formal approval
process, including a Last Call.  They replace the "STD" model
and are intended to be sufficiently stable and appropriate for
referencing that they can be used in procurement and other
formal external specifications, not just as a guide to things
that might go together.   They are definitely more heavyweight,
if only because they are sufficiently tied to the standards
process that, for standards-track documents, the ISDs would be
Last Called (and I-Ds that will become RFCs only a part of the
package).  That contrasts with the present situation, which the
SRD concept presumably would not change, rather than putting
collections of individual documents through Last Call without
explicitly binding them together.  

The ISDs also would become an appropriate way to create and
locate authoritative/ normative errata and other small changes
and, partially as a consequence, for moving documents and
standards up in maturity designation without having to reissue
them in every case and might have some hope for finally getting
us unstuck from the general external assumption that all RFCs
are standards.  Again, the SRDs don't seem to address those
issues.

To get those benefits, one "pays" in somewhat more procedure and
a formal new document series.  Whether the benefits are worth it
is, of course, up to the community.

    john

.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 21 Jul 2005 17:40:03 -0700

> are standards.  Again, the SRDs don't seem to address those

The concept behind the use of SRDs was to have this rather a simple  
document submitted and considered together in tandem with the I-D  
submitted to the IESG for action.  Requiring the SRD document added  
should not burden this process.  To the contrary, it would instead  
provide greater overview for all involved what is being affected.

The SRD would allow stable references for authoritative RFCs (the  
only authoritative source) rather than attempting to supplement this  
information.  SRDs facilitate an association and classification of  
related RFCs.  Any information to supplicate that found in RFCs would  
be considered topical, such as security issues, incompatibilities  
discovered with the latest releases, etc.  A reference using ISD or  
SRD names would provide the same utility in centralizing where the  
Status information is found.  I see the difference between ISD and  
SRD is the amount of glue required.  Again, as the RFCs are the  
authoritative source, SRDs actually create less risk of conflicting  
with what is considered authoritative.

-Doug
.

---

From: Michael Richardson <[email-masked]>
Date: Fri, 22 Jul 2005 19:45:48 -0400

>>>>> "John" == John C Klensin <[email-masked]> writes:
    John> --On Tuesday, 19 July, 2005 20:25 -0400 Michael Richardson

    >> mentioned in only a single ISD. Or did I miss something?

    John> You did not.  There is no such restriction.

    John> The key difference, IMO, is the one Doug noted in his response:

  thanks.

  So I think that SRDs wouldn't be formal enough to be useful for RFPs.

  The most serious difficulty in getting Canadian Government procurement
process for computing equipement to adhere to the NAFTA treaty, is that few
government departments have any idea how to specify anything that they need
in terms of RFCs.
  It's a hard problem for them --- ISDs would solve this.

--

---

From: John C Klensin <[email-masked]>
Date: Sat, 23 Jul 2005 17:53:11 -0400

--On Friday, 22 July, 2005 19:45 -0400 Michael Richardson
<[email-masked]> wrote:

>   The most serious difficulty in getting Canadian Government
> procurement process for computing equipement to adhere to the
> NAFTA treaty, is that few government departments have any idea
> how to specify anything that they need in terms of RFCs.

And, when private companies try it, they often don't know that
they don't have any idea how to do this, and we see procurement
specs that require support for RFC 1149 and 2549 on the theory
that they want support for IP over every possible medium and
want QoS to boot.

>   It's a hard problem for them --- ISDs would solve this.

That was the intent.  The question, I think, remains whether it
is worth it.   I guess we try to argue about that in Paris.

   john

.

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 24 Jul 2005 09:49:31 -0400

> is worth it.

There are two questions: Where does one draw the line regarding what
is or is not excluded (presumably 1149 and 2549 would both be excluded
unless one is writing an April 1 ISD), and who is going to do the work.
It is not inconceivable that an ISD for a particular technology might
include various media and low-level technologies as well as issues
such as QoS.  ISDs are no silver bullet; to cover various uses, they
would have to be fairly comprehensive and each user would have to
select the specific RFCs applicable to the particular RFP or other
intended use.  Having such a comprehensive list would be a useful tool
inasmuch as (if it is truly comprehensive) it would help to limit
errors of omission (and if the ISDs omit the April ! RFCs, silly errors
of inclusion), but it won't be a substitute for careful thought and
selectivity when writing RFPs etc.

Application technologies are probably the most difficult to handle
as they utilize many lower-level technologies which in turn use
still lower-level technologies (reminiscent of aphorisms about fishes,
fleas, and turtles).  In theory, the higher-level ISDs could refer to
lower-level ISDs rather than individual RFCs, but it's not quite that
simple:
o a given application-level technology (e.g. mail) may use some features
  of a lower-level technology such as DNS (e.g. MX RRs) but not all
  features.
  - perhaps the lower-level ISDs could be organized into sections, so
    that higher-level ISDs could reference the particular sections of
    relevance
    * that would require considering all ISDs together when deciding
      how to organize the low-level ISDs (so that they are partitioned
      for reference by high-level ISDs)
    * and unfortunately what is or is not relevant tends to change over
      time and depending on circumstances (e.g. at this time certain
      types of DNS RRs are only of interest to mail implementers wishing
      to participate in certain experiments; those who wish to participate
      need to handle the RRs, and in the (IMO unlikely) event that the
      experiments lead to standardization, that may apply to all mail
      implementations).
o different technologies use different features of lower-level technologies;
  partitioning the low-level ISDs for suitable use by multiple higher-level
  ISDs will be challenging.  In the worst case, the partitioning will be at
  the individual RFC level, in which case one wonders how much advantage
  there is to reference to lower-level ISDs as opposed to referencing the
  individual RFCs in the high-level ISDs, especially considering how much
  effort would be required to partition the low-level ISDs.
o ultimately, while a particular technology may involve many possible
  underlying technologies (mail uses DNS; DNS runs on TCP and/or UDP,
  TCP and UDP run on IP, IP can run on various types of media and lower-
  level technologies such as Ethernet, token-ring, optical media, wireless),
  a given RFP for an organization which has a particular need and a given
  existing infrastructure need not be concerned with IP over token-ring if
  the organization has no token-ring networks.  With partitioned ISDs
  referenced by higher-level ISds, how does an organization specify which
  parts of a low-level ISD 4 levels of reference down from the application
  level ISDs are or are not applicable?

Some problems -- putting together an RFP that specifies everything necessary
but no more -- are inherently difficult.  While ISDs might be a tool to
help in accomplishing difficult tasks, they can't by themselves "solve"
the problem.  An organization which needs to prepare an RFP or accomplish
any similarly difficult task either needs to have competent staff which is
available to do the work or needs to contract outside expertise to do so.
As it was in the beginning, is now, and will be even with ISDs.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: John C Klensin <[email-masked]>
Date: Sun, 24 Jul 2005 11:13:58 -0400

--On Sunday, 24 July, 2005 09:49 -0400 Bruce Lilly
<[email-masked]> wrote:

> There are two questions: Where does one draw the line
> regarding what is or is not excluded (presumably 1149 and 2549
> would both be excluded unless one is writing an April 1 ISD),
> and who is going to do the work. It is not inconceivable that
> an ISD for a particular technology might include various media
> and low-level technologies as well as issues such as QoS.
> ISDs are no silver bullet; to cover various uses, they would
> have to be fairly comprehensive and each user would have to
> select the specific RFCs applicable to the particular RFP or
> other intended use.  Having such a comprehensive list would be
> a useful tool inasmuch as (if it is truly comprehensive) it
> would help to limit errors of omission (and if the ISDs omit
> the April ! RFCs, silly errors of inclusion), but it won't be
> a substitute for careful thought and selectivity when writing
> RFPs etc.
> 
> Application technologies are probably the most difficult to
> handle as they utilize many lower-level technologies which in
> turn use still lower-level technologies (reminiscent of
> aphorisms about fishes, fleas, and turtles).  In theory, the
> higher-level ISDs could refer to lower-level ISDs rather than
> individual RFCs, but it's not quite that simple:
>...

Bruce,

(1) There are no magic bullets, for this or any other
non-simplistic problem.

(2) ISDs cannot reasonably be expected to substitute for either
thinking or whole-system analysis.   The question "what
documents make up the Internet's email standard?" is much easier
to answer, and generally much more sensible, than "top to
bottom, what documents and protocols are needed to support
email?".  Fortunately, in the real world, plausible procurement
specifications need the answer to the first question and not the
second.

(3) The quest for, or insistence on, a perfect solution will
always prevent development and deployment of approaches that
make things incrementally better.   One can always say "there
are issues this will not address and problems it will not
solve".  Whether that is constructive or not is often another
question entirely and depends on whether, e.g., one's interest
is in making a situation better or in killing a proposal on
principle.

My personal conclusion is that Doug's idea would move us forward
from, and have advantages over, our present situation.  The ISD
approach would, as Michael pointed out, do a better job of
helping with the procurement issue and, I would suggest, with a
few others.  Relative to Doug's proposal, the ISD one would also
involve some incremental costs.  Those costs might perhaps not
as be great as Doug and others assume, at least in some areas:
with appropriate definitions and flexibility, if one type of
document can be automatically generated, so can the other.  But
no one has suggested that the costs of ISDs are higher than the
costs of something that is mechanically produced and that just
lists documents rather than containing text.  The questions,
again, are

	* whether the marginal value of, and opportunities
	created by, ISD is worth the marginal costs over a
	lighter-weight and narrower approach... and 
	
	* whether the value of that lighter-weight and narrower
	approach is sufficient to justify the incremental costs
	over doing nothing or leaving the "RFC grouping" problem
	to the commercial marketplace (see, e.g., Eric Hall's
	books) 

But "not a silver bullet" does not seem to me to be a useful
contribution to the discussion.   Just IMO, of course.

     john

.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 24 Jul 2005 16:06:32 -0500

> contribution to the discussion.   Just IMO, of course.

... since the Paris agenda item for ISDs focuses on whether ISDs are 
bullets at all, it seems that we could defer the metallurgy analysis 
until after Paris? :-)

Spencer 

.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 25 Jul 2005 12:03:29 -0400

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:
    Bruce> There are two questions: Where does one draw the line regarding
    Bruce> what is or is not excluded (presumably 1149 and 2549 would both be
    Bruce> excluded unless one is writing an April 1 ISD), and who is going
    Bruce> to do the work.  It is not inconceivable that an ISD for a

  Both good questions.

  I think that we can cover 90% of what the IETF thinks of standard in a
relatively short amount of time.  Getting a couple of reviews from past
chairs of relevant working groups (probably now concluded) should be
relatively easy. There would not initially be a lot of information.

  As to who will do the work on an ongoing basis --- I think that this is
something that procurement people will be interested in doing. That is a
group of people that we have not, to date, had a lot of involvement from.

(The OPSEC WG is a one of the refreshing new things. A lot of people at
pwgsc.gc.ca were interested to learn about the group. I think that there are
still political obstacles to them getting involved, but they are going away
slowly)

    Bruce> omission (and if the ISDs omit the April ! RFCs, silly errors of
    Bruce> inclusion), but it won't be a substitute for careful thought and

  I have, btw, code for Subliminal Message option in a telnet proxy. 
  It was written by a new-hire back in 1994...

    Bruce> Application technologies are probably the most difficult to handle
    Bruce> as they utilize many lower-level technologies which in turn use
    Bruce> still lower-level technologies (reminiscent of aphorisms about
    Bruce> fishes, fleas, and turtles).  In theory, the higher-level ISDs

  Yes, do you reference the TCP and IP RFCs in the SMTP or HTTP ISD?

  The answer is no, I think. We will have to have a hierarchy of documents,
but I think that we can probably put a big seam with words like "reliable
transport" instead of saying "TCP".

    Bruce> as DNS (e.g. MX RRs) but not all features.  - perhaps the
    Bruce> lower-level ISDs could be organized into sections, so that
    Bruce> higher-level ISDs could reference the particular sections of

  No.

    Bruce> Some problems -- putting together an RFP that specifies everything
    Bruce> necessary but no more -- are inherently difficult.  While ISDs
    Bruce> might be a tool to help in accomplishing difficult tasks, they
    Bruce> can't by themselves "solve" the problem.  An organization which
    Bruce> needs to prepare an RFP or accomplish any similarly difficult task
    Bruce> either needs to have competent staff which is available to do the
    Bruce> work or needs to contract outside expertise to do so.  As it was
    Bruce> in the beginning, is now, and will be even with ISDs.  .

  I think that ISDs will help a great deal.