Subject: Re: [ippm] review of draft-ietf-ippm-metric-registry-03

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 14 Sep 2015 18:46:29 -0400

Hi Barbara, Thanks for your review and comments.

please see a reply to one comment below,
Al

> -----Original Message-----
...
> 4. Scope
> FROM
> This document is meant for two different audiences.
> TO
> This document is meant for three different purposes.
> 
> ADD the following as penultimate paragraph:
> The third purpose is to define a Performance Metrics Registry format
> that will be used by the IANA Performance Metrics Registry and other
> Performance Metrics Registries. This will be particularly useful to
> implementation of the Large-scale Measurement Platform framework
> [framework reference] and information model [information model
> reference], who will need to use registries for Performance Metrics
> defined by other entities.
> 

[ACM] 
One key theme in your comments is the proposal to separate the registry format from the IANA Registry creation/process, and possibly re-use the format elsewhere. There's quite a bit of editorial work to accomplish the separation and include the additional goal of a separable registry format.

Perhaps the goal of a re-usable format can be accomplished with far fewer edits by saying that the format *may* serve other purposes, rather than making re-use an equal goal (at the end of section 2, Introduction):

Barbara's proposed re-wording:
This document therefore creates an IANA-maintained Performance Metrics Registry and a Performance Metrics Registry format (that will be usable by IANA and any other organization which wishes to create a Performance Metrics Registry).

NEW
This document therefore requests that IANA create and maintain a Performance Metrics Registry, according to the maintenance procedures and the Performance Metrics Registry format defined in this memo. Although the Registry format is primarily for use by IANA, any other organization that wishes to create a local Performance Metrics Registry MAY use the format for its purposes.
<then add something like:
The authors make no guarantee of the format's applicability to *any* possible set of Performance Metrics envisaged by other organizations.

I think this is effectively the same outcome. We probably have some words about re-use because I remember Phil was an advocate of that possibility.

I discussed this approach with the co-authors. We don't want the additional goal of a re-usable format to open a new can of worms that we never thought-of ...  that would be tragic at this point. By making re-use a MAY, we fend-off any expansions someone might dream-up more easily, and keep the finish line in sight.

Can you live with this compromise?

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Wed, 16 Sep 2015 13:56:34 +0000

> This document therefore creates an IANA-maintained Performance Metrics
> Registry and a Performance Metrics Registry format (that will be usable by
> IANA and any other organization which wishes to create a Performance
> Metrics Registry).
> 
> NEW
> This document therefore requests that IANA create and maintain a
> Performance Metrics Registry, according to the maintenance procedures and
> the Performance Metrics Registry format defined in this memo. Although the
> Registry format is primarily for use by IANA, any other organization that
> wishes to create a local Performance Metrics Registry MAY use the format
> for its purposes.
> The authors make no guarantee of the format's applicability to *any*
> possible set of Performance Metrics envisaged by other organizations.
> >
> 
> I think this is effectively the same outcome. We probably have some words
> about re-use because I remember Phil was an advocate of that possibility.
> 
> I discussed this approach with the co-authors. We don't want the additional
> goal of a re-usable format to open a new can of worms that we never
> thought-of ...  that would be tragic at this point. By making re-use a MAY, we
> fend-off any expansions someone might dream-up more easily, and keep
> the finish line in sight.
> 

Hi Al (and Marcelo),
Sorry it took me a couple of days to reply. I needed to go back and look at the detailed comments I provided the authors (in a separate email sent just to the authors) to see if I could figure out why my suggestion for this draft to have a goal of creating a common registry format is objectionable. Given I provided (in that email to the authors) a very detailed proposal for dealing with my comments, I have to disagree with Marcelo that what I suggest would add years to the approval process for metric-registry.

I believe that IETF could provide a big service to the industry as a whole if it presented metric-registry as defining a format explicitly intended to be usable by others. It would then be possible for lmap-information-model to normatively reference metric-registry, instead of informatively. A normative reference in lmap-information-model would mean that software could be written to follow the registry URI (that is a data element in lmap-information-model ) and pull out certain information for presentation to the user or to act upon. Having the format of a registry entry behind an lmap-information-model registry URI be non-standard weakens the usefulness of the overall lmap solution significantly. 

This weakened overall solution is particularly unfortunate, since all it would take (IMO) would be to state that a goal of metric-registry is to provide a standard format, and then normatively reference metric-registry in lmap-information-model as specifying the required (or strongly recommended, i.e., SHOULD) registry format for a registry URI. And doing this doesn't impact the proposed format in metric-registry at all. In all but one instance, the changes I suggested were to text that describes the purpose of the document. The only normative element of registry-format that I proposed modifying was the mandatory naming requirement of inserting "Act_" or "Pas_" in front of the name.

But if people also intend to be steadfast in refusing to have metric-registry as a normative reference in lmap-information-model, then I don't really care whether any changes are made to metric-registry to suggest this as a distant possibility. Without a normative reference in lmap-information-model, my suggestion is irrelevant. The lmap platform will be weaker than hoped. It would take significantly more work to get to the level of standardization that could have been achieved by these changes, and I think it's unlikely anyone will bother with that.

I won't stand in the way if I'm alone in believing that a standardized registry format would make the overall lmap solution much stronger.
Barbara

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 16 Sep 2015 19:41:55 -0400

Hi Barbara,

Let's see if we can sort this out. Sorry for the long message.

Co-authors don't want to add a *goal* that the Registry format be re-used,
because:
- it potentially adds an unknown amount of work to meet that goal.
  work on the registry has been in-progress for almost 3 years now.
- we could be asked to liaise the registry with some number of SDOs,
  wait for their feedback, and implement the changes they want.
  I can imagine that process taking months alone, just to get feedback.
- we are completely fine if others use this exact Registry Format,
  but we can't assure them it will be fit for all imaginable purposes.
  If the amount of time we spent on this is any indication, we should
  have extremely good coverage for network performance measurements.

Adding the goal doesn't change the technical content
(immediately), but somehow allows the Registry to become a normative
reference in the LMAP Information Model draft. I don't understand how
this works:

you wrote:
> I believe that IETF could provide a big service to the industry as a
> whole if it presented metric-registry as defining a format explicitly
> intended to be usable by others. It would then be possible for lmap-
> information-model to normatively reference metric-registry, instead of
> informatively. A normative reference in lmap-information-model would
> mean that software could be written to follow the registry URI (that is
> a data element in lmap-information-model ) and pull out certain
> information for presentation to the user or to act upon. ...

We can't require others to use our registry format in their local registries
from the Registry draft itself.

But it seems to me that adding an Information Model requirement to use a 
URI identical to the one in the Registry, and to connect the Info Model 
URI to the URI and the related the Standardized Registry Format entry through a 
normative reference is sufficient.

> ... Having the
> format of a registry entry behind an lmap-information-model registry URI
> be non-standard weakens the usefulness of the overall lmap solution
> significantly.

I'm having trouble following the sentence above:
The Registry Format is part of a Best Current Practice doc., certainly
not Informational or Experimental.  When other SDOs follow the format of
the BCP, there's no problem for the URI to link to the right registry entry.

It seems like we need a very carefully crafted requirement in the 
LMAP Information Model. A standard URI linked to the URI in the Registry
and the format of Registry entries by a normative reference.

So...
Tell me where the copied URI and Reference doesn't work, 
or how we can *encourage* Registry Format
adoption beyond IANA without doing more than saying it, 
explicitly, in the draft.

regards,
Al

> From: STARK, BARBARA H
> Sent: Wednesday, September 16, 2015 9:57 AM
> To: MORTON, ALFRED C (AL); [email-masked]
> Subject: RE: review of draft-ietf-ippm-metric-registry-03
> 
> > Registry and a Performance Metrics Registry format (that will be
> > usable by IANA and any other organization which wishes to create a
> > Performance Metrics Registry).
> >
> > NEW
> > Performance Metrics Registry, according to the maintenance procedures
> > and the Performance Metrics Registry format defined in this memo.
> > Although the Registry format is primarily for use by IANA, any other
> > organization that wishes to create a local Performance Metrics
> > Registry MAY use the format for its purposes.
> > I think this is effectively the same outcome. We probably have some
> > words about re-use because I remember Phil was an advocate of that
> possibility.
> >
> > I discussed this approach with the co-authors. We don't want the
> > additional goal of a re-usable format to open a new can of worms that
> > we never thought-of ...  that would be tragic at this point. By making
> > re-use a MAY, we fend-off any expansions someone might dream-up more
> > easily, and keep the finish line in sight.
> >
> Sorry it took me a couple of days to reply. I needed to go back and look
> at the detailed comments I provided the authors (in a separate email
> sent just to the authors) to see if I could figure out why my suggestion
> for this draft to have a goal of creating a common registry format is
> objectionable. Given I provided (in that email to the authors) a very
> detailed proposal for dealing with my comments, I have to disagree with
> Marcelo that what I suggest would add years to the approval process for
> metric-registry.
> 
> I believe that IETF could provide a big service to the industry as a
> whole if it presented metric-registry as defining a format explicitly
> intended to be usable by others. It would then be possible for lmap-
> information-model to normatively reference metric-registry, instead of
> informatively. A normative reference in lmap-information-model would
> mean that software could be written to follow the registry URI (that is
> a data element in lmap-information-model ) and pull out certain
> information for presentation to the user or to act upon. Having the
> format of a registry entry behind an lmap-information-model registry URI
> be non-standard weakens the usefulness of the overall lmap solution
> significantly.
> 
> This weakened overall solution is particularly unfortunate, since all it
> would take (IMO) would be to state that a goal of metric-registry is to
> provide a standard format, and then normatively reference metric-
> registry in lmap-information-model as specifying the required (or
> strongly recommended, i.e., SHOULD) registry format for a registry URI.
> And doing this doesn't impact the proposed format in metric-registry at
> all. In all but one instance, the changes I suggested were to text that
> describes the purpose of the document. The only normative element of
> registry-format that I proposed modifying was the mandatory naming
> requirement of inserting "Act_" or "Pas_" in front of the name.
> 
> But if people also intend to be steadfast in refusing to have metric-
> registry as a normative reference in lmap-information-model, then I
> don't really care whether any changes are made to metric-registry to
> suggest this as a distant possibility. Without a normative reference in
> lmap-information-model, my suggestion is irrelevant. The lmap platform
> will be weaker than hoped. It would take significantly more work to get
> to the level of standardization that could have been achieved by these
> changes, and I think it's unlikely anyone will bother with that.
> 
> I won't stand in the way if I'm alone in believing that a standardized
> registry format would make the overall lmap solution much stronger.
> Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 17 Sep 2015 14:04:32 +0000

> Co-authors don't want to add a *goal* that the Registry format be re-used,

Would it be possible for the document to at least say that it defines a Performance Metrics Registry format in addition to defining the IANA Performance Metrics Registry?  Make the format definition a distinct and important aspect of the draft? No need to say that a reason is for re-use. Just that the document defines a Performance Metrics Registry format. That makes it easier for others to reference just the format. From an external org perspective, referencing a format in a document that doesn't even acknowledge in its abstract, introduction, and scope that it defines a format is very awkward.

And then (preferably) recommend (no need to capitalize) that anyone wanting to create a registry that is intended to be used within the context of the LMAP framework and information model use the same format.

If not willing to "recommend", then don't use MAY (which suggests "you can if you want, but we don't know why you'd want to and don't really see a benefit to it"). Say rather "can". Anyone wanting to create a registry that is intended to be used within the context of the LMAP framework and information model can use the same format.

But then also maybe mention that there are benefits to doing this, like: Using the same format can benefit developers of Measurement Agents by allowing them to develop code that can parse entries in any Registry that uses this format.

But the big thing really is just acknowledging that the document defines a Registry format. It would be great to mention that the format can be used by other registries, but that wasn't really the biggest thing I was trying to accomplish.

Does this bring us any closer?
Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 17 Sep 2015 22:49:45 +0000

> Co-authors don't want to add a *goal* that the Registry format be re-used,

I've thought some more, and would like to suggest the following, much briefer, slightly modified, and more concrete proposal. This is mostly consistent with the previous proposal.

1. Acknowledge in Abstract, Introduction, and Scope that the document defines a Registry format. [This is really important, in order for other orgs to feel comfortable referencing just the format.]
2. Mention in the Introduction that while the Registry format is being defined for the IANA Registry, other Registries may choose to use it as well. (that's the non-normative may)
3. Also in the Introduction, mention that others who are creating Registries to be used in the context of an LMAP framework are encouraged to use the Registry format, because this makes it easier for developers of LMAP Measurement Agents (MAs) to programmatically use information found in those other Registries' entries.

Barbara

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 17 Sep 2015 23:41:40 -0400

Thanks Barbara, this helps a lot and I think the co-authors will
probably agree.  See replies below (which are hopefully readable,
it's getting late here...).

regards,
Al

> Sent: Thursday, September 17, 2015 6:50 PM
> I've thought some more, and would like to suggest the following, much
> briefer, slightly modified, and more concrete proposal. This is mostly
> consistent with the previous proposal.
> 
> 1. Acknowledge in Abstract, Introduction, and Scope that the document
> defines a Registry format. [This is really important, in order for other
> orgs to feel comfortable referencing just the format.]
[ACM] 
I think that's fine, at least the text I proposed for the Intro (Monday) did that.
It's fairly easy to separate the request to create the Registry,
the format, and the process to admin the registry, because they're
all in different sections. There's some editing to do, but it seems 
reasonable to me.

> 2. Mention in the
> Introduction that while the Registry format is being defined for the
> IANA Registry, other Registries may choose to use it as well. (that's
> the non-normative may) 
[ACM] 
Yes, also in the text I proposed for the Intro.

> 3. Also in the Introduction, mention that others
> who are creating Registries to be used in the context of an LMAP
> framework are encouraged to use the Registry format, because this makes
> it easier for developers of LMAP Measurement Agents (MAs) to
> programmatically use information found in those other Registries'
> entries.
> 
And that's what us hockey players call a "hat trick", and as you suggest,
we'll s/MAY/are encouraged to/
and add the last part of your sentence above
[ACM] 

NEW (from Monday, yet to be modified).
This document therefore requests that IANA create and maintain a Performance Metrics Registry, according to the maintenance procedures and the Performance Metrics Registry format defined in this memo. Although the Registry format is primarily for use by IANA, any other organization that wishes to create a local Performance Metrics Registry MAY use the format for its purposes.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 18 Sep 2015 06:55:55 +0200

I am ok with this (or something in these lines)

thanks, marcelo

El 18/09/15 a las 05:41, MORTON, ALFRED C (AL) escribió:
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>
>