Subject: [arch-d] draft-iab-rfc7500-bis-00 - new principle of measureability

From: Jay Daley <[email-masked]>
Date: Thu, 27 Jun 2019 09:44:22 +1200

I support the principles as documented but feel that there is one more that needs to be added:

* Measurability
The operations of IANA registries must include the collection and publication of sufficient data, in accordance with the other principles of openness and transparency, to allow their performance, both current and historical, to be measured and to assess whether they are  delivering their services in accordance with the other principles and the registry policies.

My reasoning to add this as a specific principle is because any assessment of the performance of an IANA registry requires data.  However, I have too often come across the situation where openness, transparency and even accountability are insufficient to ensure that this happens.  Common excuses include "we don’t collect the data", "the data belongs to our customers not us", "the data is unreliable", "the data we collect doesn’t measure our performance" and so on.  Adding this as an explicit principle ensures that data is collected and that it is fit for purpose and collected under the terms that allow it to be used for this purpose. 

As a background to this comment I should note that I was part of the team that designed the naming functions SLA for PTI and an inaugural member of the Customer Standing Committee within ICANN that assesses and reports on PTI’s delivery of the IANA naming function in accordance with the SLA.

cheers
Jay

-- 
Jay Daley
[email-masked]
+64 21 678840
skype: jaydaley
twitter: @jay_daley

---

From: Geoff Huston <[email-masked]>
Date: Wed, 26 Jun 2019 23:02:21 +0000

OK, I’ll bite.

What _exactly_ do you mean by "the performance of an IANA registry”?

> On 27 Jun 2019, at 7:44 am, Jay Daley <[email-masked]> wrote:
> 
> _______________________________________________
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Jun 2019 11:15:14 +1200

> On 27/06/2019, at 11:02 AM, Geoff Huston <[email-masked]> wrote:
> 
> OK, I’ll bite.
> 

Performance in this context is the measurement of processes and services provided by the IANA registry.  This is then assessed against expectations set in policies.

For example,

- Availability of the well known locations for protocol identifiers.  This could be measured by recording each outage. 

- Responsiveness of publication.  This could be measured by recording how long it takes from the publication of an RFC that includes a new protocol identifier for it to be registered and published.

cheers
Jay

-- 
Jay Daley
[email-masked]
+64 21 678840
skype: jaydaley
twitter: @jay_daley

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 27 Jun 2019 00:19:01 +0100

On 27/06/2019 00:15, Jay Daley wrote:
> - Responsiveness of publication.  This could be measured by recording
> how long it takes from the publication of an RFC that includes a new
> protocol identifier for it to be registered and published.
It happens the other way about in all cases where RFCs register
new codepoints - the IANA registration happens before the RFC
issues. The time taken for that vs. the rest of RFC production
is IIRC negligible.

S.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 27 Jun 2019 11:31:16 +1200

Jay,

An interesting topic no doubt, but as I understand it this draft is narrowly
scoped to the changes required by the creation of IETF LLC and the closing
of the IAOC. So other changes should really be left for future work.

Also, the actual details of how the IETF-IANA performance is measured are
in the formal agreement between the IETF and ICANN. The latest version:
https://www.ietf.org/documents/224/2018ICANN-IETFMoUSupplementalAgreement.pdf
and there are performance reports to be found if you search for them.

Regards
   Brian

On 27-Jun-19 11:15, Jay Daley wrote:
> 
> 
>> On 27/06/2019, at 11:02 AM, Geoff Huston <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> OK, I’ll bite.
>>
>>> On 27 Jun 2019, at 7:44 am, Jay Daley <[email-masked] <mailto:[email-masked]>> wrote:
>>>
>>> [email-masked] <mailto:[email-masked]>
>>> +64 21 678840
>>> skype: jaydaley
>>> twitter: @jay_daley
>>>
> [email-masked] <mailto:[email-masked]>
> +64 21 678840
> skype: jaydaley
> twitter: @jay_daley
> 
>

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Jun 2019 12:08:04 +1200

Thanks Brian

I was trying to keep this strictly at the principle level and a gap I see in the proposed principles, rather than get into any details of how performance expectations are set or measured.

cheers
Jay

> On 27/06/2019, at 11:31 AM, Brian E Carpenter <[email-masked]> wrote:
> 
> Jay,
> 

-- 
Jay Daley
[email-masked]
+64 21 678840
skype: jaydaley
twitter: @jay_daley

---

From: John C Klensin <[email-masked]>
Date: Thu, 27 Jun 2019 19:35:45 -0400

--On Thursday, June 27, 2019 12:08 +1200 Jay Daley
<[email-masked]> wrote:

> Thanks Brian
> 
> I was trying to keep this strictly at the principle level and
> a gap I see in the proposed principles, rather than get into
> any details of how performance expectations are set or
> measured.

Jay,

Please take this as a comment about the principle and not about
this particular I-D -- I'd be very opposed to altering the
document beyond the current scope of changes even if there were
agreement on your proposed principle, which, at least IMO, there
isn't.

There is a long and unfortunate history about how efforts to
quantify and measure complex administrative procedures (or
complex procedures with significant administrate components)
tend to work out.   There are inevitably some things that are
easy to quantify and measure.  Using IANA as an example,
consider registration lag (time between the arrival of a
registration request and its appearance in the relevant
registry), registrations completed per month, and so on.  Other
things are much harder to measure.  For example, requests
sometime come along that ask IANA to reorganize a registry.
While one could count the number or frequency of such requests,
that would ultimately measure the behavior of the body making
the requests -- the only thing it measures about IANA is their
ability to read minds and forecast the future, neither of which
should be anything we try to write into a contract (I hope for
obvious reasons).  But what they happens in practice is that the
things that are easy to quantify and measure get redefined as
being _the_ performance criteria and the other things tend to
get submerged and lost.  

That can be pathological because it may reward bad behavior and
discourage good behavior.   For example, we have "IANA
Considerations" sections in documents so IANA can review what
they are being asked to do before the IETF finalizes the
request.   Obviously, we don't want them to take any longer to
make those evaluations than necessary.  But it is far more
important that the make them with great care and as much thought
and dialogue as needed.  If they are encouraged (or forced) to
race through things by being evaluated based on throughput and
don't spot important issues as a result, they lose, the IETF
loses, and the users of the resulting registries and database
lose most of all.

So I suggest that trying to enshrine "let's find things to
measure and take those measurements" as a principle -- and it is
where "include the collection and publication of sufficient data
...to allow their performance ...to be measured and to assess
whether they are delivering their services" has a long history
of taking us and others -- is, by itself, a bad idea.

Now those concerns about misuse of quantitative data and
effective discarding of qualitative information can be mitigated
by deep understanding of issues, careful design, implementation,
and oversight so I'm not arguing against data collection under
the right set of circumstances.  But, unless whatever data
collection and measurement plans are developed considers the
issues, risks, and controls --things that generally goes far
beyond (and are orthogonal to) openness and transparency -- they
are often more of a problem than an advantage.

If you want to write an Internet-Draft that proposes what should
be measured in the IANA case _and_ that provides a context for
proper consideration of the qualitative factors and protection
against rewarding them for rushing through things rather than
doing them carefully and with due consideration, I, for one,
would be very interested in seeing it.  Probably so would the
IANA staff as well as the IAB.  But the "principle of
measurability" as proposed above is, independent of anything
else that has been said in this thread, just not a good idea.

best,
    john

---

From: Jay Daley <[email-masked]>
Date: Fri, 28 Jun 2019 12:08:03 +1200

Hi John

Thank you for such a detailed and thoughtful reply.

Rather than address each point, let me just note one thing, which is that the absence of this principle doesn’t stop the problems you outline below. Metrics are still set, expectations still formed and measurements still made.  Without a principle that is generally very narrowly defined as a set of contractual requirements and so becomes a pull of data for contractual purposes only rather than a principled push of data for use as the community wishes.

However, you’re quite right that this proposed principle doesn’t have anything close to consensus and partly I think that’s because the language I used fell into the same trap of constraining the measurement for the primarily 'contractual' purpose of assessing performance, ("performance" being in itself a contentious word).

I still think there's a need for a principle that requires data collection and open publication, and if we could leave it at "in an open, transparent, accountable and *measureable* way" then that would do it for me, but it’s hard to see how that can be detailed further while avoiding the issues.

kind regards
Jay

> On 28/06/2019, at 11:35 AM, John C Klensin <[email-masked]> wrote:
> 
> 
> 

-- 
Jay Daley
[email-masked]
+64 21 678840
skype: jaydaley
twitter: @jay_daley