Subject: [DNSOP] Informal meeting about root KSK futures at IETF 103

From: Paul Hoffman <[email-masked]>
Date: Fri, 26 Oct 2018 19:49:52 +0000

[[ I'm sending this to DNSOP because a lot of the people on this list will be at IETF 103 in Bangkok ]]

There will be an informal discussion of DNS root KSK futures at IETF 103 on Friday, November 9, at 0900 local (Bangkok) time in room Chitlada 3. The purpose is to allow IETF participants to ask questions and propose ideas for how the DNS root KSK should be maintained in the future. Suggestions from the mic lines at recent meeting include suggestions for fixed rollover schedules, suggestions for data collection before the next rollover, standby KSKs and how they would affect the rollover schedule, and how resolver software vendors' actions can be taken into account. The discussion is expected to last for one hour or less.

Unfortunately, there is no remote participation for the Friday sessions, so the meeting will be limited to those in the room. However, I will make a recording to create notes on what is said.

--Paul Hoffman

---

From: Joe Abley <[email-masked]>
Date: Fri, 26 Oct 2018 14:27:36 -0700

> On Oct 26, 2018, at 15:49, Paul Hoffman <[email-masked]> wrote:
>
> There will be an informal discussion of DNS root KSK futures at IETF 103 on Friday, November 9, at 0900 local (Bangkok) time in room Chitlada 3.

I'm sure I'm not the only person planning to fly out from Bangkok on
Friday morning, given that there are no working group meetings
scheduled on that day.

Might it be possible to find a slot earlier in the week instead of or
as well as your Friday meeting?

Good idea to host a session though, I am fully supportive :-)

Joe

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 29 Oct 2018 12:26:51 -0400

> Friday morning, given that there are no working group meetings

Count me as another, for that very reason.  When I first saw Paul's
message I thought, "oh that's a shame" but figured it to be fairly
set.  If there's flexibility for making the meeting happen earlier in
the week, I'd be interested.

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 29 Oct 2018 12:36:28 -0400

> the week, I'd be interested.

Following up to my own message, since this was further down in my box
on another list...

------- start of forwarded message (RFC 934 encapsulation) -------
From: Paul Hoffman <[email-masked]>
Subject: Re: [ksk-rollover] Informal meeting at IETF 103
To: "[email-masked]" <[email-masked]>

Based on some requests from folks who are leaving the IETF meeting
early, I have also reserved a meeting room for 1600-1700 Wednesday
afternoon (local time), Pagoda Room on the 4th floor.

And just to emphasize: the purpose of this week's informal gatherings is
to let folks in the IETF community chat about their ideas in front of
other IETFers. This is similar to the KSK-related mic lines at the
DNS-OARC and RIPE meetings a few weeks ago. These IETF side-meetings
really are just slightly-better-organized hallway discussions. Given the
wide range of proposals we have already heard, it is good to get a bit
of face-to-face sharing going early.

We won't start formal planning about the KSK futures until after the
rollover process is complete*. When we do, we'll do it in discussion
environments that are much more inclusive than these informal IETF
side-meetings or the mic lines at other technical meetings.

[...]
------- end -------

---

From: Steve Crocker <[email-masked]>
Date: Mon, 29 Oct 2018 18:46:19 -0400

I won't be in Bangkok, so I won't be able to participate.  In my view,
there were two specific problems that dominated the rollover problem.  The
first was the inability to determine the configuration of querying
resolver.  The second was the in ability to notify resolver operators if it
was evident their software was misconfigured.

Although these two problems became evident and important during the
rollover, they also apply more generally, so if the IETF is going to work
on improvements, it would be helpful if the improvements would be useful in
the event of other configuration or operational issues.  Ideally, every
resolver that issues a query would provide configuration information,
perhaps in a controlled fashion, and there should also be a way to notify
the resolver operator of operational problems.

We will need to do more rollovers in the future, driven at least in part by
the need to change algorithms.  I would hope we can work toward making
these relatively straightforward.

Thanks,

Steve

On Mon, Oct 29, 2018 at 12:36 PM Dave Lawrence <[email-masked]> wrote:

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: Joe Abley <[email-masked]>
Date: Mon, 29 Oct 2018 20:23:46 -0400

Hi Steve,

There will always be the potential for tension between the desire to perform measurement and the need for privacy. In this case it seems to me that a well-intentioned and competent authority, supported by a well-intentioned and occasionally-coherent community has a plausible and sensible desire to understand the configuration of resolvers. I imagine others might see things differently, however. I suspect that having a clean signal that can be relied upon is going to at least change the extent to which client infrastructure is private.

Your last paragraph caught my eye, though. It could be argued that the rollover that just completed was straightforward, at least from a technical perspective. The delays, uncertainty, concern and unfulfilled doomsaying were the things that made it difficult. (It's easy to load that last sentence so that the concern looks ridiculous given the outcome; I am fully aware that the hyperbole would be the thing that looked ridiculous if things had turned out differently.)

To what extent does a regular and frequent cadence of key rollovers obviate the need for concern? It seems to me that at some point down the road if a key roll breaks someone's network, it's going to be much easier to point the finger at the network operator than at whomever is responsible for rolling the key. Perhaps the way to eliminate the things that made the first rollover hard is just to keep doing it until it's officially easy.

Joe

> On 29 Oct 2018, at 18:46, Steve Crocker <[email-masked]> wrote:
> 
> I won't be in Bangkok, so I won't be able to participate.  In my view, there were two specific problems that dominated the rollover problem.  The first was the inability to determine the configuration of querying resolver.  The second was the in ability to notify resolver operators if it was evident their software was misconfigured.
> 
> Although these two problems became evident and important during the rollover, they also apply more generally, so if the IETF is going to work on improvements, it would be helpful if the improvements would be useful in the event of other configuration or operational issues.  Ideally, every resolver that issues a query would provide configuration information, perhaps in a controlled fashion, and there should also be a way to notify the resolver operator of operational problems.
> 
> We will need to do more rollovers in the future, driven at least in part by the need to change algorithms.  I would hope we can work toward making these relatively straightforward.
> 
> Thanks,
> 
> Steve
> 
> 
> On Mon, Oct 29, 2018 at 12:36 PM Dave Lawrence <[email-masked] <mailto:[email-masked]>> wrote:
> Dave Lawrence writes:
> From: Paul Hoffman <[email-masked] <mailto:[email-masked]>>
> To: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/dnsop <https://www.ietf.org/mailman/listinfo/dnsop>

---

From: Steve Crocker <[email-masked]>
Date: Mon, 29 Oct 2018 20:29:15 -0400

I had advocated early and frequent rollovers for precisely the reason: keep
doing it until it’s easy, so we’re in strong agreement.

Yes, this one actually went smoothly but there was some tension.  Aside
from any specific improvement, reducing the tension and sense of drama is
mainly what I had in mind.

Thanks,

Steve

On Mon, Oct 29, 2018 at 8:23 PM Joe Abley <[email-masked]> wrote:

> Hi Steve,
>
> There will always be the potential for tension between the desire to
> perform measurement and the need for privacy. In this case it seems to me
> that a well-intentioned and competent authority, supported by a
> well-intentioned and occasionally-coherent community has a plausible and
> sensible desire to understand the configuration of resolvers. I imagine
> others might see things differently, however. I suspect that having a clean
> signal that can be relied upon is going to at least change the extent to
> which client infrastructure is private.
>
> Your last paragraph caught my eye, though. It could be argued that the
> rollover that just completed was straightforward, at least from a technical
> perspective. The delays, uncertainty, concern and unfulfilled doomsaying
> were the things that made it difficult. (It's easy to load that last
> sentence so that the concern looks ridiculous given the outcome; I am fully
> aware that the hyperbole would be the thing that looked ridiculous if
> things had turned out differently.)
>
> To what extent does a regular and frequent cadence of key rollovers
> obviate the need for concern? It seems to me that at some point down the
> road if a key roll breaks someone's network, it's going to be much easier
> to point the finger at the network operator than at whomever is responsible
> for rolling the key. Perhaps the way to eliminate the things that made the
> first rollover hard is just to keep doing it until it's officially easy.
>
>
> Joe
>
> We will need to do more rollovers in the future, driven at least in part
> by the need to change algorithms.  I would hope we can work toward making

---

From: George Michaelson <[email-masked]>
Date: Tue, 30 Oct 2018 10:41:43 +1000

There is a tension between assumed privacy ("this is my resolver, what
I run is my business, how I run it is my business") of entities
running resolvers, and customers ("this is my DNS query. what I ask is
my business") and providers of infrastructure ("this is my liability:
the consequences of not understanding what we are doing expose me to
high consequence risk")

Some significant players in DNS conversations have been remarkably
oppositional to instrumenting the protocol.

I think on balance the question "can your resolver survive the KSK
roll" is not a massive invasion of privacy, is not a huge impost on
providers of DNS service, does not interfere with implied rights to
privacy. We should have better knowledge. We should instrument the
resolver. The protocol. We should not be forced to do ludicrous things
to find this out, it should be as innate as the telnet capability
negotiation, or the HTTP client/server capability negotiation. We know
how to do these things.

I am statist. I do not pretend states don't exist. I am not a
libertarian, I am not trumpeting DNS as a tool for individual liberty.
DNS is a useful functional element which now has consequence when it
doesn't operate correctly.

We've walked a long way into an industry self-regulation outcome which
I feel is as consequential as 250v or 50hz, both of which are defined
parameters with variances and tolerances which people in the
electricty supply industry have to respect, to remain protected by
public liability insurance. We should consider the public DNS in the
same light: If we want to avoid opprobrium over mis-steps in operating
the root of the DNS, we should adopt a rational posture akin to the
ones in Power, Water, Sewerage, because thats what we are: a public
utility function. With public utility liability questions.

Rightly or wrongly DNS is a critical service, which has potential for
wide impact which is not good. It didn't (this time) at scale, in as
much as nobody has come forward to say it did: The assumption there
has been *no* damage, or *no* loss of service feels to me wrong. I am
(as a gambling man) confident it was low, in as much as 2.3b users
didn't rise up and grab pitchforks. I bet it was big enough (remember,
0.001 of 2.3b is 23,000) that you'd fill an auditorium.

Is 1 in 1000 an acceptable damage rate? is 99.999% unaffected ok? Why
is 98% not ok? or 97%? Where IS the acceptable damage boundary? Who
gets to say what will be tolerated as loss in the DNS system, for what
period of time?

If we want to "plan" for rolls, I think we should stop allowing "not
on my server" to impede DNS protocol design which would provide
un-ambiguous signal of capability in the field. And "not in my
protocol" too. If you offer a service like 8.8.8.8 or 9.9.9.9 or
1.1.1.1 you should be in a position of community obligation to be
completely clear what your posture is towards KSK roll, algorithm
roll. If you can't support it, you should say so. If that means people
are advised to TURN YOU OFF that needs to be understood.

If we want to perform algorithm roll, get stand-by keys, we should be
prepared to argue for the changes which make it easier to do these
things.

-George
On Tue, Oct 30, 2018 at 10:29 AM Steve Crocker <[email-masked]> wrote:
>
> I had advocated early and frequent rollovers for precisely the reason: keep doing it until it’s easy, so we’re in strong agreement.
>
> Yes, this one actually went smoothly but there was some tension.  Aside from any specific improvement, reducing the tension and sense of drama is mainly what I had in mind.
>
> Thanks,
>
> Steve
>

---

From: George Michaelson <[email-masked]>
Date: Tue, 30 Oct 2018 10:45:21 +1000

as usual, billions arithmetic got me. 0.001 * 2,300,000,000 is 2,300,000

so thats a few dodgers stadiums more than I said...

-G

---

From: Paul Vixie <[email-masked]>
Date: Mon, 29 Oct 2018 18:55:15 -0700

i hope we can make rollover happen every 18 to 30 months. often enough 
that its contribution of chaos is lost in the noise margin of normal 
chaos. often enough that we never forget how to do it.

---

From: Tony Finch <[email-masked]>
Date: Tue, 30 Oct 2018 11:05:52 +0000

> doing it until it’s easy, so we’re in strong agreement.

Yes, I would like to see annual rollovers. Keep that hinge greased :-)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Shannon, Rockall: Cyclonic becoming west 5 to 7. Rough or very rough. Rain or
showers. Good, occasionally poor.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 31 Oct 2018 09:31:52 +1100

Ultra frequent key rolls are not necessary.  It takes years the latest releases of name servers to make it into shipping OS’s.  The last KSK worked so well in part because there was a large amount of time between publishing the new KSK and using the new KSK.  This allowed name server vendors to publish releases with the new KSK and for those release to make it into some OS releases.

> On 30 Oct 2018, at 10:05 pm, Tony Finch <[email-masked]> wrote:
> 
> showers. Good, occasionally poor._______________________________________________
> DNSOP mailing list
> [email-masked]

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Dr Eberhard W Lisse <[email-masked]>
Date: Tue, 30 Oct 2018 23:38:40 +0100

Mark,

but would regular rolls not put vendors into a 'habit' of getting
updates onto their package managers?

el

On 2018-10-30 23:31 , Mark Andrews wrote:
> Ultra frequent key rolls are not necessary.  It takes years the latest
> releases of name servers to make it into shipping OS’s.  The last KSK
> worked so well in part because there was a large amount of time
> between publishing the new KSK and using the new KSK. This allowed
> name server vendors to publish releases with the new KSK and for those
> release to make it into some OS releases.
> 
>>> I had advocated early and frequent rollovers for precisely the
>>> reason: keep doing it until it’s easy, so we’re in strong agreement.
>>

-- 
Dr. Eberhard W. Lisse          / Obstetrician & Gynaecologist (Saar)
[email-masked]            / *     |   Telephone: +264 81 124 6733 (cell)
PO Box 8421                  /
Bachbrecht, Namibia     ;____/

---

From: Mark Andrews <[email-masked]>
Date: Wed, 31 Oct 2018 10:40:39 +1100

Name server vendors have NO CONTROL over when down streams pick up changes.
We would like OS vendors to pick up maintenance release sooner than they do.
It would reduce the amount of time we spend diagnosing already fixed issues.
We spend the time back porting fixes so people can have stable interfaces
and fixed code.  The more maintenance releases installed the better the bang
for buck that work achieves.

> On 31 Oct 2018, at 9:38 am, Dr Eberhard W Lisse <[email-masked]> wrote:
> 
> Mark,
>

---

From: Jim Reid <[email-masked]>
Date: Wed, 31 Oct 2018 00:16:56 +0000

On 30 Oct 2018, at 22:31, Mark Andrews <[email-masked]> wrote:
> 
> Ultra frequent key rolls are not necessary.  It takes years the latest releases of name servers to make it into shipping OS’s.

So what? Key rollover policies cannot and should not be driven by vendor OS release schedules. Or the BIND/whatever version they choose to distribute. If key rollovers became dependent on these considerations, we’d never be able to roll the root’s KSK.

Software that had hard-wired the old key caused trouble for the recent rollover so we simply have to be in a much better place next time. I hope you don't want to perpetuate that legacy behaviour, albeit in a slightly different form.

If the (hypothetical) problem is DNS software gets shipped with the current KSK on the release date and that might lurk in vendor distributions long after the KSK has rolled, the solution is obvious. Don’t do that.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 31 Oct 2018 11:27:40 +1100

> On 31 Oct 2018, at 11:16 am, Jim Reid <[email-masked]> wrote:
> 

Bootstrap is still a issue.  Over fast TA rolling makes it more of a issue.

I think we would replace RFC 5011 with something else before we roll the root key next while using RFC 5011 timings for the roll for legacy implementations.  We really need a way to signal how long a TA is expected to be valid for.  This allows machines to safely fail to insecure if they have been off for too long.

Mark

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 31 Oct 2018 00:43:59 +0000

Just a brief note that the threads about KSK futures started on the [email-masked] <mailto:[email-masked]> mailing list and should probably still be there. The only bit that was meant to be on this Working Group mailing list was an announcement of the side-meetings next week.

If at some point there is DNS protocol or operational work to be done, it can be brought here. Until then, let's leave this mailing list to the quite active list of topics that the Chairs are already juggling.

--Paul Hoffman

---

From: George Michaelson <[email-masked]>
Date: Wed, 31 Oct 2018 11:52:09 +1000

I feel backup key, and alg, are sufficiently of wide benefit, that the
qualms about frequency are second-order to the primary goal of an
improved outcome

1) backups go to stability of unplanned events
2) new alg would permit a return to shorter packet sizes even across
keyroll, which makes IPv6 DNS on UDP more reliable

I understand your concern, but I think its cart-before-horse stuff. We
*want* shorter crypto sigs. and we *want* more reliable behaviour in
unexpected circumstances. We can't get there, without another keyroll.
probably two more.

-G

---

From: Jim Reid <[email-masked]>
Date: Wed, 31 Oct 2018 08:19:00 +0000

> On 31 Oct 2018, at 00:27, Mark Andrews <[email-masked]> wrote:
> 

Indeed. And that's the underlying problem that needs to be fixed IMO - for instance when/if there's an emergency rollover.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 31 Oct 2018 11:54:46 -0700

Jim Reid wrote:
>
>> On 31 Oct 2018, at 00:27, Mark Andrews<[email-masked]>  wrote:
>>
> Indeed. And that's the underlying problem that needs to be fixed IMO
> - for instance when/if there's an emergency rollover.

bootstrappers should have https access to a complete history of root 
ksk, each one signed by its predecessor. this doesn't handle revocation, 
but nothing in dnssec handles revocation, and that's by design, and so 
i'm inclined not to worry about it.

but that's the backup plan. the primary expectation is, devices which 
come off the shelf after a dnssec ksk roll will have some means of 
reaching and trusting their manufacturer's software update service, 
which will offer them a current ksk for validation.

manufacturers who don't last long enough to do this, or who for whatever 
other reason don't do this, will be shipping future bricks. and i'm fine 
with that, since it's in their power to do the right thing, which is the 
best we can offer.

-- 
P Vixie

---

From: Joe Abley <[email-masked]>
Date: Wed, 31 Oct 2018 15:08:30 -0400

Hi Paul,

On 31 Oct 2018, at 14:54, Paul Vixie <[email-masked]> wrote:

> bootstrappers should have https access to a complete history of root ksk, each one signed by its predecessor. this doesn't handle revocation, but nothing in dnssec handles revocation, and that's by design, and so i'm inclined not to worry about it.

The existing scheme provides bootstrappers with a complete history of trust anchors (published by digest). The published object was intended to be accompanied by a detached signature that could be trusted by a vendor, since there was to be a certificate chain from the object-signing certificate back to an X.509 trust anchor that made sense to the client (e.g. an iOS code-signing key, a Windows code-signing key, an ISC code-signing key, etc, etc). This is described in RFC 7958.

The only "vendor" that ever published such a certificate was the (largely proof-of-concept) "ICANN" vendor.

I'm not suggesting that the scheme described in that document is good, or that it shouldn't be replaced with something better, but it's one possible starting point for the discussion.

Joe

---

From: Michael StJohns <[email-masked]>
Date: Wed, 31 Oct 2018 18:24:52 -0400

On 10/31/2018 2:54 PM, Paul Vixie wrote:
> ksk, each one signed by its predecessor. this doesn't handle 
> revocation, but nothing in dnssec handles revocation, and that's by 
> design, and so i'm inclined not to worry about it.
there are at least two things wrong with the above:

1) 5011 provides revocation for ALL keys, not just the root keys, and 
that's by design.  Set the revoke bit on a key, self-sign the RRset with 
that key, and it should prevent trust being traced through the key.
2) the simple history approach will not work.  Simply consider that you 
might need to deal with compromised keys and tell me how I can ensure I 
have ALL the information I need based on a history that can actually 
reflect all types of real world input.   Something like a block chain 
(with yet another set of signatures) over a set of versions of the root 
dnskey RRSet might work, but that will require another chain of trust 
besides the dnssec chain.

> which will offer them a current ksk for validation.

I think the primary expectation should really be the only plan.
>
> manufacturers who don't last long enough to do this, or who for 
> whatever other reason don't do this, will be shipping future bricks. 
> and i'm fine with that, since it's in their power to do the right 
> thing, which is the best we can offer.
>
Yup -

Mike