Subject: Re: [DNSOP] Adoption of new EDNS opcode "rrserial"

From: Mauricio Vergara Ereche <[email-masked]>
Date: Thu, 6 May 2021 22:02:29 -0700

Hi Hugo,

I just want to bring back to life this topic as it solves an issue that
several operators (like me) seem to be in need to solve while debugging
issues.

Mauricio

On Mon, Jan 27, 2020 at 7:09 AM Hugo Salgado <[email-masked]> wrote:

> Dear DNSOPers, as an operator I tend to have this need to couple
> an answer for a query to an auth server, with the actual "SOA zone
> version" used. So I think it'll be valuable to have an EDNS option
> for it.
>
> Here I'm proposing it with this new draft. The 'camel index' for
> its implementation/hack/proof-of-concept is about 37 lines in
> NSD 4.1 (more details in Appendix A).
>
> I've asked some operators and they see value on it. Is there any
> support for adoption in DNSOP?
>
> -----
> Name:           draft-salgado-rrserial
> Revision:       01
> Title:          The "RRSERIAL" EDNS option for the SOA serial of a RR's
> zone
> Document date:  2020-01-27
> Group:          Individual Submission
> Pages:          5
> URL:
> https://www.ietf.org/internet-drafts/draft-salgado-rrserial-01.txt
> Status:         https://datatracker.ietf.org/doc/draft-salgado-rrserial/
> Htmlized:       https://tools.ietf.org/html/draft-salgado-rrserial-01
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-salgado-rrserial
> Diff:
> https://www.ietf.org/rfcdiff?url2=draft-salgado-rrserial-01
>
> Abstract:
>    The "RRSERIAL" EDNS option allows a DNS querier to ask a DNS
>    authoritative server to add a EDNS option in the answer of such query
>    with the SOA serial number field of the origin zone which contains
>    the answered resource record.
>
>    This "RRSERIAL" data allows to debug problems and diagnosis by
>    helping to recognize the origin of an answer, associating this answer
>    with a respective zone version.
> -----
>
> Best,
>
> Hugo Salgado
>
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

-- 
Mauricio Vergara Ereche
keybase.io/mave

---

From: Hugo Salgado <[email-masked]>
Date: Fri, 7 May 2021 12:47:49 -0400

I'll upload a new version to revive it, and ask for a slot
in IETF111 for further discussion!

Thanks,

Hugo

---

From: Joe Abley <[email-masked]>
Date: Fri, 7 May 2021 13:03:04 -0400

Hi Hugo,

> in IETF111 for further discussion!

Just to add my voice to the chorus, I missed this the first time around so thanks, Mauricio, for mentioning it.

I haven't read the draft in detail but I do like the idea and could think of many times this would have been useful for data collection, debugging, etc.

Joe

---

From: "John Levine" <[email-masked]>
Date: 7 May 2021 13:39:56 -0400

It appears that Hugo Salgado  <[email-masked]> said:
>-=-=-=-=-=-

It looks like it's worth considering, but I also wonder how
relevant it is for DNS servers that don't use AXFR/IXFR and SOA
serial numbers to keep versions in sync.

I put serial numnbers in my zones but in fact my servers use
rsync from a hidden master so the serials don't tell you much.

---

From: Frederico A C Neves <[email-masked]>
Date: Fri, 7 May 2021 14:55:00 -0300

> serial numbers to keep versions in sync.

In that case not much, but for many that uses in-band zone transfer
mechanisms it is very relevant. Specially in the case that the serial
in a zone is time deterministic.

Fred

---

From: Joe Abley <[email-masked]>
Date: Fri, 7 May 2021 14:09:27 -0400

> serial numbers to keep versions in sync.

SOA serial numbers are definitely useful for the apex SOA/IN queries that are prerequisites to zone transfers. However, they're also really useful information for any dynamic zone that you're trying to troubleshoot. Successive queries are not necessarily going to return matching information.

By analogy, you can send a query to a nameserver and get a troublesome response, and debug with a HOSTNAME.BIND/CH/TXT query to see what server it was. But for a nameserver provisioned with anycast or as part of a cluster, those data points are not necessarily going to match, which is why NSID is good.

Joe

---

From: Brian Dickson <[email-masked]>
Date: Fri, 7 May 2021 11:39:51 -0700

On Fri, May 7, 2021 at 10:03 AM Joe Abley <[email-masked]> wrote:

> I haven't read the draft in detail but I do like the idea and could think
> of many times this would have been useful for data collection, debugging,
> etc.
>
>
I concur with everything Joe wrote. So, very much in favor of this, thanks
for doing this.

Brian

---

From: Donald Eastlake <[email-masked]>
Date: Fri, 7 May 2021 17:15:22 -0400

Seems like a good idea to me. I think the WG should adopt it.

Thanks,
Donald
===============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 2386 Panoramic Circle, Apopka, FL 32703 USA
 [email-masked]

On Mon, Jan 27, 2020 at 10:09 AM Hugo Salgado <[email-masked]> wrote:
> Title:          The "RRSERIAL" EDNS option for the SOA serial of a RR's zone
> URL:            https://www.ietf.org/internet-drafts/draft-salgado-rrserial-01.txt
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-salgado-rrserial
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-salgado-rrserial-01
>
> Abstract:

---

From: Matthew Pounsett <[email-masked]>
Date: Fri, 7 May 2021 18:21:31 -0400

> version" used. So I think it'll be valuable to have an EDNS option

I also missed this the first time around.  Mauricio brought it up
during the OARC workshop last night; I think it's a very good idea,
and the WG should adopt it.  There are many occasions in the past
where this would have been helpful in debugging, and I'm sure there
will be many more in the future.

I haven't read this in detail yet, but I will happily provide reviews
if the WG adopts it.

---

From: Petr Špaček <[email-masked]>
Date: Mon, 10 May 2021 10:19:17 +0200

> Name:           draft-salgado-rrserial

...

>     with a respective zone version.

I think it needs more thought. Here's why:

Extending TTLs
==============
First, people in this thread have floated ideas about using SOA serial 
for extending TTLs.

I think it's a bad idea because:
a) It would be pretty complex to implement right.
b) There are no guarantees that the SOA serial did not overflow in the 
meanwhile. As an extreme example, auth server can legally increment SOA 
value by 2^29 for each update, thus rolling SOA serial over very frequently.

Debugging
=========
Secondly, let's consider just the "debugging" use-case, which removes 
the requirement to make the proposed mechanism 100 % reliable:

In practice, this RRSERIAL option is a specialized way to conflate two 
queries into one:
- First query specified by the standard (qclass, qtype, qname)
- Second query with hardcoded qtype=SOA, qclass derived from the primary 
qclass in question section, and SOA name derived from qname in the 
question section.

To me, this sounds like a crude hack, and I think the proper solution 
should be a real multi-query option, which incidentally provides a 
superset of RRSERIAL capabilities.

A multi-query option would also work multi-master architectures that 
cannot rely on SOA serial but on some other (presumably queriable) 
information.

Epilogue
========
Generally, I think SOA serial design was fine in 1985 but is ill-fitted 
for 2021. (I have to admit I'm biased because I'm implemented 
multi-master auth in topologies with tens of DNS servers, all accepting 
dynamic updates at the same time.)

For this reason, I think the DNS protocol should be gradually getting 
rid of the dependency on SOA serial, not increasing it.

-- 
Petr Špaček

---

From: Vladimír Čunát <[email-masked]>
Date: Mon, 10 May 2021 10:32:08 +0200

On 10. 05. 21 10:19, Petr Špaček wrote:
> I think the proper solution should be a real multi-query option, which 
> incidentally provides a superset of RRSERIAL capabilities.

If multi-queries require the records being in sync (if from the same 
zone), I really dislike the implications of them being sent to 
resolvers, especially for many questions at once.  Caching, in particular.

--Vladimir | knot-resolver.cz

---

From: Hugo Salgado <[email-masked]>
Date: Mon, 10 May 2021 12:37:16 -0400

Hello everyone. Thanks for the comments, I just uploaded an unchanged
version (just to revive it) at:
  https://datatracker.ietf.org/doc/draft-salgado-dnsop-RRSERIAL/

I agree RRSERIAL doesn't have much relevance in zones that don't give
serial versioning a meaning to its content. We can add a paragraph
about it, to make the applicability more clear. I also don't think we
should start to "deprecate" the SOA serial meaning at this point.
One can say that "modern" implementations using complex backends makes
SOA serials irrelevant, but there's certainly a use case for classic
and mainly static behavior. Just as NSID adds an "infrastructure"
identification dimension to a response, RRSERIAL adds the data
versioning dimension.

And responding to the comment that having multi-queries is better, is
something that is long overdue, and would certainly make this hack
obsolete. But I think the multi-query issues are so complex that it's
worth moving forward incrementally in the meantime. For the same
reason, I think we could aim for it to have "experimental" status at
the start, and after a few years we measure it to see its use and we
can adapt or change it to standard or deprecate it in favor of another
qtype.

Other issues that I think need more analysis is deciding whether it
makes any sense to expose RRSERIAL in resolvers. The original idea was
only in authoritatives, but it might make some sense to debug in
resolvers as well, to somehow identify the "data source" of a cached
record. In this sense, I fear an increase in cache requirements of
resolvers, which should somehow maintain the extra data; and also
in traffic and option availability for upstream auths.

Hugo

---

From: Joe Abley <[email-masked]>
Date: Mon, 10 May 2021 12:55:22 -0400

Hi Vladimir,

On 10 May 2021, at 04:32, Vladimír Čunát <[email-masked]> wrote:

>> I think the proper solution should be a real multi-query option, which incidentally provides a superset of RRSERIAL capabilities.
> 
> If multi-queries require the records being in sync (if from the same zone), I really dislike the implications of them being sent to resolvers, especially for many questions at once.  Caching, in particular.

Yes, I agree with that. Perhaps we could specify that a request for an RRSERIAL MUST only be satisfied for a response with AA=1, so that only servers that are able to provide an authoritative response to the original query will return it.

In contrast, I think there are probably use cases for a server that can't provide an authoritative response (e.g. a recursive server) to process a more generic multi-question query. So this seems like a reason to consider the two problem statements separately. I don't know exactly what use cases Petr had in mind for that though, so I'm guessing slightly.

Joe

---

From: Mauricio Vergara Ereche <[email-masked]>
Date: Mon, 10 May 2021 09:59:50 -0700

I agree with what Hugo said

I would also like to point out that this draft spirit is aiming to be a
debugging tool to be used by humans and not in between servers. If we start
introducing all these new use cases in-between servers (like authoritatives
signaling secondaries or resolvers storing new data) it would add an
inmense amount of complexity that might be out of scope at the moment.

Would it be simpler to progress with this as a read-only way to query an
authoritative server in an atomic way for what's the current status of a
record? I guess if we agree on that, we could expand on a better
description or explanation to reflect such intention.

Mauricio

- --
Mauricio Vergara Ereche
keybase.io/mave

On 2021-05-10 at 16:37, [email-masked] wrote:
> > > > Status:
https://datatracker.ietf.org/doc/draft-salgado-rrserial/
> > > > Htmlized:
https://tools.ietf.org/html/draft-salgado-rrserial-01

---

From: Peter van Dijk <[email-masked]>
Date: Mon, 10 May 2021 21:36:50 +0200

>   https://datatracker.ietf.org/doc/draft-salgado-dnsop-RRSERIAL/

Thanks Hugo, that is useful.

In section 3.2, 'the resource record of the ANSWER section' is
ambiguous. The answer section can contain many resource record
(s/sets). I suggest a reference to 'QNAME (original)' from RFC8499, or
a careful copy of the words in that definition.

Also in section 3.2, I do not think responding with the option should
be limited to NOERROR. Specifically, I'd very much also want it to work
for NXDOMAIN, and I can imagine some cases of it being useful even in
SERVFAIL cases (at least in database-driven name servers like PowerDNS,
where individual records inside a zone can be broken).

> about it, to make the applicability more clear.

I agree, and I do not think this is a reason to not have this feature.

> I also don't think we

+1

> obsolete. 

Multi-query has not happened. I doubt it will happen. And in fact,
mapping this specific use case onto it would limit implementations. I
can imagine multi-query being implemented in some proxy/frontend, that
sends out parallel queries to 'simpler' auths that do not even know
about multi-query, and then the atomicity is gone.

> in traffic and option availability for upstream auths.

To expose it in resolvers, resolvers would have to set the option on
all their outgoing queries, so that they can remember the serial
involved in each RRset that they got. I don't think this puts a big
burden on storage requirements, but adding EDNS options to all your
resolver-to-auth traffic is always a gamble, finding out which auths
will suddenly return SERVFAIL, or REFUSED, or just drop your query -
or, in some observed cases, tell you NXDOMAIN because they're confused.
Now, those auths are wrong, and should not exist, but I trust there
will be some reluctance in deployments.

That said, supporting this feature in resolvers does not require any
changes to the protocol itself; the EDNS option, as your draft
currently defines it, looks fine to me. Of course, if the document does
want to support the resolver case, it should explain what that means.

(Unrelated to anything above, I can see reasons to put the whole SOA in
there instead of just the serial; this also reinvokes the 'why not put
it in AUTHORITY or ADDITIONAL' question, but I really like the short
EDNS option that does not change the processing of any RRsets from a
query response.)

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

---

From: Tony Finch <[email-masked]>
Date: Mon, 10 May 2021 22:09:49 +0100

Peter van Dijk <[email-masked]> wrote:
> for NXDOMAIN,

Isn't the SOA record usually present in a negative response?

> and I can imagine some cases of it being useful even in SERVFAIL cases
> (at least in database-driven name servers like PowerDNS, where
> individual records inside a zone can be broken).

Perhaps also in cases where the server has a copy of zone serial number
NNN but it has expired.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  https://dotat.at/
Viking, North Utsire: Cyclonic 6 to gale 8, becoming southerly 3 to 5.
Moderate or rough, becoming moderate in North Utsire. Rain, fog
patches. Moderate or good, occasionally very poor.

---

From: Peter van Dijk <[email-masked]>
Date: Mon, 10 May 2021 23:33:47 +0200

> Isn't the SOA record usually present in a negative response?

Good point! In that case, I retract most of that and suggest the draft
points out that in those cases, a serial can be extracted anyway. That
said, I'm not sure that's a reason to skip including the option in the
response.

+1
 
Kind regards,