Subject: Re: [DNSOP] [Ext] Lameness terminology (was: Status of draft-ietf-dnsop-terminology-bis)

From: Edward Lewis <[email-masked]>
Date: Wed, 2 May 2018 19:47:57 +0000

On 4/23/18, 10:23, "DNSOP on behalf of Shane Kerr" <[email-masked] on behalf of [email-masked]> wrote:

>    I don't know if this is documented anywhere so that it can be
>    referenced properly, sorry. I am happy to discuss further but I think
>    this basically covers all I know. I don't mind proposing text, but
>    probably someone (Ed maybe?) would be a better person.

I've been looking for something I recall writing a long time ago, but haven't found it.  I'm not the authority on lameness (insert self-deprecating joke here), the term has been repeatedly defined in a number of documents.  What I wrote was a survey of them, including the citations, but I haven't been able to find it.  Not on the web, that is.

The reason I bother with this was, when I was tasked with the work in 2002 or 2003, I was genuinely surprised at the definition of "lame" delegation.  I was surprised that it did not include non-responsive servers - the term referred to responding servers only.

If I can't find the text soon, I'll try to recreate the list of references at least.

(Only if you like reading history:)

The reason was a flaw in "certain old resolvers" that followed the "upwards referral" to the root that the "predominate server code of the time" had decided to use for lameness.  The result was a lot of resolver stuck in an infinite loop, hitting the root servers.  I.e., this was an operational issue.  The solution was updating and redeploying the buggy code, not stamping out lame servers (which was the goal of the task).  FWIW, the "upwards referrals" were discontinued when it became apparent they were being used in noticeable amplification attacks.

---

From: Amreesh Phokeer <[email-masked]>
Date: Thu, 3 May 2018 10:12:42 +0400

> If I can't find the text soon, I'll try to recreate the list of references

We are in process of implementing a "Lame delegations" policy at AFRINIC
<http://tiny.cc/afrinic-lame>

We consider "lame" any NS which is either:
- Not responding at all.
- Responding in some way, but not for the specific domain queried.
- Responding for the correct domain, but without the authority bit set.

We used the definition in RFC1713:

A lame delegation is a serious error in DNS configurations, yet a
   (too) common one.  It happens when a name server is listed in the NS
   records for some domain and in fact it is not a server for that
   domain.  Queries are thus sent to the wrong servers, who don't know
   nothing (at least not as expected) about the queried domain.
   Furthermore, sometimes these hosts (if they exist!) don't even run
   name servers.  As a result, queries are timed out and resent, only to
   fail, thus creating (more) unnecessary traffic.

Cheers,
Amreesh

---

From: David Huberman <[email-masked]>
Date: Thu, 3 May 2018 15:05:18 +0000

> The reason was a flaw in "certain old resolvers" that followed the "upwards referral" to the root that 
> the "predominate server code of the time" had decided to use for lameness.  The result was a lot of 
> resolver stuck in an infinite loop, hitting the root servers.  I.e., this was an operational issue.  The 
> solution was updating and redeploying the buggy code, not stamping out lame servers (which was 
> the goal of the task).  FWIW, the "upwards referrals" were discontinued when it became apparent 
> they were being used in noticeable amplification attacks.
  
I sat on the front lines of ARIN’s war against lame delegations for the entire war.  We spent quite a few
years testing delegations for our definition of lameness, and then notifying the listed tech-c and admin-c.
E-mail recipients would either ignore the email, not understand the email and move on to the next thing,
or would write-in or call-in and speak to either me or my co-worker Jon Worley. Very few lame delegations
were fixed, even among those who called-in or wrote-in for clarification.  rDNS worked for the user, and
they weren’t willing to change anything.

The war was unwinnable. 
    
/david

---

From: Mark Andrews <[email-masked]>
Date: Fri, 4 May 2018 07:24:38 +1000

It’s amazing how fast people can fix lame delegations once email and other services stop flowing. The only reason you think it is un- winnable is that you are unwilling to remove the delegation for failing to maintain a properly working configuration. 

Start removing lame delegation after fair warnings and you will see others fixing their lame delegations on initial notice.  No system works if there are not checks and balances.
-- 
Mark Andrews

> On 4 May 2018, at 01:05, David Huberman <[email-masked]> wrote:
> 
> Ed Lewis wrote:
> 
>> the "predominate server code of the time" had decided to use for lameness..  The result was a lot of 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: David Huberman <[email-masked]>
Date: Thu, 3 May 2018 22:27:30 +0000

Mark Andrews stated:

>It’s amazing how fast people can fix lame delegations once email and other 
>services stop flowing. The only reason you think it is un- winnable is that you 
>are unwilling to remove the delegation for failing to maintain a properly 
>working configuration. 

Ideally, yes – of course.

But in practical terms, when any type of registry strips away a lame delegation
attached to a real, operating network with users behind it, and things break
as a result, it has a high potential of affecting the innocent 3rd parties using that
network.  Especially at scale, the legal liability issues implicated by such an action
are frightening, and quickly outweigh the ‘for the common good’ arguments.

---

From: Bill Woodcock <[email-masked]>
Date: Thu, 3 May 2018 15:36:52 -0700

> On May 3, 2018, at 3:27 PM, David Huberman <[email-masked]> wrote:
> In practical terms, when any type of registry strips away a lame delegation
> as a result…

But isn’t that, by definition, impossible?  What could break as a result of a _lame_ delegation being removed?

They’d have to be spoofing a real IP address behind their firewall, which isn’t supposed to work anyway.

                                -Bill

---

From: Frederico A C Neves <[email-masked]>
Date: Thu, 3 May 2018 19:41:36 -0300

> are frightening, and quickly outweigh the ‘for the common good’ arguments. 

As long as there is community support Mark's observation works as
expected.

Slightly variatios of this policy are in place for LACNIC and APNIC
regions and is very effective.

http://www.lacnic.net/686/2/lacnic/6-lame-delegation-policy
https://www.apnic.net/manage-ip/manage-resources/reverse-dns/lame-dns-reverse-delegation/

Fred

---

From: Mark Andrews <[email-masked]>
Date: Fri, 4 May 2018 09:23:04 +1000

> On 4 May 2018, at 8:36 am, Bill Woodcock <[email-masked]> wrote:
> 
> 
> 

A “Lame Delegation” can be to a particular machines.  These delegation for .fj are lame and have been for over a year.

fj. @128.32.136.3 (adns1.berkeley.edu.): dns=refused edns=refused edns1=ok edns@512=refused ednsopt=refused edns1opt=ok do=refused ednsflags=refused optlist=refused signed=refused ednstcp=refused
fj. @2607:f140:ffff:fffe::3 (adns1.berkeley.edu.): dns=refused edns=refused edns1=ok edns@512=refused ednsopt=refused edns1opt=ok do=refused ednsflags=refused optlist=refused signed=refused ednstcp=refused
fj. @128.32.136.14 (adns2.berkeley.edu.): dns=refused edns=refused edns1=ok edns@512=refused ednsopt=refused edns1opt=ok do=refused ednsflags=refused optlist=refused signed=refused ednstcp=refused
fj. @2607:f140:ffff:fffe::e (adns2.berkeley.edu.): dns=refused edns=refused edns1=ok edns@512=refused ednsopt=refused edns1opt=ok do=refused ednsflags=refused optlist=refused signed=refused ednstcp=refused

But because .fj has multiple servers

fj.			86400	IN	NS	ns1.berkeley.edu.
fj.			86400	IN	NS	teri.usp.ac.fj.
fj.			86400	IN	NS	ns2.berkeley.edu.
fj.			86400	IN	NS	rip.psg.com.
fj.			86400	IN	NS	manu.usp.ac.fj.
fj.			86400	IN	NS	auth00.ns.uu.net.

the service is still up.  In the meantime .fj doesn’t have the level of redundancy they presumably think they have and every resolver in the world spends time discovering that they are broken.

There are others like these where the servers fail to respond to certain classes of legitimate DNS queries.

westlaw.com.au. @146.242.20.65 (ns-emea-1.thomsonreuters.net.): dns=ok edns=noopt edns1=timeout edns@512=noopt ednsopt=timeout edns1opt=timeout do=ok ednsflags=noopt optlist=timeoutsigned=ok ednstcp=ok
westlaw.com.au. @146.242.18.65 (ns-amers-1.thomsonreuters.net.): dns=ok edns=noopt edns1=timeout edns@512=noopt ednsopt=timeout edns1opt=timeout do=ok ednsflags=noopt optlist=timeoutsigned=ok ednstcp=ok
westlaw.com.au. @146.242.19.65 (ns-amers-2.thomsonreuters.net.): dns=ok edns=noopt edns1=timeout edns@512=noopt ednsopt=timeout edns1opt=timeout do=ok ednsflags=noopt optlist=timeoutsigned=ok ednstcp=ok
westlaw.com.au. @146.242.21.65 (ns-emea-2.thomsonreuters.net.): dns=ok edns=noopt edns1=timeout edns@512=noopt ednsopt=timeout edns1opt=timeout do=ok ednsflags=noopt optlist=timeoutsigned=ok ednstcp=ok
westlaw.com.au. @146.242.22.65 (ns-apac-1.thomsonreuters.net.): dns=ok edns=noopt edns1=timeout edns@512=noopt ednsopt=timeout edns1opt=timeout do=ok ednsflags=noopt optlist=timeoutsigned=ok ednstcp=ok
westlaw.com.au. @146.242.24.65 (ns-apac-2.thomsonreuters.net.): dns=ok edns=noopt edns1=timeout edns@512=noopt ednsopt=timeout edns1opt=timeout do=ok ednsflags=noopt optlist=timeoutsigned=ok ednstcp=ok

Why should resolvers have to work around idiocy like this?

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: David Huberman <[email-masked]>
Date: Thu, 3 May 2018 23:28:02 +0000

> But isn’t that, by definition, impossible?  What could break as a result of a _lame_ delegation 

Mark provided you with a forward DNS example. Here’s a _common_ reverse DNS example:

You are the registrant of 192.168.0.0/17.
You setup a single SOA record for 168.192.in-addr.arpa instead of properly defining 128 records 
for each /24 reverse zone.

PTR queries to the NSes will work (for the /17).  

But you’ll fail the lameness checking at an RIR because the RIR checks all zones in the
SOA record, and assumes that if you assert 168.192.in-addr.arpa, that you really meant
to claim authority over the /16.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 4 May 2018 09:33:47 +1000

> On 4 May 2018, at 9:28 am, David Huberman <[email-masked]> wrote:
> 

You see the same with forward zones with domain parking. They set up a .com (or root) zone for all the *.com zones parked on the server and break all negative responses as a consequence.

---

From: David Huberman <[email-masked]>
Date: Thu, 3 May 2018 23:37:07 +0000

>regions and is very effective.
 
Thank you, Fred, and apologies for my incomplete answer that was biased
by my geography.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 4 May 2018 10:03:10 +1000

> On 4 May 2018, at 9:37 am, David Huberman <[email-masked]> wrote:
> 
> Fred wrote:

Which is easily fixed by having wording that says that you need to maintain
working RFC compliant DNS servers for the delegated reverse zones or the
delegation will be removed.  This is a policy failure on ARIN’s part for
failure to make explicit what is implicit in the RFCs.

List the tests you know intend to make.  Mark the list as being non-exhaustive
which allows you to add additional tests in the future.  Start testing and
reporting immediately.  Enforce after next contract renewal.

---

From: Bill Woodcock <[email-masked]>
Date: Thu, 3 May 2018 17:11:31 -0700

> On May 3, 2018, at 4:23 PM, Mark Andrews <[email-masked]> wrote:

Right, of course, but what breaks if you remove the lame nameservers from the NS list?  What am I not understanding?

                                -Bill

---

From: Bill Woodcock <[email-masked]>
Date: Thu, 3 May 2018 17:13:57 -0700

> On May 3, 2018, at 4:28 PM, David Huberman <[email-masked]> wrote:
> 

Errrr, but that’s a special RIR alternate understanding of what lameness means, specific to CIDR, no?

In the general DNS sense, if the NS can answer PTR queries for the thing delegated to it, it’s not lame.  And we’re talking about forward here, not in-addr, and more specifically, not the special CIDR in-addr.

                                -Bill

---

From: Bill Woodcock <[email-masked]>
Date: Thu, 3 May 2018 17:15:22 -0700

> On May 3, 2018, at 4:33 PM, Mark Andrews <[email-masked]> wrote:

Right, but again, that’s not lameness, per se, it’s just a bad configuration.  They’re not denying that they know anything about the zones that are delegated to them.

                                -Bill

---

From: "Paul Hoffman" <[email-masked]>
Date: Thu, 03 May 2018 17:22:35 -0700

Please not the change of the subject line. Discussion of how to fix the 
problem has nothing to do with the definition.

Carry on.

--Paul Hoffman

---

From: Mark Andrews <[email-masked]>
Date: Fri, 4 May 2018 10:24:17 +1000

> On 4 May 2018, at 10:11 am, Bill Woodcock <[email-masked]> wrote:
> 
> 
> 

The problem is that you can’t just remove the NS records from the parent side because name servers learn the NS records from the child side of the delegation as well as the parent side.  The offending NS records need to be removed from both sides.  This (or fixing whatever is broken) is what should be happening when faults are reported.  When that doesn’t happen in a *reasonable* amount of time the *entire* delegation needs to be removed to force the issue.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 4 May 2018 10:34:32 +1000

> On 4 May 2018, at 10:13 am, Bill Woodcock <[email-masked]> wrote:
> 
> 
> 

No.  That is one form of basic DNS lameness.  The server delegated to doesn’t server the delegated zone.

> In the general DNS sense, if the NS can answer PTR queries for the thing delegated to it, it’s not lame.  And we’re talking about forward here, not in-addr, and more specifically, not the special C…

DNS servers need to produce VALID negative answers as well as VALID positive answers for the name space delegated to them.  You need to test for BOTH types of answers.  Failure to produce VALID negative answers causes problems when new types are introduced.

Look at the history of AAAA, SPF, and TLSA records.  All of these are/were issued by clients to see if the records existed at given names.  While these are mostly forward zone issues the same issues could arise in the IN-ADDR.ARPA and IP6.ARPA name spaces.  We don’t know what is coming in the future.  We have however defined how those queries should be answered.  We can test to ensure that correct behaviour occurs.

---

From: John Kristoff <[email-masked]>
Date: Thu, 3 May 2018 20:06:02 -0500

On Thu, 3 May 2018 06:12:42 +0000

> - Responding for the correct domain, but without the authority bit set.

Friends,

I've been referring to a class of problems I call DNS Inconsistency.
This can be thought to be a bit broader in scope than what is typically
meant by a delegation, but it is related.

I'm particularly interested in parent/child NS RRset consistency at the
moment and I recently compiled this list of possible NS RR results when
considering what may happen as one might try to traverse a graph to a
domain name node.  Imperfect list, classifications, and severity
perhaps, but maybe stimulates more useful discussion than it does
detract from it.

--- some amount of broken ---
error           |  bad type (e.g. CNAME)
error           |  bad rdata (e.g. IPaddr for NS)
error           |  TTL disagreement in the NS RRset
error           |  DNSSEC validation error
error           |  timeout/unreachable transient (e.g. down time)
error           |  timeout/unreachable permanent (e.g. misconfiguration)
query_response  |  NXDomain
query_response  |  REFUSED
query_response  |  SERVFAIL / FORMERR / NOTIMP / etc...
query_response  |  referral after a referral
query_response  |  aa==0 when aa==1 expected
query_response  |  malicious or incorrect rdata

--- properly formed answer, but not yet provably correct ---
name            |  maps to an ipv4addr
name            |  maps to an ipv4addr set
name            |  maps to an ipv6addr
name            |  maps to an ipv6addr set
name            |  maps to an ipv4addr + ipv6addr
name            |  maps to an ipv4addr set + ipv6addr
name            |  maps to an ipv4addr + ipv6addr set
name            |  maps to an ipv4addr set + ipv6addr set

John

---

From: Paul Vixie <[email-masked]>
Date: Thu, 03 May 2018 22:47:11 -0400

Mark Andrews wrote:
>> ...
>
> The problem is that you can’t just remove the NS records from the
> parent side because name servers learn the NS records from the child
> side of the delegation as well as the parent side.  The offending NS
> records need to be removed from both sides.  This (or fixing whatever
> is broken) is what should be happening when faults are reported.
> When that doesn’t happen in a *reasonable* amount of time the
> *entire* delegation needs to be removed to force the issue.

+1. it's a cooperative ecosystem. destructive noncooperation can't be 
ignored.

-- 
P Vixie

---

From: Shane Kerr <[email-masked]>
Date: Fri, 04 May 2018 09:33:00 +0000

Ed,

> If I can't find the text soon, I'll try to recreate the list of references at least.

Awesome thanks!

--
Shane

---

From: Shane Kerr <[email-masked]>
Date: Fri, 04 May 2018 09:59:00 +0000

Amreesh,

>    fail, thus creating (more) unnecessary traffic.

A reference! Nice.

Pending Ed's archival research, it seems like we need to actually do
some work to structure the concepts around lameness. Digging in...

Within a given NS RRset for a zone, we have a few failure modes:

A. One or more NS do not resolve
B. NS RR points to a CNAME (technically disallowed, right?)
C. NS RR does not point to any A or AAAA that resolve
D. An A or AAAA RR is for one or more addresses that are not
   authoritative

Case C might not strictly be lame, if for example it points to a .ONION
address or similar.

Case D might be usefully split into addresses that reply and those that
timeout.

I think that all of these fit into the definition of "lame delegation"
in RFC 1713.

I don't know if it makes sense to have any more definitions for anything
in this list beyond that. Probably not.

I think that there may be something useful in creating a term when a
delegation only points to lame servers, thus cannot be resolved at all.
Perhaps "broken delegation"? 😉

There are also a few related issues coming from mismatches at parent &
child.

1. "Lame hint" might describe an NS that is above the zone cut, and
   points to one or more lame servers
2. "Authoritatively lame" might describe an NS that is below the zone
   cut
3. "Totally lame, man" might describe a lame NS that is in both

We can also have:

4. "Confusingly lame" which might describe when there is a mismatch
   between NS answers of authoritative servers, some of which point to
   lame servers 😆

I hesitate to suggest it, but is there value in a draft around lameness?

Cheers,

--
Shane

---

From: Matthew Pounsett <[email-masked]>
Date: Fri, 4 May 2018 09:12:10 -0400

> Case D might be usefully split into addresses that reply and those that

That seems complete to me.

> Perhaps "broken delegation"? 😉

The way this has always worked in my head is that a zone can be delegated
to one or more lame servers.  If the zone is delegated entirely to lame
servers, then the delegation is lame.

But I take your point that perhaps that is too much overloading of the term
'lame'.

>    points to one or more lame servers

The NS set above the zone cut comprises the delegation, doesn't it?  That's
not just a hint.

>    between NS answers of authoritative servers, some of which point to

Now I don't feel so bad about using 'lame server' and 'lame delegation' to
mean not-entirely-overlapping things. :)

> There's value in describing common misconfigurations and how to
detect/name/avoid them.  Does it need to just be a draft about lameness?

---

From: Edward Lewis <[email-masked]>
Date: Fri, 4 May 2018 13:40:30 +0000

Just noticed this (and why terminology is a problem):

On 5/3/18, 17:25, "Mark Andrews" <[email-masked]> wrote:

>Start removing lame delegation ...

Are we talking about "lame servers" or "lame delegations"?  If the latter, is a "delegation" a single NS / glue record or a the set of NS records and associated glue for the owning domain name?  I've been "trained" to think of lame servers, not the entire delegation of a name, when thinking about this kind of maintenance activity.

I've been looking through documents and find, for example, "DNS Resolver MIB Extensions" in May 1994 using the term "Lame Delegation" while "Negative Caching of DNS Queries (DNS NCACHE)" in March 1998 using "lame server."  I.e., the documents are flip-flopping.

I still need more time to trawl the RFCs.  Apparently I haven't kept my cache of them around.

---

From: Joe Abley <[email-masked]>
Date: Fri, 4 May 2018 10:28:51 -0400

On 4 May 2018, at 05:59, Shane Kerr <[email-masked]> wrote:

> Perhaps "broken delegation"?

One thing that I think has been missing from much of this conversation (but not all) is the idea that the vantage point matters when determining if a server is lame for a particular zone [1].

Just because from my vantage point I can't get responses from any of the nameservers cited in a referral response doesn't mean that my experience is universal. Some of the servers might be reachable in other routing domains; some servers might be blocking my queries due to some transient network problem that affects just a subset of its potential clients, or because addresses near me have been participating in an attack, etc, etc.

I am wary of any description that implies that there is some simple measure of lameness that is universal.

Joe

[1] the grammar also seems to vary between opinions. Is a server lame for a domain? Or for a particular query? What do we mean by server, in the case where a single instance of nameserver software running in a single compute environment responds to multiple addresses, e.g. is dual-stacked?

---

From: Joe Abley <[email-masked]>
Date: Fri, 4 May 2018 10:24:09 -0400

Joe

---

From: Edward Lewis <[email-masked]>
Date: Fri, 4 May 2018 15:16:36 +0000

Quick and dirty, using a directory with "rfc$i.txt" files so far up to 5000 or so:

(BTW, my search is: "grep lame * | grep -e delegation -e server | sort -u -k 1,1"
Yes, perhaps not optimal but quick and easy to run.)

rfc1470.txt - mentions a tool called LAMER to detect lame delegations
rfc1612.txt - a MIB definition tracking lame delegations
rfc1713.txt - first example of a lame delegation (when describing LAMER)
rfc1912.txt - first definition, here it is (with grammatical error):

"A lame delegations exists when a nameserver is delegated responsibility for providing nameservice for a zone (via NS records) but is not performing nameservice for that zone (usually because it is not set up as a primary or secondary for the zone)."

rfc2308.txt - author uses "lame server" instead of delegation (poke at Mark)
rfc2832.txt - back to lame delegation, when talking about RRP the "pre-"EPP
rfc3658.txt - defining the DS and uses the term lame nameserver
rfc4074.txt - lame delegation in the context of IPv6
rfc4697.txt - lame or lame server is used

From this quick and dirty pass, my comments:

The concept is uniform - the situation is that the server of interest is not configured to host the zone.  I.e., this is a per-nameserver concept but not a per-cutpoint concept.

The precise term for lame "delegation|server|name server" seems to have never been cemented, although lame delegation seems most common.  Grammatically I'd choose (if I had a vote) for "A lame delegation occurs when a name server is named by a parent to be authoritative for a zone despite the name server not being configured to be authoritative for the zone.  If a name server receives a query for a domain (as opposed to zone!) for which it is not configured, the response is (Blank)."  The latter part of that would be important when writing interoperable code. ;)

I say Blank because there is no universally used response in this situation and I'm not going to promote any solution here.

FWIW, if one is to cite a definition of lame, I'd use "Common DNS Operational and Configuration Errors", February 1996, aka RFC 1912.

---

From: "Paul Hoffman" <[email-masked]>
Date: Fri, 04 May 2018 08:53:17 -0700

On 4 May 2018, at 8:16, Edward Lewis wrote:

> FWIW, if one is to cite a definition of lame, I'd use "Common DNS 
> Operational and Configuration Errors", February 1996, aka RFC 1912.

Yep, that's the one we already have. I think it's worthwhile to also add 
the wider definition that Amreesh pointed out in RFC 1713.

--Paul Hoffman

---

From: Edward Lewis <[email-masked]>
Date: Fri, 4 May 2018 15:57:36 +0000

Ah, yes, I missed the definition in "Tools for DNS Debugging" in quickly skimming this morning.

I read the "A lame delegation is a serious error in DNS configurations, yet a (too) common one." and skipped to the next paragraph, assuming this was yet another definition by example.

On 5/4/18, 11:53, "Paul Hoffman" <[email-masked]> wrote:

    
    
    --Paul Hoffman