Subject: [DNSOP] Measuring DNS TTL Violations in the wild

From: "Giovane C. M. Moura" <[email-masked]>
Date: Fri, 1 Dec 2017 16:48:21 +0100

Hi,

In the light of the recent discussions on TTL violations and server
stale here on the list, I decided to take a look on how often resolvers
perform TTL violations in the wild.

To do that, I used almost 10K Ripe Atlas probes. You can find a report
and datasets at:

https://labs.ripe.net/Members/giovane_moura/dns-ttl-violations-in-the-wild-with-ripe-atlas-2

Now, what was more scary were the violations that *increased* the TTL of
 of RR some more than 10x. That may put users at risk of service domains
that may have been already taken down.

/giovane

ps: related thread on oarc list at :
https://lists.dns-oarc.net/pipermail/dns-operations/2017-November/017039.html

---

From: Ólafur Guðmundsson <[email-masked]>
Date: Fri, 1 Dec 2017 16:38:07 +0000

I strongly disagree with your "terminology", TTL is a hint about maximum
caching period, not a demand or a contract.
A resolver can at any time for any reason discard cached entries.
Many Authoritative operators have "unreasonable" TTL's like less than 10
seconds or multiple days and I see no reason why resolvers do not
apply minimal and/or max caching rules that are reasonable.

Olafur

On Fri, Dec 1, 2017 at 3:48 PM, Giovane C. M. Moura <[email-masked]>
wrote:

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: Jared Mauch <[email-masked]>
Date: Fri, 1 Dec 2017 11:43:50 -0500

> On Dec 1, 2017, at 11:38 AM, Ólafur Guðmundsson <[email-masked]> wrote:
> 
> 
> I strongly disagree with your "terminology", TTL is a hint about maximum caching period, not a demand or a contract. 

Agreed.

> Many Authoritative operators have "unreasonable" TTL's like less than 10 seconds or multiple days and I see no reason why resolvers do not 

Yes, I remember a load balancer from the last century that had serious issues with DNS requests with low TTLs.  We ended up replacing it.

TTL is certainly a MAX, but no reason you can’t return a shorter value.  My stub resolver may see a lower number if an item is about to be evicted from cache, should we not see that?  Clamping the max seems helpful and causes the least enduser harm, so is quite wise.  

The same would be true hitting a large anycast dns server like 75.75.75.75 or 8.8.8.8, 4.2.2.1, you may hit a different backend for whatever reason so see varying TTLs for the same query within a 10 second interval based on that.  That’s not bad, it’s working as designed.

I think measurements are interesting though, so identifying TTL clamping in the wild would be an interesting study.

- jared

---

From: "Wessels, Duane" <[email-masked]>
Date: Fri, 1 Dec 2017 17:02:09 +0000

> On Dec 1, 2017, at 8:38 AM, Ólafur Guðmundsson <[email-masked]> wrote:
> 

You say its just a hint.  If you put a TTL of 1 hour on your data, and I have a recursive name server that reuses it for 2 hours, 12 hours, 5 days... thats okay?

If its just a hint then we are we spending all this effort on "serve stale"?

DW

---

From: Ólafur Guðmundsson <[email-masked]>
Date: Fri, 1 Dec 2017 17:16:47 +0000

On Fri, Dec 1, 2017 at 5:02 PM, Wessels, Duane <[email-masked]>
wrote:

> You say its just a hint.  If you put a TTL of 1 hour on your data, and I
> have a recursive name server that reuses it for 2 hours, 12 hours, 5
> days... thats okay?
>
Strictly speaking yes, it is the same as when a Secondary does not update
the zone for a long time.
DNS is not a strict coherency protocol, thus playing  loose with the time
things are with in reason is ok.
Stretching TTL from 1 Hour to 1 day is IMHO BAD, doing it for 10% or 10
minutes is fine.
We are getting into religion here, the original poster called people that
cap TTL's Heretics,
I disagree with that labeling of myself and others that are applying sane
caps.

Olafur

---

From: "Paul Hoffman" <[email-masked]>
Date: Fri, 01 Dec 2017 09:23:38 -0800

> We are getting into religion here, the original poster called people that

Looking through the mail archives, no one other than you is using that term.

--Paul Hoffman

---

From: Jared Mauch <[email-masked]>
Date: Fri, 1 Dec 2017 12:52:57 -0500

> On Dec 1, 2017, at 12:23 PM, Paul Hoffman <[email-masked]> wrote:
> 

I think this is subject to interpretation, some people view the done differently.
The subject line felt hostile.. 2nd attempt to adjust subject-line to make it less hostile.

- jared

---

From: Steve Crocker <[email-masked]>
Date: Fri, 1 Dec 2017 13:12:34 -0500

I would be very interested in a bit more precision here.  Is there a way to say what is permissible vs impermissible re TTLs, and is there a way to say what is desirable vs undesirable re TTLs?  We all understand that longer TTLs reduce the frequency of refresh at the expense of slower response whenever the authoritative information changes.  However, some fraction of the recursive resolvers impose minimum and/or maximum limits on the TTLs they receive from the authoritative servers.

Shortening TTLs increases the amount of traffic between the recursive resolvers and authoritative resolvers and lengthens the response time for some queries.  However, I don’t think there is any service guarantee with respect to an individual query that is violated by shortening the TTL.

Lengthening a TTL, on the other hand, does change one of the service guarantees.  When there is a change in the entry in the authoritative server, what is the maximum time until that change is guaranteed to be propagated throughout the net?  This depends primarily on the TTL.  However, when the TTL is lengthened by the recursive resolvers, the upper bound for propagation of a change is similarly increased.

Is there any common understanding of how much lengthening is permitted?  Is commonplace?

Let me make a guess that the only lengthening that takes place in practice is a floor of ten seconds.

Comments?

Thanks,

Steve

> On Dec 1, 2017, at 12:52 PM, Jared Mauch <[email-masked]> wrote:
> 
> 
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 2 Dec 2017 11:29:35 +0100 (CET)

> practice is a floor of ten seconds.

I might be misinterpreting, but from the data presented in the graph in 
section 3.2 it looks like some will increase TTL to 7200 seconds at the 
highest. There seems to be large bumps at the 600, 1200 and 1800 second 
"minimum TTL" capping (if I guess correctly from looking at that graph).

It would be interesting to hear what problems these operators are trying 
to solve by implementing these minimums. 7200 seconds does seem like a 
pretty high value to lower bound TTLs at.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Åke Nordin <[email-masked]>
Date: Sat, 2 Dec 2017 12:19:29 +0000

> pretty high value to lower bound TTLs at.

Whoah, extending TTL to 2h becomes pretty bad when the name was nefarious
and taken down within, say, 15 minutes. That renders a lot of good people's
work less useful.

BR,

--
Åke Nordin <[email-masked]>, resident Net/Lunix/telecom geek.

---

From: Mukund Sivaraman <[email-masked]>
Date: Sat, 2 Dec 2017 20:09:25 +0530

> the zone for a long time.

An authoritiative server operator knows what the consequence of setting
SOA RDATA fields is. It isn't the same as a cache extending TTL as it
sees fit, in spite of the loose coherency among primary and secondaries.

I don't agree a downstream cache has authoritiative say about extending
TTLs (except exceptional circumstances where the authority is
unreachable ~serve-stale).

		Mukund

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Mon, 4 Dec 2017 09:53:46 +0100

Hi,

> The subject line felt hostile.. 2nd attempt to adjust subject-line to make it less hostile.

Oh I apologize for the terminology issue. I actually borrowed the term
from two previous academic papers: [1] and [2].

[1] http://www.cs.wm.edu/~haos/papers/sigcomm-ccr-dns.pdf
[2] https://dl.acm.org/citation.cfm?doid=3143361.3143375

/giovane

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 5 Dec 2017 17:46:49 +0100

On Fri, Dec 01, 2017 at 01:12:34PM -0500,
 Steve Crocker <[email-masked]> wrote 

> Lengthening a TTL, on the other hand, does change one of the service guarantees.  When there is a change in the entry in the authoritative server, what is the maximum time until that change is guar…

I fully agree about this very important difference (which is
completely missing in the RIPE Labs article). Lengthening the
TTL is a protocol violation (otherwise, we wouldn't discuss about
draft-ietf-dnsop-serve-stale). Shortening it systematically is bad
manners, and is selfish but is not a protocol violation.

We should not discuss the two in the same thread: they are very
different practices.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 5 Dec 2017 16:07:16 -0500

> unreachable ~serve-stale).

I will note that this WG spent a fair amount of effort on RFC 7583.
That text is actually bad advice if you are supposed to expect
resolvers to extend the TTL on RRsets, because the Ready state depends
on caches expiring.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: 神明達哉 <[email-masked]>
Date: Tue, 5 Dec 2017 13:09:54 -0800

At Sat, 2 Dec 2017 20:09:25 +0530,

> unreachable ~serve-stale).

+1.  I'd accept some level of liberty that an implementation can take,
such as ISC BIND 9 extending a 0-TTL of glue to 1 second:

        /*
         * Glue with 0 TTL causes problems.  We force the TTL to
         * 1 second to prevent this.
         */
        if (rdataset->ttl == 0)
            rdataset->ttl = 1;

but it should be limited to a quite small range.  How much is
acceptable may be debatable, but I wouldn't consider "Stretching TTL
from 1 Hour [...] for 10% or 10 minutes" to be acceptable at the
discretion of an implementation.

--
JINMEI, Tatuya

---

From: Lanlan Pan <[email-masked]>
Date: Wed, 06 Dec 2017 04:04:41 +0000

Mukund Sivaraman <[email-masked]>于2017年12月2日周六 下午10:39写道：

> > > > On Dec 1, 2017, at 8:38 AM, Ólafur Guðmundsson <
> [email-masked]>
> > > wrote:
> > > >

Some authorititatives set the NS RR TTL<60s, they don't follow the best
practice guide.
Authoritatives think they "do the right thing" to spread latest NS RR.
Needless to say A RR.

Some resolvers  extend all RR's TTL<60s  to 3600s，just to reduce the
queries, when the authority is reachable.
Resolvers think they "do the right thing", too.

-- 
致礼  Best Regards

潘蓝兰  Pan Lanlan

---

From: Joe Abley <[email-masked]>
Date: Wed, 6 Dec 2017 06:17:36 -0500

On Dec 5, 2017, at 23:04, Lanlan Pan <[email-masked]> wrote:

> Some authorititatives set the NS RR TTL<60s, they don't follow the best practice guide.

The trouble here is understanding the motivations of any particular parameter, and doing so at scale.

You could assume as a resolver operator that a zone manager (note, not an authoritative server operator) has made a mistake when you get a response with a low TTL, or you could take the position that it's not your business to make such inferences.

Local policy to meet your own objectives in the operation of your resolver is different from seeking to correct decisions made by zone managers. Assuming errors (hence "correction") is a slippery slope to having no single source of truth.

There are protocol design decisions relating to TTLs that relate to metadata that a resolver operator *can't* reasonably adust, like signature validity periods.

More philosophically, the loose-coherency and data persistence associated with caching works because its boundaries are understood. The ability to use the system as a whole relies upon an explicit contract between zone manager and resolver operator, and if that contract fractures, applications and user-experience will follow. Any changes to the contract (and I'm not suggesting there can't be changes) need to be conservative, well documented and carefully implemented. Playing fast and loose with the interpretation of TTLs on every other resolver doesn't sound like any of those things.

Joe