Subject: [DNSOP] "The Forgotten Object Lesson Of The Dyn DDoS Attack"

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 3 Jan 2019 12:03:00 +0100

https://www.forbes.com/sites/forbestechcouncil/2018/12/19/the-forgotten-object-lesson-of-the-dyn-ddos-attack/

This article talks about "There have been discussions within the
Internet Engineering Task Force, the organization responsible for
developing and enhancing internet protocols, to come up with some
standard means of specifying and synchronizing these value-added
services" and I believe Cricket Liu refers to draft-woodworth-bulk-rr
(now expired, never adopted here).

---

From: Töma Gavrichenkov <[email-masked]>
Date: Thu, 3 Jan 2019 16:02:45 +0300

On Thu, Jan 3, 2019 at 2:03 PM Stephane Bortzmeyer <[email-masked]> wrote:
> I believe Cricket Liu refers to draft-woodworth-bulk-rr

Certainly not.  See
https://portswigger.net/daily-swig/middle-aged-dns-tech-still-has-legs-to-kick-on
from the same author where the idea is explained kind of better:

"The challenge really is that DNS hosting providers like Dyn or
Neustar tend to offer these value-added services, such as load
balancing or traffic management services. Those services are
non-standard.
It’s very easy to synchronize basic DNS configuration because there’s
a built-in mechanism to DNS called zone transfers that will
synchronize that data. With this value-added stuff there’s no easy way
to synchronize that. That’s the challenge."

If I were to trace that through the recent DNSOP activity, I could
bring up my own draft (draft-gavrichenkov-dnsop-dnssapi), also not
adopted and now expired.  Maybe there were discussions of the same
sort before me that I'm not aware of.

---
Töma

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 3 Jan 2019 16:25:49 +0000

> sort before me that I'm not aware of.

Or he was possibly talking about draft-huque-dnsop-multi-provider-dnssec, which expired yesterday, and also has not been adopted.

--Paul Hoffman

---

From: Tim Wicinski <[email-masked]>
Date: Thu, 3 Jan 2019 21:33:23 -0500

Actually  draft-huque-dnsop-multi-provider-dnssec was adopted, but the
author (whom I work with and has heard from
me about this regularly) has failed to push up an updated version.   I'm
going to force him to turn the authorship
over to one of the other authors who is more responsive to the needs of the
chairs.

Tim

On Thu, Jan 3, 2019 at 11:26 AM Paul Hoffman <[email-masked]> wrote:

> Or he was possibly talking about draft-huque-dnsop-multi-provider-dnssec,
> which expired yesterday, and also has not been adopted.
>
> --Paul Hoffman_______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: Steve Crocker <[email-masked]>
Date: Thu, 3 Jan 2019 22:01:29 -0500

Weighing in on this, I think this is an important piece of work.  I’m particularly interested in what else is necessary to introduce the multiple dns operators to each, to authorize their cooperation, and facilitate cross signing of keys whenever a new operator is introduced and whenever one or more of the operators rolls its/their key(s).

This draft is framed in terms of having multiple dns operators serve the same zone on a continuing basis.  An important corner case is the transfer of a signed zone from one operator to another without loss of resolution and without loss of validity.

If I can be helpful moving this forward, I’ll be glad to help.

Steve Crocker

Sent from my iPhone

> On Jan 3, 2019, at 9:33 PM, Tim Wicinski <[email-masked]> wrote:
> 
> Actually  draft-huque-dnsop-multi-provider-dnssec was adopted, but the author (whom I work with and has heard from 
> me about this regularly) has failed to push up an updated version.   I'm going to force him to turn the authorship
> over to one of the other authors who is more responsive to the needs of the chairs.
> 
> Tim
> 
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]

---

From: Shumon Huque <[email-masked]>
Date: Thu, 10 Jan 2019 10:34:59 -0500

Quick backstory for the group ..

Hi Tim,

As you know, I had been working on the draft. A new version of the -00
draft was mostly ready to go in the github repo. I was mainly waiting for
confirmation from one of the new listed authors on the draft that it was
okay for his name to be added. And since I was about to leave for vacation
for 2 weeks, I decided to wait until after I returned to push the new draft
even though it would have expired for a few days by then (I guess I should
have informed the chairs of that). One of my co-authors has pushed the -00.
There are a few more edits in the pipeline, including addressing Steve
Crocker's request to add the inter-provider handoff subcase to the
document, so you'll be seeing an -01 soon.

Shumon.

---

From: Warren Kumari <[email-masked]>
Date: Fri, 18 Jan 2019 17:16:56 -0500

> over to one of the other authors who is more responsive to the needs of

draft-ietf-dnsop-multi-provider-dnssec (nee
draft-huque-dnsop-multi-provider-dnssec) only solves part of the problem,
and for many of the Dyn customers it wouldn't have solved enough of the
problem to allow them to use multiple providers.

Scenario:
I have a service which is served from 8 datacenters on 2 CDNs - I use a
service like Dyn's Traffic Director to provide geo balancing, to only send
traffic to hosts which are up **and to share the load**.
The first two are relatively trivial to support with multiple DNS providers
- they can independently do geo, they can independently do health checks --
but unless the service does a really good job of exposing its current
available capacity and committed capacity at good resolution, having
independant load balancing gets tricky really quickly.
A single DNS provider can know that location B can serve Nqps, can infer
offered load from how many times it has directed load to B the last X
seconds - them means it can overflow to C when B is *approaching* capacity.
Having a second provider who is unaware of the first one's actions creates
a very real risk of synchronization -- both providers choose B, and so B
gets overloaded - so they both choose C. C is now overloaded, and B is
underutilized, so they both choose B, etc.

This is a well known issue in service load-balancing, in routing protocols
which take circuit utilization into account, etc. It *can* be mitigated /
solved with careful dampening of a Markov decision process, much higher
resolution probing, having one of the providers simply lag by 1/2 the
reaction time, etc.
Balazs Csanad Csaji and Laszlo Monostori's "Stochastic Reactive Production
Scheduling by Multi-agent Based Asynchronous Approximate Dynamic Programming"
(http://old.sztaki.hu/~csaji/csaji-ceemas-2005.pdf) and Down and
Lewis' "Dynamic
Load Balancing in Parallel Queueing Systems: Stability and Optimal Control"
(https://people.orie.cornell.edu/melewis/pubs/load.pdf) are both good reads
on the topic.

There was also a fascinating web page where someone wrote up a system to
avoid exceeding a service's capacity / this sort of issue by using a
standard servo PID controller (https://en.wikipedia.org/wiki/PID_controller)
-- he mapped the "sensed position" as the job capacity, and used the output
of the PID to offer load to the service.
It was obviously one of those of those late-night "Oooh, I wonder what
would happen if..."-type ideas (which morphed into "Here, hold my beer"),
but it actually worked surprisingly well (and was really funny).

I spent some time looking for the page, but cannot find it at the moment...
:-(

W

-- 
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
   ---maf