Subject: [DNSOP] Ask for advice of 3 new RRs for precise traffic scheduling

From: "[email-masked]" <[email-masked]>
Date: Wed, 13 Dec 2017 15:40:50 +0800

Hi  everyone,
 
 Here’s a problem about CDN traffic scheduling.  So far as I know, many business companies use multi-CDN to speed up their websites and the CDN providers have requirements for precise traffic scheduling especially in the rush hour of traffic. 
        
 As CDN providers usually manage authoritative DNS for their clients, the most common method for real-time traffic scheduling is to change the A records of  CDN nodes. It does have some positive effects. But because of the lack of DNS protocol support (especially on the recursive server side), a CDN company can’t schedule traffic very precisely. 
 For example, a CDN provider can’t schedule 70% of traffic to node A and 30% of traffic to node B.  Even though it places the addresses of both A and B, it can’t determine recursive server’s response to clients. For example, some recursive server may round-robin the address to clients.
 
For better precise CDN traffic scheduling, I have an idea that defines 3 new records from extending 3 existing DNS resource records: A, AAAA and CNAME, by adding a “weight” attribute，as below:
   [<owner> <ttl> <class> CNAME <target>]  -> [<owner> <ttl> <class> CNAMEX <weight> <target>] 
   [<owner> <ttl> <class> A <address>]  ->  [<owner> <ttl> <class> AX <weight> <address>]
   [<owner> <ttl> <class> AAAA <address>]  ->  [<owner> <ttl> <class> AAAAX <weight> <address>]
 
The reasons for doing this are :
(1) By adding “weight” in CNAMEX, a multi-CDN user can manage the traffic ratio among different CDN providers by itself easily.
(2) By adding “weight” in A/AAAA, a CDN provider can manage the traffic ratio among different nodes by itself easily.
 
 For compatibility, an authoritative server should place the CNAMEX/AX/AAAAX records in additional section in a DNS response for a “A/AAAA” query. A “weight-aware” recursive server should make use of the “CNAMEX/AX/AAAAX” in the additional section to manage the answer to clinents according to the weight of each RR. A “weight-not-aware” recursive server can just ignore these RRs and still work normally.
 
 Here is an example:  If a CDN provider configures AX records for “www.example.com” as below which indicates “1.1.1.1” should account for 80% in response while “2.2.2.2” accounting for 20%.  A “weight-aware” recursive server should reply to clients accordingly.
      Any comment or advice is highly appreciated!
     Thank you!!

[email-masked]

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 13 Dec 2017 09:18:23 +0100

On Wed, Dec 13, 2017 at 03:40:50PM +0800,
 [email-masked] <[email-masked]> wrote 
 a message of 1343 lines which said:

>  For example, a CDN provider can’t schedule 70% of traffic to node A
>  and 30% of traffic to node B [...] adding a “weight” attribute

First, the obvious question: why reinventing RFC 2782?

---

From: bert hubert <[email-masked]>
Date: Wed, 13 Dec 2017 09:43:42 +0100

> First, the obvious question: why reinventing RFC 2782?

Implementing this worthwhile capability can be done in four ways/places:

1) Get browsers to honour RFC 2782

2) Get resolvers and auths to support a weighted A/AAAA record (as proposed
in this thread)

3) Serve up multiple copies of the same A record, and weigh like this:
www IN A 1.2.3.4
www IN A 1.2.3.4
www IN A 10.11.12.13
And hope that record shuffling will deliver the 2:1 ratio

4) Get authoritative servers to serve A/AAAA with weighted frequency and
short TTL

Getting browsers to move is a 5 year project at least. You could get the
resolver/auth landscape moving somewhat more quickly ('we know these
people'), but it will still take a long time and support will be spotty.

The 'multiple IP address listings' thing is done in practice, but some
server remove duplicates, so it doesn't work that well.

And the last possibility is what CDNs are actually doing. It does not quite
spread out traffic perfectly, but it is good enough.

In PowerDNS Authoritative Server 4.1.1 (upcoming in January), this looks
like:

@ IN LUA A "wrandom{{2, '1.2.3.4'}, {1, '10.11.12.13'}}"

Or even:

@ IN LUA A "whashed{{2, '1.2.3.4'}, {1, '10.11.12.13'}}"

Which will keep the same IP address going to the same record.

This is documented in https://powerdns.org/auth-4.1.1-docs/lua-records.html
- which also lists things like @ IN LUA A "closest{'1.2.3.4', '10.11.12.13'}"
which will attempt to serve up the geographically closest address.

     Bert

---

From: "[email-masked]" <[email-masked]>
Date: Wed, 13 Dec 2017 17:31:06 +0800

thanks for your comments!

According to my understanding, here are 2 main differences between RFC2782 and this idea :

(1) RFC2782 does not solve the problem of using multi-CDN  (multiple CNAME cannot coexist);
    here we can use multipile weighted CNMAEXs (need to coexist with CNAME) to accomplish this;
    besides, weighted CNAMEXs can control traffic ratio among several CDN providers;

(2) RFC2782 requires browser's support;
    Using this method, a browser has no idea about weighted AX/AAAAXs. 
    It requires no change of browsers.

[email-masked]
 
From: bert hubert
Date: 2017-12-13 16:43
To: Stephane Bortzmeyer
CC: [email-masked]; dnsop
Subject: Re: [DNSOP] Ask for advice of 3 new RRs for precise traffic scheduling

 

 

 

Or even:

 
 
     Bert

---

From: [email-masked]
Date: Wed, 13 Dec 2017 18:34:45 +0900 (JST)

> From: bert hubert <[email-masked]>

Same RDATA is not allowed by RFC 2181.

| 5. Resource Record Sets
|
|   Each DNS Resource Record (RR) has a label, class, type, and data.  It
|   is meaningless for two records to ever have label, class, type and
|   data all equal - servers should suppress such duplicates if
|   encountered.  It is however possible for most record types to exist
|   with the same label, class and type, but with different data.  Such a
|   group of records is hereby defined to be a Resource Record Set
|   (RRSet).

> 1) Get browsers to honour RFC 2782

  RFC 6763 DNS-Based Service Discovery shows "_http._tcp" service usage.

> From: "[email-masked]" <[email-masked]>

  You can add many SRV RRs. (merge SRV RRs from multi-CDN).

>     It requires no change of browsers.

  Softwares that support DNS-SD may support _http._tcp.

--
Kazunori Fujiwara, JPRS <[email-masked]>

---

From: "[email-masked]" <[email-masked]>
Date: Wed, 13 Dec 2017 17:36:32 +0800

Or even:

 
--------------> [zuopeng]  :  so far as i know, many CDNs already use similar methods as you mentioned in PowerDNS 4.1.1 
--------------> [zuopeng]  :  but  i think only the  Authoritative Server change is not enough,  support on the recursive server is also very important .
--------------> [zuopeng]  :   because the resolver determines the reponse to clients.

 
     Bert

---

From: bert hubert <[email-masked]>
Date: Wed, 13 Dec 2017 10:50:15 +0100

On Wed, Dec 13, 2017 at 05:36:32PM +0800, [email-masked] wrote:
>  so far as i know, many CDNs already use similar methods as you mentioned in PowerDNS 4.1.1 
>  but  i think only the  Authoritative Server change is not enough,  support on the recursive server is also very important .
>   because the resolver determines the reponse to clients.

This is true.  A typical resolver will serve around 50,000 to 2,000,000
users (although this is rare). This means that for 60 seconds, you shift
around 'a hundred thousand' potential users. 

In practice, this appears to be good enough from what I hear.

Or let me put it another way, before we burden the DNS protocol with another
record type we have to add downgrade/workaround/DNSSEC support for, we
should have numbers that say it solves a problem.

CDNs could maybe chime in.

	Bert

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 13 Dec 2017 10:55:34 +0100

On Wed, Dec 13, 2017 at 05:31:06PM +0800,
 [email-masked] <[email-masked]> wrote 

> (2) RFC2782 requires browser's support;

Client support. This is even worse, there are much more HTTP clients
than browsers.

>     It requires no change of browsers.

But it requires change in all resolvers. One may wonder which will be
the fastest path.

---

From: Lanlan Pan <[email-masked]>
Date: Wed, 13 Dec 2017 10:25:20 +0000

> But it requires change in all resolvers. One may wonder which will be
something like ANAME ?
ANAME can let resolvers choose IP for clients.

>
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>
-- 
致礼  Best Regards

潘蓝兰  Pan Lanlan

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 14 Dec 2017 09:36:54 +0800

From: Lanlan Pan
Date: 2017-12-13 18:25
To: Stephane Bortzmeyer
CC: [email-masked]; dnsop; bert hubert
Subject: Re: [DNSOP] Ask for advice of 3 new RRs for precise traffic scheduling

-----------[zuopeng] it is not like ANAME. But ANAME has similar function with CNAME, adding a weight attribute for ANAME may also make sense.

潘蓝兰  Pan Lanlan

---

From: Robert Edmonds <[email-masked]>
Date: Thu, 14 Dec 2017 16:40:56 -0500

> CDNs could maybe chime in.

Hi,

With my CDN hat on, I don't see any need to turn over scheduling
decisions to resolvers. Extremely precise amounts of traffic can already
be scheduled to individual CDN nodes because you have a large pool of
owner names to work with, not a single owner name, and every (QNAME,
QTYPE, resolver IP, client subnet [if present], anycast location that
receives the query) tuple is an opportunity to make a unique scheduling
choice. Generally, however, assignments to CDN nodes should be
relatively sticky. You want to be shifting traffic for performance
reasons, not capacity reasons.

Or, put another way, we like existing resolver implementations just
fine, we just wish there were a lot more resolver instances, and closer
to clients :-)

-- 
Robert Edmonds

---

From: Paul Vixie <[email-masked]>
Date: Thu, 14 Dec 2017 13:42:37 -0800

Robert Edmonds wrote:
> to clients:-)

on it. no smiley.

-- 
P Vixie

---

From: 左鹏 <[email-masked]>
Date: Fri, 15 Dec 2017 16:19:44 +0800 (GMT+08:00)

thanks for your comment!

> -----原始邮件-----
> 发件人: "Robert Edmonds" <[email-masked]>
> 发送时间: 2017-12-15 05:40:56 (星期五)
> 收件人: "bert hubert" <[email-masked]>
> 抄送: "[email-masked]" <[email-masked]>, dnsop <[email-masked]>
> 主题: Re: [DNSOP] Ask for advice of 3 new RRs for precise traffic scheduling
> 
> bert hubert wrote:

The "precise traffice scheduling" i mentioned is not only for performance reason, but also for capacity reason.
For performance, i think the existing system already works fine. As long as a CDN DNS has "view" function, the local DNS can 
bring the client to the closest CDN nodes easily.  
For capacity, here is a user case. I learned this case from people working in a CDN company,though i am not sure if it is a 
common problem worldwide. For example, a CDN provider has 10G capacity in province A and 8G in province B. (B is close to A 
geographically).  For some reason(eg. large population in A), 10G is not enough in the rush hour. Assume there is 12G trafffic during rush hour, the CDN needs to schedule 2G to province B while keep 10G in A.  

> fine, we just wish there were a lot more resolver instances, and closer

yes, more resolvers are good to improve user experience.
but also maybe we should notice each CDN node has different capacity.(it is the real in practice).
a "weight-aware" rosolver can schedule clients to diffent nodes according to weight pricisely.
or shall we do something only for authoritative server like defining the weighted A/AAAAx? 

btw, any comments on the weightd CNAMEXs for multi-CDN? :)

---

From: Paul Vixie <[email-masked]>
Date: Fri, 15 Dec 2017 09:16:45 -0800

On Friday, December 15, 2017 04:19:44 PM 左鹏 wrote:
> but also maybe we should notice each CDN node has different capacity.(it is
> the real in practice). a "weight-aware" rosolver can schedule clients to
> diffent nodes according to weight pricisely.

load vs. capacity information varies too often to publish guidance about it in DNS. if a 
CDN can topolocate an HTTP initator, it can vary its answer according to load, far 
better than it can predict or estimate both load, capacity, and topolocation in a way 
expressible in the DNS.

> or shall we do something only
> for authoritative server like defining the weighted A/AAAAx? 

we should do neither. the path you are recommending increases complexity by a 
greater order of magnitude than any possible or prospective resulting benefit.

-- 
P. Vixie

---

From: Robert Edmonds <[email-masked]>
Date: Fri, 15 Dec 2017 14:32:09 -0500

> The "precise traffice scheduling" i mentioned is not only for performance reason, but also for capacity reason.

CDNs already have the ability to do precise traffic scheduling for both
performance and capacity reasons, using the existing capabilities built
into the DNS.

> or shall we do something only for authoritative server like defining the weighted A/AAAAx? 

No, the DNS's existing capabilities provide more than enough power for
CDNs for these use cases.

> btw, any comments on the weightd CNAMEXs for multi-CDN? :)

I don't have any direct experience of the multi-CDN use case. I would
think the intra-CDN case for CDN node selection can be generalized to
the multi-CDN case for CDN provider selection, though you probably have
fewer owner names to work with.

-- 
Robert Edmonds