Subject: [dnsext] client-subnet - questions for the authors, and outside-the-box brainstorming

From: Brian Dickson <[email-masked]>
Date: Tue, 1 Jun 2010 09:15:04 -0300

One of the issues concerning the CDN <-> "big cache operators" desire to
"standardize" and extend "stupid DNS tricks", is how those changes to the
protocol might better be accomplished, so as to add benefit everyone.

It certainly is a worthy goal in standardizing things, but the more edge use
cases get folded into the overall DNS protocol family, the more balkanized
the protocol risks becoming.

The open question is, are there similar means to accomplishing the same goal
(pushing CDN dns "tricks" to the cache operators' farms), which have better
overall design (scaling, efficiency, timeliness, neutrality WRT CDN
architecture), and which may have additional benefit to the overall DNS
community?

The main problem, as I see it, is the need to provide "off-axis thrust" -
giving answers which differ according to the location of the stub resolver,
rather than the intermediate recursive resolver(s).

And the main issue to be addressed, is the notion of these "resolvers"
having some "special" behavior/semantics that differs from the traditional
caching resolver in very substantial ways, at the DNS architectural level.

Here's how I think this might better be addressed, which BTW introduces a
new and different problem to solve, which I think *will* be of interest to
the whole DNS community, and in fact has previously been established as an
area of interest to the WG, and needs to be picked up and further developed:

Rather than having the "big caches" being true caches, in the current sense
of what a cache is, change the relationship between the CDN DNS servers and
these "BC"s to that of master and slave, on the zones served by the masters.

This changes the overall flow of information, and changes where the "dns
tricks" get done, to the place where the original data needed (the
real-world client IP address) is available - i.e., the BC.

Timeliness is achieved by the pushing of zone updates, via notify/XFR.

Stub clients get the relevant CDN-specific answers, quite possibly achieved
by way of "views" on the zone data.

The tools for doing this already exist, except for one area, which is where
this WG already has interest: automated provisioning of slave zones, to
bootstrap the operational update of secondary servers.

IMHO, having a strong motivation to handle this, means if this alternate
design is accepted and adopted by the CDN/BC folks, we can expect good work
from them in this area - something which will be of benefit to the whole of
the DNS community.

And, it keeps the "stupid DNS tricks" localized to authoritative servers
(even if they are operated by third parties, per se), which is where they
belong, IMHO.

Brian

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 01 Jun 2010 21:42:10 +0900

> The open question is, are there similar means to accomplishing

For HTTP, redirection just works.

						Masataka Ohta

---

From: Martin Barry <[email-masked]>
Date: Tue, 1 Jun 2010 16:09:23 +0200

$quoted_author = "Brian Dickson" ;
> by way of "views" on the zone data.

This bilateral approach:
- does not scale
- forces a "one size fits most" approach to the "stupid DNS trick" algorithm
- would be easier as a private implementation of what EDNS-client-ip
  envisions

cheers
Marty

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 1 Jun 2010 07:46:41 -0700

> For HTTP, redirection just works.

For HTTP, redirection JUST DOES NOT WORK FOR THIS, for two very important reasons [1]:

1:  HTTP redirection adds 3+ RTTs of additional latency, between the TCP handshake and the 302 redirect and the new DNS lookup needed for the customized name.  If your goal is to shave a second off of page load, this eliminates the benefit.

2:  HTTP redirection can only provide a different URL, which is unacceptable for many web site providers.  You don't want to see

http://a351.www.mysite.com/URL

or

http://273.51.152.23/URL

presented to the user, which is what must occur for HTTP redirect, since there is no notion of "the IP for servicing the URL should be Y but the URL should not be changed", and you don't want to redirect in-page sub-pieces because that is much slower and breaks the notion of relative URLs for components. [2]

[1] Yes, I'm feeling very repetitive here. 

[2] We actually DO leak host-specific URLs, but only very reluctantly since we wanted to be VERY careful on same origin policy for Netalyzr, which required each back end node to have a separate DNS name.  I wish people did NOT see "n3.netalyzr..." etc, but we did it for a reason.

---

From: Brian Dickson <[email-masked]>
Date: Tue, 1 Jun 2010 12:41:29 -0300

> > by way of "views" on the zone data.

It's only bilateral when implemented trivially. Nothing in the overall
design (master/slave) differs from current use of master/slave, i.e.
non-trivial graphs among or between parties.

E.g. CDN operator has hidden master(s), with numerous slaves which are
public masters. Large cache operators subscribe to multiple CDN operator
zones. Smaller parties can potentially get "feeds" of this DNS data
aggregated, from large cache operators.

The basic model looks quite a bit like BGP. This is all control plane data.
Routes and "stupid DNS trick data" look quite a lot a like, when you look at
them from the right angle (preferably at a bar BOF). :-)

> - does not scale

... yet. I.e. it needs the provisioning pieces to scale, which was what my
post was about, that the provisioning pieces are useful not just to this use
case, but to numerous others.

> - forces a "one size fits most" approach to the "stupid DNS trick"

I prefer "permits" or "enables" rather than forces. This is all opt-in.

> - would be easier as a private implementation of what EDNS-client-ip
No arguments there.... :-)

Brian

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 02 Jun 2010 05:57:43 +0900

> For HTTP, redirection JUST DOES NOT WORK FOR THIS,

IT DOES.

> 1:  HTTP redirection adds 3+ RTTs of additional latency,
> between the TCP handshake and the 302 redirect and the new
> DNS lookup needed for the customized name.  If your goal is
> to shave a second off of page load, this eliminates the benefit.

If initial RTTs were the problem, never use DNS for extra several
RTTs. Instead, use:

	http://273.51.152.23/URL

Not that cache does not help the initial query.

Or, even if RTTs of DNS are allowable but additional 3 RTTs were
unacceptable, there is no point to use open resolver services
with remote servers at a distance with large RTTs. If the
resolver services is extensively anycast, you are done.

But, the reality is that initial several RTTs are acceptable.

> 2:  HTTP redirection can only provide a different URL, which is
> unacceptable for many web site providers.  You don't want to see
> 

If you use "www.yahoo.com", you'll see a lot of "a.l.yimg.com"
etc. used.

Try it to see the reality of HTTP.

> [1] Yes, I'm feeling very repetitive here.

And you still don't know the initial RTTs are not a problem.

Why?

					Masataka Ohta

---

From: Colm MacCárthaigh <[email-masked]>
Date: Tue, 1 Jun 2010 14:46:15 -0700

>> For HTTP, redirection JUST DOES NOT WORK FOR THIS,

Ouch, all this shouting, my poor ears, For comparison, below are two
scenarios - one with HTTP tricks, one with DNS tricks. In both cases
we want a user going to http://example.com/ to be directed to their
nearest internet node that serves that website, the goal is speedy
delivery.

C = client, R = resolver, A = authoritative nameserver , I =
intermediary HTTP server, S = Ultimate HTTP server

 (i) DNS tricks

 Client queries the resolver for example.com
 Resolver queries the auth server, auth server responds
 Resolver responds to client
 Client initialises TCP connection to Ultimate HTTP server

 Total network time to first byte;    |CR| + |RA| + 2x |CS|

 (ii) HTTP tricks

 Client queries the resolver for example.com
 Resolver queries the auth server, auth server responds
 Resolver responds to the client
 Client initiaises TCP connection to Intermediary HTTP server
 Client Receives redirect to "192.0.2.1.example.com" say,
 Client queries the resolver for this new name.
 Resolver queries the auth server, auth server responds
 Resolver responds to the client
 Client initialises TCP connection to ultimate HTTP server

 Total network time to first byte:  2x |CR| + 2x |RA| + 2x |CI| + 2x |CS|

  Total difference between the two;  |CR| + |RA| + |CI|  .

Obviously these are the boundary cases, which involve cache misses.
Each have different caching properties, the second example can use
more long-lived TTLs for example, whereas the first example - with the
option we are discussing - can perform CIDR-level caching. However
obviously for at least some number of queries which are misses, things
will be much worse.

Some have stated that HTTP level redirects can be used to mitigate this, e.g.

  http://example.com/ -> http://192.0.2.1.example.com/  [301 Permanent]

It is true that the browser can remember this redirect (though many
fail to) , however it does apply to the entire hierarchy.
http://example.com/foo/ must go through the entire dance again :)

Another point of consideration is that many website maintainers simply
do not like the appearance of  "192.0.2.1.example.com" in URLS - there
may also be anonymity concerns.

-- 
Colm

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 1 Jun 2010 14:41:00 -0700

> But, the reality is that initial several RTTs are acceptable.

They are not, because if they were, people would be using HTTP redirects if they were.  They aren't.

An HTTP redirect, even excluding the DNS transfer, is uncacheable, uncached, and significantly slower than the DNS lookup, as even with the stupid-short TTLs of the terminal names (eg, any-fp-wa1.b.yahoo.com), thats only one or two RTT for the name lookup, compared with 3 RTTs for a 302 redirect.

> Try it to see the reality of HTTP.

These are NOT externally visible URLs to the title bar, AND are always consistent.

You don't see a234.d.yimg.com spewed throughout the page, you just see d.yimg.com

You do NOT see the top URL as a234.www.yahoo.com, you see www.yahoo.com.

For latency, you need to redirect the top URL on the page.

And d.yimg.com does NOT do 302 redirects to a125.d.yimg.com either for the actual fetches, rather, d.yimg.com IS using the same DNS games as well!

The reality of HTTP is that 302 redirects are a far inferior solution to minimizing latency compared with the stupid DNS tricks.

> And you still don't know the initial RTTs are not a problem.

Some simple math:

DNS "stupid trick" has NO substanial effect on the DNS lookup latency, but can reduce the RTT to the server by 100ms for the simple case of "US west coast" vs "US east coast" (beyond that starts to be more marginal, but at least getting the right part of the right continent is a huge win right there!).

If you did that in HTTP redirection, you'd get 3-4 RTTs of latency piling on, delaying page load by 300-400ms.  AND the HTTP server, without DNS tricks, is almost certainly guaranteed to be suboptimal for a huge fraction of the users (after all, in the US, you're pretty much either on the east coast or the west coast, so 1/2 the users are 100ms away).

This counts!

In summary:

Stupid DNS trick present saving 100ms, 10 RTTs of total latency on page load:  Net savings: 1s just to start with.  Nice.

HTTP redirection "instead":  Additional 3 RTTs, to a 100ms latency server for at least 1/2 the userbase, cuts the net savings by 300ms.  Owch.

So yes, the initial RTTs are a problem, and HTTP redirection is NOT the solution.  

"Proper" authority behavior is.  And thats why people do the stupid DNS games that a vocal portion of this group decry.  They work, and the savings for just the simple case (get the sub-continent right) is huge right there.

---

From: Alex Bligh <[email-masked]>
Date: Tue, 01 Jun 2010 18:13:36 -0400

> may also be anonymity concerns.

plus https certification issues.

-- 
Alex Bligh

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 02 Jun 2010 13:49:21 +0900

Nicholas Weaver wrote:

> They are not, because if they were, people would be using
> HTTP redirects if they were.  They aren't.

>> If initial RTTs were the problem, never use DNS for extra several

As you can see, such URLs are not used very often.

> An HTTP redirect, even excluding the DNS transfer, is uncacheable,
> uncached,

Cache?

As cache is not helpful to reduce latency for the initial query,
we have agreed that initial RTTs are not a problem.

Or, are you just asking for something impossible?

> These are NOT externally visible URLs to the title bar,

They are visible at the status bar, which is why I noticed "yimg".

> The reality of HTTP is that 302 redirects are a far inferior
> solution to minimizing latency compared with the stupid DNS tricks.

Though I have been considering 300, feel free to use 302. Or, you
may modify HTTP. They are minor implementation issues of HTTP.

Anyway, initial RTTs are not a problem, especially because use of
open resolvers increases RTTs.

Shorter RTTs may be useful for large contents, because shorter RTTs
often mean less backbone load.

Thus, HTTP redirect just works.

					Masataka Ohta

---

From: Brian Dickson <[email-masked]>
Date: Wed, 2 Jun 2010 13:49:07 -0300

Replying to myself to get things back on subject...

To summarize the idea (in a top-reply):
Have the "big cache" (resolvers) actually be authoritative for the CDN
"trick" zones, by way of being slaves.

When a stub is pointed at a nameserver that happens to be authoritative, it
drops RTT to the absolute minimum, and number of queries needed to 1.

And, having the "tricks" done there, means the right information (client IP)
is available implicitly, to always give the "best" answer.

It means not having to add to the DNS protocol for the tricks, but it does
put more pressure on the provisioning systems, particularly inter- (rather
than intra-) provider provisioning of slaves.

Brian

On Tue, Jun 1, 2010 at 9:15 AM, Brian Dickson <[email-masked]

---

From: Tony Finch <[email-masked]>
Date: Thu, 3 Jun 2010 15:51:15 +0100

> "trick" zones, by way of being slaves.

Your idea implies that you have to standardize the logic in the stunt
authoritative server so that the open resolver can do it on behalf of the
CDN. This isn't feasible because this logic is part of the CDN's secret
sauce, so they wouldn't want to offload it onto another company or ossify
it into a standard. Your idea would also require extensions to the master
file format and to AXFR and IXFR to encode complicated information about
the CDN's connectivity for use by the stunt DNS logic, which is a lot less
plausible than the EDNS option we have been discussing.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
LUNDY: EAST, VEERING SOUTH LATER, 3 OR 4. SLIGHT OR MODERATE. SHOWERS LATER.
MODERATE OR GOOD, OCCASIONALLY POOR.

---

From: Brian Dickson <[email-masked]>
Date: Thu, 3 Jun 2010 12:18:47 -0300

On Thu, Jun 3, 2010 at 11:51 AM, Tony Finch <[email-masked]> wrote:

> it into a standard.

I'd like to hear it from them, that this is the case, if it is indeed the
case.

AFAIK, the "secret sauce" is in the "how" of doing geo-IP, and the "how" of
transmogrifying web pages on their web servers.
The DNS side, to an outside observer, looks like DNS, because it has to. The
only differences are in the quantity of entries, and the rate at which they
are updated.

> Your idea would also require extensions to the master
Yes. However, it scales better, removes caching issues (TTLs, lack of
pre-fetch, cacheability, lack of "push"/notify), and is by its very nature
completely compatible with existing clients and resolvers (where existing
can include really old, obsolete systems).

I don't think it directly touches AXFR/IXFR.

It does, however, definitely require some means for automated provisioning
of zones, i.e. delegate, create, modify, delete, undelegate, etc. This could
be out-of-band, or more likely, via some kind of convention to some kind of
well-known child zones, along the lines of _srv "magic".

Brian

---

From: Tony Finch <[email-masked]>
Date: Thu, 3 Jun 2010 16:40:46 +0100

> only differences are in the quantity of entries, and the rate at which they

No, the key difference is that the DNS replies are generated dynamically
by the geo-IP logic.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
PLYMOUTH: EAST 4 OR 5, BECOMING VARIABLE 3 OR 4 LATER. SLIGHT OR MODERATE.
FAIR. MODERATE OR GOOD.

---

From: John Payne <[email-masked]>
Date: Wed, 16 Jun 2010 16:15:11 -0400

On Jun 3, 2010, at 11:18 AM, Brian Dickson wrote:

> I'd like to hear it from them, that this is the case, if it is indeed the case.

What answer to give to queries is part of the secret sauce.  In many cases, it's more than just latency, there's liveness, load, fiscal and political considerations as well.
There's no way CDNs are going to open everything to everyone to allow this.

> AFAIK, the "secret sauce" is in the "how" of doing geo-IP, and the "how" of transmogrifying web pages on their web servers.
> The DNS side, to an outside observer, looks like DNS, because it has to. The only differences are in the quantity of entries, and the rate at which they are updated.

Also, CDNs do more than HTTP, RTSP and friends. Broadly speaking,  if your app talks IP it's a candidate for taking advantage of CDN technology.

> Yes. However, it scales better, removes caching issues (TTLs, lack of pre-fetch, cacheability, lack of "push"/notify), and is by its very nature completely compatible with existing clients and resolvers (where existing can include really old, obsolete systems).
> 
> It does, however, definitely require some means for automated provisioning of zones, i.e. delegate, create, modify, delete, undelegate, etc. This could be out-of-band, or more likely, via some kind of convention to some kind of well-known child zones, along the lines of _srv "magic".
>  
> Tony.
> --