Subject: [DNSOP] CLIENT-SUBNET bis appetite?

From: Mukund Sivaraman <[email-masked]>
Date: Thu, 14 Dec 2017 23:09:13 +0530

Any appetite for it? Don't throw things at me.. I ask because the
current thing is slowly getting more widely deployed and there are
design issues that can do with a ECS2 that breaks from ECS1 protocol. I
ask because I'm once again having to deal with myriad implementation
cases and dislike it.

		Mukund

---

From: bert hubert <[email-masked]>
Date: Thu, 14 Dec 2017 19:00:58 +0100

> ask because I'm once again having to deal with myriad implementation

Could you elaborate what you dislike most? The biggest thing we are noticing
is that while it does great things to getting to a server the content
provider likes, it unavoidably drives doen cache hitrates a lot, introducing
a latency penalty.

The operators we see deploying ECS have tens of thousands of subnets
which all need to be mapped to only a few servers. But you still end up
with tens of thousands of cache entries and therefore tiny cache hitrates.

Such things could be addressed by answering with lists of subnet masks to
which this answer would also apply, but this makes little sense
operationally I think. 

Can you share your ideas for ECS2?

	Bert

---

From: Warren Kumari <[email-masked]>
Date: Thu, 14 Dec 2017 13:09:30 -0500

> ask because I'm once again having to deal with myriad implementation

<no hats, other than author on the original doc.>
As I'm sure you know / remember, getting the first version though was
quite a slog. I suspect that, seeing as there seems to be good support
for the initial version, getting a new one done will be much simpler,
but the obvious question is:
How would ECS2 differ, and how would it interact with ECS1 / backward
compatibility?

W

</no hats>
>                 Mukund
>
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Mukund Sivaraman <[email-masked]>
Date: Fri, 15 Dec 2017 22:20:08 +0530

On Thu, Dec 14, 2017 at 07:00:58PM +0100, bert hubert wrote:
> Could you elaborate what you dislike most?

It is too complicated to implement ECS correctly. There are a large
number of corner cases. The things that resolvers and authoritative
sides have to take care of are quite different. It is more complex
than anything else in DNS.

I think this should be built again from scratch.

> The biggest thing we are noticing is that while it does great things
> to getting to a server the content provider likes, it unavoidably
> drives doen cache hitrates a lot, introducing a latency penalty.
> 

Firstly, correct deaggregation is an important requirement of reducing
cache usage. With the current design of ECS protocol, it's very
important that correct disjoining of prefixes be done optimally to avoid
cache pollution, yet the draft does not specify a suitable algorithm for
it (we know how to do it, I think the draft should have stated it).

A /n address prefix as specified by ECS option is a perfect binary
tree of 1<<n addresses. To correctly deaggregate 0.0.0.0/0 (scope=0)
data from a longer prefix such as 10.0.0.0/24, this will result in all
these answers to be generated:

* 10.0.0.0/24 answer
* 10.0.0.0/23 exact match answer (scope > source)
* 10.0.1.0/23 answer
* 10.0.0.0/22 exact match answer (scope > source)
* 10.0.3.0/22 answer

and so on.. there are about 2n+1 answers necessary so that a 0.0.0.0/0
answer does not override a /n client from receiving its specific answer.

                  x
           x            x
         x   x        x   x
        x x x x      x x y y

If ECS option had more fields, we could have put the above pattern as
a difference of trees (with direction bit and height, e.g., "x"s in
the diagram above) and it would have reduced cache usage
considerably.

This can be generalized with more differences but anyway I think that
using QUERY for ECS is a badly done idea (not even mentioning privacy
loss).

As an example, RPZ does not rely on queries.. it transfers all prefixes
for matching in a zone so that the longest prefix match algorithm will
not suffer from a previously cached shorter prefix matching and
preventing future fetches.

Another related problem is this: We often want to match against a GeoIP
database (containing what may be changing but maintained prefixes) in
associating zone data with geographic/network-topo clients. We want to
say "serve this answer for country X or city Y or ASNNNN" and we don't
care about managing the actual prefixes.

GeoIP is a custom database format, where I can match against GeoIP, but
I can't easily deaggregate all its prefixes for an ECS zone to be cached
properly.

I feel it would have been better to say to a downstream resolver "This
answer is for country X", or "A is the answer for country X, B is the
answer for country Y, the rest use answer C".

The design of ECS needs to be reconsidered. I'd prefer something like a
zone format for it, than using QUERY. QUERY cannot give complete
information about all prefixes and there is a possibility of incorrect
caching, and a very high probability of redundant cache pollution.

>From a resolver's point of view, a non-ECS answer (no client-subnet
option) is different from an ECS answer with scope=0 which is different
from an ECS answer with source=0, whereas all these may be the same from
an authority's point of view. They all need to be cached differently
(from an intermediate resolver's view).

This thing of scope > source meaning for-exact-match-only is weird as
hell when implementing longest prefix matching. It is not convenient
to use an off-the-shelf radix tree.

ECS relies on the option always being returned for any kind of
answers, as some resolvers use that as an indicator of ECS support
(and stop using ECS if it ever stops). But ECS does not apply to
several kinds of answers (e.g., anything but NOERROR, esp. NXDOMAIN
and NODATA have to be consistent across all prefixes.) It doesn't
apply to SOA, DNSKEY, NS in answer section, referrals, etc. Yet,
many of these need to answer with SCOPE=0.

An ACL config option about whether the NS supports ECS or not (to
return the option or not) is different from a config option whether
the NS passes through ECS or not: the latter would always pass through
SOURCE=0 but return REFUSED for any ECS queries that didn't match the
ACL; where as the former would return non-ECS reply for any ECS
queries that didn't match the ACL).

Transitivity of the option has corner cases.

I don't have to point out how easy it is for a erroneous /16 to
prevent queries to /24 answers shadowed by the /16.

Some cache cases: Obviously an ECS cache is different from a
zone.. it's not from a single zone, it is not an atomic collection of
a single version of zone and ever changing. If there's a /24 answer in
cache, and a newer query brings in a /16 answer that shadows it,
should the resolver assume that the /16 has precedence because it's
newer (hence the /24 should no longer exist) or do a
longest-prefix-match against the older /24? What if the /16 then
expires and the /24 hasn't expired? An NXDOMAIN answer should expire
any previously cached prefix-specific cache entries for that name. A
NODATA answer should expire any previously cached prefix-specific
cache entries for that type.  Non-ECS data is different from SCOPE=0
data. There are questions about trust ranking with usage of ECS data.

These are just some topics that I can quickly think of. There are many
other issues we faced and discussed during resolver ECS development.

The draft leaves many things unspecified, such as more clarity in DNSSEC
and handling of negative answers. Many issues were fixed during the
draft phase, but I feel it was insufficient.

> Can you share your ideas for ECS2?

There are many quirks in ECS. I don't want to propose specific ideas
now, except that we should gather requirements and start from
scratch. We have to reduce complexity of the protocol on both auth and
caching resolver sides. I think it should be designed again from
requirements without being a tweak of ECS1. The current protocol
complicates DNS implementation significantly.

		Mukund

---

From: Warren Kumari <[email-masked]>
Date: Fri, 15 Dec 2017 14:33:27 -0500

> answer for country Y, the rest use answer C".

<No hats. other than working at a place that uses ECS to provide
optimized answers>
Please see the thread
https://www.ietf.org/mail-archive/web/dnsop/current/msg19614.html

There is no *best* answer for a country, nor a city or even a postal address.

There is instead a best answer for a specific IP address / subnet,
which depends upon the ISP, the peering connections with that ISP, the
utilization of the peering links with that ISP, the utilization of the
datacenter, and then much lower down, where in the physical topology
that subnet resides.

I'm within spitting distance of Ashburn Equinix (well, I cannot quite
spit there, but I could easily walk there, it's < 5 miles), but my ISP
is Comcast. This means that, instead of hitting anything in IAD
(Ashburn), I (currently) instead get sent to MRN (Lenoir, North
Carolina), which is 400 miles away. My packets also happen to go
through 111 8th Ave, NY - even though there is stuff in NY, the MRN
location is better / faster.

If I happened to have chosen a different ISP, Verizon for instance, my
optimal answers would be very different.
And no, the "obvious" answer of "just use the AS number then" fails
equally badly - my latency to 8.8.8.8 (to chose something at random)
is ~12ms. If I ware a Comcast customer in California and the same
location were handed out, the latency would be ~80ms.

Note that I'm in a place which is well connected - in many places, the
"less optimal" answer is much much less optimal...

> caching, and a very high probability of redundant cache pollution.

Sure, happy to reconsider the design, but it is important to know more
about what the constraints are, and how specific and dynamic the
answers are.

For exmaple, Akamai says they are in 1,600 networks
(https://www.akamai.com/uk/en/about/facts-figures.jsp),  "The
Cloudflare Global Anycast Network"
(https://www.cloudflare.com/network/) is is powered by 118 data
centers around the world. "

Google Cloud Platform has added new regions in São Paulo and Mumbai.
GCP has 13 regions, 39 zones, over 100 points of presence, and a
well-provisioned global network with 100,000s of miles of fiber optic
cable." (embarrassingly I couldn't easily find a public page with more
detail)

Dyn has a map here: https://dyn.com/dns/network-map/

> scratch.

Yes, much of my soapbox rant was about just this -- understanding the
requirements is important - the reason that CDNs provide different
answers based upon the IP address it is a proxy for latency /
performance.
I'm sure we got many things wrong in ECS, but a redesign needs to be
informed by the use case and requirements.

(This mail not meant to sound as grumpy as it turned out :-) )

W

> We have to reduce complexity of the protocol on both auth and

---

From: Lanlan Pan <[email-masked]>
Date: Sat, 16 Dec 2017 11:11:56 +0000

Hi Warren,

Thank you for the mention, :-)

We all know that, because of network topology, client subnet is the *best*
indicator for CDN traffic management, expecially for Akamai, Google, ...
Totally agree with you:  There is no *best* answer for a country, nor a
city or even a postal address.

My thought is :
For GeoIP-enabled Authoritative Servers, they offen map Client Subnet into
EIL <COUNTRY, AREA, ISP>, then return tailored response only based on
<COUNTRY, AREA, ISP>.
There is *sufficient* answer for <COUNTRY, AREA, ISP>, on a GeoIP-enabled
Authoritative Server‘s *own view*.
EIL can be a trade-off choice for GeoIP-enabled Authoritative Server, and
optimize Recursive Resolver's cache cost.

Latest EIL document is at:
https://github.com/abbypan/dns_test_eil/blob/master/ietf_draft/draft.txt
There are many compatibility, cost vs benefit, operational concerns on EIL
from WG experts now, I am *still revising* it, welcome discussions and
issues.

Example:
xxx.com's Authoritative Server deploy GeoDNS proactively.
=> xxx.com's Authoritative Server itself choose accept *<COUNTRY, AREA,
ISP>* level traffic management.
=> for some operation problem/privacy concerns (maybe like Mukund mentioned
above),  we can use EIL to mitigrate.

ECS: I am network topology close to <111.201.133.0/24>, which is the best
answer on network topology now ?
GeoIP-enabled Authoritative Servers map <111.201.133.0/24> into <China,
Beijing, Unicom ISP>, then find the tailored answer.
Recursive Resolver cache the response with <111.201.133.0/24>, or some
shorter prefix.

EIL: I am network topology close to <China, Beijing, Unicom ISP>, which is
the best answer on network topology now ?
Recursive Resolver cache the response with <China, Beijing, Unicom ISP>,
which can cover many *network topology closed* client subnets.

Warren Kumari <[email-masked]>于2017年12月16日周六 上午3:34写道：

> is Comcast. This
> <https://maps.google.com/?q=s+Comcast.+This&entry=gmail&source=g> means
> that, instead of hitting anything in IAD
-- 
致礼  Best Regards

潘蓝兰  Pan Lanlan