EDNS Masters Thesis

THE STATE OF ADOPTION OF DNS ECS EXTENSION
ON THE INTERNET
By
FAJAR UJIAN SUDRAJAT
Submitted in partial fulfillment of the requirements for the degree of
Master of Science
Department of Electrical Engineering and Computer Science
CASE WESTERN RESERVE UNIVERSITY
May, 2017
CASE WESTERN RESERVE UNIVERSITY
SCHOOL OF GRADUATE STUDIES
We hereby approve the thesis of
FAJAR UJIAN SUDRAJAT
candidate for the MASTER OF SCIENCE degree*.
Committee Chair: Michael Rabinovich
Committee Member: Vincenzo Liberatore
Committee Member: Andy Podgurski
Date: 01 / 26 / 2017
*We also certify that written approval has been obtained

for any propriety material contained therein.
Contents
Contents ........................................................................................................................... i
List of Tables ................................................................................................................... iii
List of Figures ................................................................................................................. iv
1. Introduction ...............................................................................................................1
2. Related Work .............................................................................................................4
3. Background ................................................................................................................5
3.1. The Domain Name System ................................................................................ 5
3.2. Content Delivery Networks ............................................................................... 7
3.3. ECS EDNS Extension ........................................................................................ 10
3.4. Enabling ECS DNS Query ................................................................................. 11
4. Datasets...................................................................................................................13
4.1. Selected CDNs ................................................................................................ 13
4.2. CDN CNAMEs .................................................................................................. 15
4.3. List of Resolvers .............................................................................................. 16
5. Mapping Akamai Using ECS ......................................................................................17
5.1. ECS DNS Queries via Google Public DNS .......................................................... 17
5.2. DNS Queries to Our Own ADNS via Google ..................................................... 22
5.3. ECS DNS Queries Using Our Own Resolver ...................................................... 25
6. Exploring ECS Adopters ............................................................................................27
6.1. Detecting ECS Adopters .................................................................................. 27
6.2. ECS Adoption by CDNs .................................................................................... 28
i
6.3. ECS Adoption by Authoritative DNS Servers .................................................... 32
6.4. ECS Adoption by Resolvers .............................................................................. 37
6.4.1. Resolvers That Initialize the Option .................................................... 37
6.4.2. Resolvers Forwarding ECS Option ....................................................... 39
6.4.3. Can ECS-Compliant Resolvers Break a DNS Query? ............................. 41
7. Exploring the Use of CDN Federations and Multi-CDNs ............................................46
7.1. Multi-CDN ....................................................................................................... 46
7.2. CDN Federation .............................................................................................. 50
7.3. CDN Federation Footprints ............................................................................. 54
7.4. Multi-CDN Footprints ...................................................................................... 56
8. Conclusion ...............................................................................................................60
9. Bibliography .............................................................................................................62
ii
List of Tables
Table 1: List of CDNs and their serving domains ........................................................ 15
Table 2: Akamai's CNAMEs and their IP addresses .................................................... 18
Table 3: Itunes.apple.com.edgekey.net.’s resolved IP addresses .............................. 20
Table 4: Queries sent through China-based resolver that returned DNS responses with
non-zero ECS prefixes. ................................................................................. 21
Table 5: An example of Akamai's CNAME and their IP addresses .............................. 25
Table 6: Trace of interaction on queries to Akamai with and without ECS. ................ 27
Table 7: CDNs ECS adoption ...................................................................................... 29
Table 8: KeyCDN direct client based on the subnet client. ........................................ 31
Table 9: CDN77 returned IP addresses correlate with the client subnet. ................... 32
Table 10: The client subnets and the returned IP addresses on queries to 8.xyz ......... 35
Table 11: List of the ECS-enabled resolvers’ organizations .......................................... 39
Table 12: ECS queries sent through the 83,523 resolver to our name server .............. 41
Table 13: Samples of hostnames detected employ multi-CDN .................................... 50
Table 14: The combination of CDNs employed in multi-CDNs – single vantage point .. 50
Table 15: Pairs of CDNs detected in the federation – single vantage point .................. 53
Table 16: Pairs of CDNs detected in the federation – multiple vantage points ............ 56
Table 17: The combination of CDNs in the multi-CDNs found – ......................................
multiple vantage points ............................................................................... 59
iii
List of Figures
Figure 1: The interaction of DNS resolution process ..................................................... 6
Figure 2: The system model of CDN ............................................................................. 8
Figure 3: An example of a poor DNS redirection......................................................... 10
Figure 4: ECS option structures .................................................................................. 11
Figure 5: An output example of dig with ECS patch .................................................... 12
Figure 6: ECS presence in queries' responses from 1 million name servers. ............... 33
Figure 7: ECS-adopters’ position on Alexa's ranking ................................................... 37
Figure 8: A sample of the EDNS0 section from the trace in a readable format............ 38
Figure 9: A name server that reject ECS queries and returns status error ................... 43
Figure 10: Queries time comparison between Azure, Akamai and CDN77 .................... 45
Figure 11: DNS resolution on multi-CDN implementation ............................................ 47
Figure 12: Example of multi-CDN DNS resolution ......................................................... 48
Figure 13: DNS resolution on CDN federation implementation .................................... 51
Figure 14: A CDN federation example .......................................................................... 52
Figure 15: CDN Federation detection from several sets of vantage points ................... 55
Figure 16: Multi-CDN detection from several set of vantage points ............................. 56
Figure 17: The number of CDNs present on CDN-directed hostname resolutions ......... 57
iv
The State of Adoption of DNS ECS Extension on the Internet
Abstract
by
Fajar Ujian Sudrajat
Collaboration between a group of DNS service providers and Content Delivery Networks
(CDNs) has led to an open IETF RFC — EDNS-Client-Subnet (ECS). ECS is an extension to
the DNS protocol that helps its adopters to better direct client requests for content to a
nearby server, thereby improving experiences for end users. In this thesis, we explore
the state of adoption of ECS on the Internet: how many CDNs already support ECS and
how widely ECS is adopted by the authoritative name servers and the resolvers. We also
explore a security implication of ECS, in that it can simplify scanning of the entire set of
servers used by a CDN. We show that Akamai, the largest CDN, has successfully
addressed this potential vulnerability while other CDNs remain explored. Finally, by
utilizing multiple vantage points, we are able to infer the extent to which individual
CDNs collaborate with each other to form so-called federated CDNs, and discover that
some content providers employ multiple CDNs to deliver their content.
v
1. Introduction
Content Delivery Networks (CDNs) have become an important infrastructure in today’s
Internet architecture. Parallel with the rise of video content, an increasing amount of
content providers use CDNs to improve performance and maintain reliability. This is
highlighted by the growing amount of inter-domain traffic that flows directly between
large content providers and CDNs [28]. Notably, one CDN – Akamai [1] – claims to serve
15-30% of all Web traffic [32].
CDNs replicate content across a set of edge servers around the world and direct
clients to a nearby replica to reduce access time. In order to map the client to the
closest server, CDNs typically rely on the Domain Name System (DNS). When a client
wants to access a CDN-hosted web object, they first need to resolve the hostname. The
client will sends the DNS request to the local DNS resolver, typically the client’s ISP. The
resolver then forwards the request to the CDN’s authoritative name server. The CDN
makes decisions on server selection based on the IP address of the local DNS resolver.
This approach builds on the assumption that the client is close to the local DNS resolver,
thus the location of the resolver could represent the location of the client.
However, several studies [23, 33] have shown that these assumptions on locality
do not always hold. In some cases, the clients can be far removed from their resolvers,
and this may worsen client experience. Notably, the use of third party resolver like
Google Public DNS [14] and OpenDNS [18], since they may be far from the clients, could
yield worse end-to-end web performance [23, 34].
1
The potential problem that arises from the use of DNS-based redirection
motivates the collaboration work between third party resolvers and CDNs to provide a
solution. Their efforts led to RFC 7871 [26]. The document leverages the general DNS
extension mechanism (EDNS0) to convey information about the client to an
authoritative name server. The mechanism, called EDNS Client Subnet (ECS) allows
recursive resolvers to forward details about the client network information when talking
to other name servers. The network information forwarded by a recursive resolver could
be used by CDNs or content providers to better redirect clients to the nearby server.
Several prior studies [34, 36] observed a better performance in the use of ECS.
While the use of ECS offers an improvement in performance, it also introduces
unintended negative consequences. Several studies [24, 37] showed that the extension
could be used to reveal the edge servers of CDNs and content providers. It is possible to
map the infrastructure of an ECS adopter by sending crafted packets with arbitrary
client subnet information from a single vantage point. ECS adopters should be aware of
this vulnerability and address the threat to avoid revealing their infrastructure to
competitors and to prevent creating target lists for attacks.
To realize its benefits, ECS must be adopted by both DNS resolvers and CDNs. In
this thesis, we explore the state of ECS adoption on the Internet, including adoption by
DNS resolvers, CDNs, and other authoritative DNS servers (i.e., name servers that
maintain and provide mappings between hostnames and IP addresses).
We use DNS scanning, utilizing a large number of ECS DNS queries and
determine the CDNs’ support for ECS using certain characteristics in their response to
2
our queries. We first scan the largest CDN, Akamai, and try to map their entire platform
using ECS DNS queries. In our attempts to uncover their infrastructure we found that
Akamai’s support for ECS is not visible to us, as their response to our ECS queries do not
indicate that. While we were unable to detect Akamai’s use of ECS, we could infer that
the majority of CDNs have adopted ECS. In fact, one can simply scan such CDNs and
uncover details of their platform. We also explored the ECS adoption by authoritative
name servers through scanning a large number of hostnames and applying the same
detection technique we used to detect CDNs. In addition to that, we examined which
resolvers forward ECS queries sent by a client, showing their support of ECS. Our
experiments show that some authoritative name servers do not support the extension
and only return answer if the extension excluded. We conducted an investigation to find
a resolver that breaks access when querying such name server.
Finally, triggered by our findings in our ECS experiment, we explored the practice
of combining several CDNs by content providers. In alignment with that, we also
explored the extent to which individual CDNs collaborate with each other to form co-
called federated CDNs.
This thesis is organized as follows. Chapter 2 surveys research related to our
work on ECS extension. In Chapter 3 we explain the technical background that is
necessary to understand the remainder of this thesis. Chapter 4 describes the datasets
used in our work. Our first work on ECS is presented in Chapter 5, where we will scan
Akamai in our attempt to uncover their serving servers. Further works involving ECS
scanning to explore the adoption of ECS by CDNs, website’s authoritative name servers
3
and resolvers are explained in Chapter 6. In Chapter 7 we present the extent of multi-
CDN and CDN federation implementations. We will summarize our work in Chapter 8.
2. Related Work
One of our objectives is to map Akamai’s entire set of edge servers using recently
proposed DNS extension, ECS. Prior research has presented several approaches to map
Akamai’s infrastructure but none of them made use of ECS. Huang et al. [29] were able
to map ~27,000 Akamai’s edge servers including ~6,000 DNS servers using queries from
a quarter million public DNS servers. Triukose et al. [38], in part of their work, utilize a
distributed measurement platform, DipZoom [11], to discover Akamai’s edge servers.
In relation to our experiments on ECS, there are several works that conducted
experiments on ECS. Otto et al. [34] presented the first study on the efficiency of ECS.
They observed that ECS provides a significant end-to-end performance improvement
over public DNS resolution without ECS. Sanchez et al. [36] presented Dasu, a
measurement experimentation platform for Internet’s edge. In their work, they
conducted HTTP requests to a set of CDN’s edge servers that were resolved using DNS
queries with ECS and without ECS. They found that the HTTP performance to the edge
servers resolved with the use of ECS is better than without ECS. Chen et al. [25]
confirmed the performance benefit of ECS during Akamai’s roll-out of ECS in the first
half of 2014. They showed an eight-fold decrease in mean mapping distance and a two-
fold decrease in RTT and content download time.
4
Several other pieces of work exploit the vulnerability introduced by ECS. Streibelt
et al. [37] showed how simple it is to expose the infrastructure of ECS adopters and
track their expansions. Calder et al. [24] presented an investigation on the growth of
Google’s global footprint over time. With their use of ECS, they uncover 15-20% more of
Google’s infrastructure from a single vantage point than by using 200,000 open
resolvers. Kintis et al. [31] discussed ways that ECS can be used to help augment
surveillance, and enable extremely targeted cache-poisoning attacks against DNS.
3. Background
3.1. The Domain Name System
The Domain Name System (DNS) is a distributed database implemented in a hierarchy of
DNS servers containing various types of data, including host names and domain names.
At the very top of the hierarchy is the root domain called dot (“.”). The root domain of
the DNS is administered on a collection of root servers. The root servers delegate
responsibility for specific domains such as .edu, .org, .com to the name servers in the
next level in the hierarchy called top-level domain (TLD). Each TLD, delegates the
responsibility to its children, which may in turn, also delegate the responsibility to their
children. Ultimately, each website manages their own domain on their own name server
called authoritative DNS server (ADNS).
The steps required to resolve a hostname to an IP address are shown in Figure 1.
Suppose the client needs the IP address of the hostname: example.com. In step 1, the
5
client queries for the IP address of example.com to the local DNS server (LDNS), also
known as the recursive DNS server. The local DNS server forwards the DNS query
message to one of the root DNS servers as we see in step 2. The root DNS server finds
the .com suffix and delegates the question to TLD servers responsible for .com by
Figure 1: The interaction of DNS resolution process
sending a reply to the local DNS servers with a list of IP addresses of .com TLD servers,
seen in step 3. Step 4 shows the local DNS server resending the query message to one of
the .com TLD servers. The TLD server finds the example.com suffix and in step 5, returns
with the IP address of the authoritative name server for example.com. In steps 6 and 7,
6
the local DNS server finally obtains the IP address of example.com provided by the
domain’s authoritative name server. The local DNS server then returns to the client with
the IP address in step 8.
3.2. Content Delivery Networks
Video streaming and multimedia content are significantly growing in the present
Internet. Serving this content to customers across the entire globe from a single location
could lead to a bottleneck in the server outgoing network connection and also links
between regions on a continent.
A Content Delivery Network (CDN) offers solutions to the problem by providing
the capacity and availability to deliver content using a globally distributed network of
servers in multiple data centers. CDNs generally push the server delivering the content
closer to the client. Content Providers (CP) such as media companies only need to
provide CDNs with their source Web server and the CDN distributes the content to the
clients through their distributed servers.
Generally, CDNs incorporates a large number of edge servers and DNS servers.
Edge servers are placed at multiple geographical locations and store a cached version of
content provided by content providers. DNS servers direct clients to the nearest edge
server that serves the requested content.
Figure 2 demonstrates how a CDN works. In step 1, when a client wants to access
content hosted by the content provider, they will send a DNS query to a local DNS server
to get the IP address of the serving server. We previously explained the resolution
7
process to obtain the IP address of a hostname. The IP address typically provided by the
domain’s authoritative name server. In a CDN system model, the domain’s authoritative
name server further delegates the question to CDN’s name server, typically by
returning, instead of the IP address sought, a special record (called a CNAME record)
Figure 2: The system model of CDN
that contains a “canonical” hostname (a synonym) for the name from the query. The
canonical name belongs to CDN’s domain. Thus, in Step 2, the local DNS server resends
the query, now for the canonical name, to CDN’s authoritative name server. In step 3,
CDN’s authoritative name server returns the IP address of one of CDN’s edge server to
local DNS server, ideally the closest to the client. The local DNS server then returns the
IP address to the client, as we see in step 4. Using the IP address of the edge server, in
8
steps 5 and 6, the client sends a request to the edge server and obtains the content
from the edge server if the edge server already cached the content. If the content not
cached, the edge server will fetch the content from the content provider and cache the
content for future requests.
Large CDNs and content providers rely on DNS to direct clients to a nearby
server. The general assumption is that clients are located close to the local DNS server
they use. Using the local DNS server address obtained from the source-IP address in the
DNS message request, a CDN’s authoritative name server may geo-locate or use certain
calculations to select a server that is close to the client. However, the assumption
proven does not always hold [23, 33]. In some cases, user may perceive a degraded
experience.
Figure 3 demonstrates that the use of a local DNS server’s IP address to
determine the location of a client does not always direct the client to the closest edge
server. As we see, the location of edge server 1 is closest to the local DNS server,
however, edge server 2 is the closest to the client. Instead of directing the client to edge
server 2, CDN’s authoritative name server directs the client to edge server 1, which is
assumed to be close to the client. The use of public DNS resolver like Google or
OpenDNS, since they may far from the clients, could yield even worse performance.
9
Figure 3: An example of a poor DNS redirection
3.3. ECS EDNS Extension
A collaboration effort between recursive DNS services and CDNs to find a solution for
the issue of miss-locating the end-user position led to RFC 7871. The document defines
an EDNS-Client-Subnet DNS extension (ECS). Traditionally, a client’s IP address is not
included in the DNS queries. ECS enables DNS queries to include the client’s IP address,
to be forwarded by all ECS enabled resolvers to authoritative name servers. The ECS
payload, as shown in Figure 4, consist of:
• OPTION-CODE, for ECS is 8 (0x00 0x08)
• OPTION-LENGTH, contains the length of payload
• FAMILY, the address family used for the client IP, 1 for IPv4 and 2 for IPv6.
10
• SOURCE PREFIX-LENGTH, representing the subnet to be used for the lookup.
• SCOPE PREFIX-LENGTH, representing the subnet the answer covers.
• ADDRESS, the IP address of the client. For privacy purposes, it is recommended
to use a prefix less than the full IP address.
Figure 4: ECS option structures
3.4. Enabling ECS DNS Query
In our work, we need a client that is able to send a DNS query with the ECS extension
enabled. We evaluated several options regarding how we set up our probe to send ECS
queries. Initially, we used a patched Bind’s dig tool [5] to test sending a DNS query with
the ECS extension enabled. The ECS patch to support ECS available at [6]. Figure 5 shows
the example of the ECS-enabled dig’s output. We see there is a CLIENT-SUBNET section
that shows the client subnet, source prefix-length and scope prefix length.
Dig also has an option to perform DNS resolution without using a recursive
resolver. By simply adding the +trace option, dig will send the query recursively in a
11
Figure 5: An output example of dig with ECS patch
typical DNS resolution scenario. Sending a large number of queries using dig requires
some script to continually execute the dig command and capture the output strings. The
output strings then need to be parsed accurately to make it easier to read. With that
complexity, instead of using dig, we use a simpler approach to make our probe. We
implemented our probe using a script written in Python, clientsubnetoption.py [9],
which enables client subnet in DNS queries. The script clientsubnetoption.py is a class
from a DNS toolkit for Python, dnspython [12]. Using functions provided in dnspython,
we can easily extract data from every section of DNS messages, simpler than parsing out
dig’s output string.
We want to mention that we also tried the only available ECS-enabled recursor
we found, PowerDNS Recursor [20]. Unfortunately, PowerDNS Recursor does not
forward the ECS data from a client. PowerDNS Recursor discards ECS data from a client
and composes new ECS data using the local machine network information. We got in
12
contact with one the developers of PowerDNS Recursor who informed us that the
exclusion of the ECS data from clients is to avoid a client hiding from an authoritative
name server by using a fake client subnet.
4. Datasets
In this chapter we present datasets that were used in our experiments and how these
are obtained.
4.1. Selected CDNs
We need the list of all known CDNs and the domain names they use to deliver contents.
The majority of CDNs on our list were selected from CDNPlanet [8]. Obtaining CDN’s
domains was not a straightforward process. None of the CDN official websites nor third
party websites publish a complete list of CDN domains. We pulled most of the
information from CDNs’ website FAQ pages where they have a quick review on how to
use the CDN. Usually they mention one or two CNAME examples, showing the domain
used by the CDN.
In addition, we resolved a large number of website hostnames, see Section 4.2,
and looked into frequent second-level domains in the CNAMEs that were returned by
the website hostnames. If the second-level domain in whois database lookup shows it
belonged to a CDN in our list of CDNs, we add the domain into the CDN domains list.
Finally, specific for Akamai, we obtained a number of their domains from the Wikipedia
13
page about Akamai [3]. In the end, we have in total 32 CDNs for which we know their
serving domains as shown in Table 1.
No CDN Domains
1 Akamai akamai.net, akamaiedge.net, akamaized.net, akamaihd.net,
edgesuite.net, edgekey.net, srip.net,
akamaitechnologies.com, akamaitechnologies.fr,
akamaitech.net, akadns.net, akam.net, akamaistream.net
2 Cloudfront cloudfront.net, amazonaws.net, amazonaws.com
3 Azure azureedge.net, msecnd.net, msedge.net
4 Edgecast edgecastcdn.net, alphacdn.net, betacdn.net, gammacdn.net,
deltacdn.net, epsiloncdn.net, zetacdn.net, etacdn.net,
thetacdn.net, iotacdn.net, kappacdn.net, lambdacdn.net,
mucdn.net, nucdn.net, xicdn.net, omicroncdn.net,
rhocdn.net, sigmacdn.net, taucdn.net, upsiloncdn.net,
phicdn.net, chicdn.net, psicdn.net, omegacdn.net,
systemcdn.net, v0cdn.net, v1cdn.net, v2cdn.net, v3cdn.net,
v4cdn.net, v5cdn.net, transactcdn.com
5 Cachefly cachefly.net
6 CDNetworks cdnetworks.net, panthercdn.net, gccdn.net, cdnga.net,
cdngc.net
7 Chinacache chinacache.net, ccgslb.net, ccgslb.com
8 Cloudflare cdn.cloudflare.net, yunjiasu-cdn.net, cloudflare.com
9 Incapsula incapdns.net
10 Instartlogic insnw.net, inscname.net
11 Internap internapcdn.net
12 Limelight llnwd.net
13 MaxCDN netdna-cdn.com
14 Mirror Image mirror-image.net
15 Level-3 footprint.net
16 Onapp worldcdn.net
17 OVH ovh.net
18 Yotta yottaa.net
19 Streamzilla xlcdn.com
20 CDN77 cdn77.net, cdn77.com
21 CDNify cdnify.io
22 CDNSun cdnsun.net, cdnsun.com
23 CDNVideo cdnvideo.ru, cdnvideo.com
24 ChinanetCenter wscloudcdn.com, ourwebpic.com, speedcdns.com,
ourwebcdn.com, cdn20.com,wscdns.com
25 Fastly fastly.net, fastlylb.net
26 Highwind hwcdn.net
14
27 Isprime isprimecdn.com
28 KeyCDN kxcdn.com
29 Leaseweb lswcdn.net
30 Ngenix ngenix.net
31 Rackspace rackcdn.com, raxcdn.net
32 Xcdn expresscdn.net
Table 1: List of CDNs and their serving domains
4.2. CDN CNAMEs
To collect CDN CNAMEs we need to know the hostnames from the CDN customer
websites that are outsourced to a CDN. The first step toward obtaining such hostnames
might be to extract them from URLs found in CDN customer websites. However, most
CDNs publish only a few of their top customers, thus limiting the number of hostnames
that we could gather. We therefore collect CDN CNAMEs by resolving a large number of
hostnames despite not knowing whether the hostnames are outsourced to a CDN or
not.
We first obtained a large number of hostnames from a publicly available source,
the Alexa list of top 1 million websites [4]. Since websites’ starting hostnames (such as
“cnn.com”) are less likely to be outsourced to a CDN, we need to get their subdomains
where the hostnames have a higher chance to be directed to a CDN. Using a simple
Python script, we probed the top page of the websites and pulled all of the URLs we
found on it. We collected a total of 5,611,473 URLs and from these URLs we extracted
2,017,367 hostnames. In the next step, we resolved all of the hostnames and captured
all of the CNAMEs present in the queries. We selected hostnames that were directed to
15
a CNAME belonging to the CDNs in our list. In total we pulled 429,563 hostnames
outsourced to CDN.
We observed that many of these hostnames belonging to the same second-level
domain, which means they are used by the same website. For example, hostnames
admission.careerage.com and forum.careerage.com, both are part of a URLs in the
Career Age website. We believe the same edge servers serve different hostnames that
belong to the same second-level domain. Triukose et al [38] showed that the use of
additional CNAMEs from the same customer do not contribute much to server
discovery; meaning that they would all be directed to largely the same edge servers.
Based on this, we reduce the number of hostnames outsourced to CDN by
picking only one, arbitrarily chosen, hostname representation for each domain. From
our example above, we may pick admission.careerage.com as a representation of the
careerage.com domain and remove forum.careerage.com from the list. From the
reduced CDN outsourced hostnames, we have in total of 89,135 CDN’s CNAMEs with
17,618 of them belonging to Akamai.
4.3. List of Resolvers
A parallel study by a colleague in our lab, Rami Al-Dalky, involved scanning the entire
Internet for open recursive resolvers by sending DNS queries via all possible IP
addresses to resolve domain names registered in his authoritative name server. Using
traces captured at the authoritative name server, we could compile a list of resolvers
that could be discovered through this technique, and which of these resolvers actually
16
send DNS queries with the ECS option included. This work provides us with a list of
resolvers comprising 83,523 IP addresses.
5. Mapping Akamai Using ECS
[24, 37] shows that a single vantage point combined with publicly available information
is sufficient to uncover the global footprint of an ECS adopter and track its expansion. To
the best of our knowledge, there is no previous work that maps the edge servers of the
biggest CDN, Akamai using ECS. Since a previous report says that Akamai already
supports ECS [2], we want to use the capability of ECS to emulate queries from multiple
vantage points around the world to trick Akamai into sending responses with IP
addresses that correlate with the client subnet we put in the ECS option. By using all
advertised IPv4 prefixes we supposedly could enumerate the IP addresses of all
Akamai’s edge servers.
5.1. ECS DNS Queries via Google Public DNS
We designed our experiment to emulate ECS DNS queries from multiple vantage points
using arbitrary ECS client subnet information. With regards to the queries, we used
Akamai’s CNAMEs as query strings and various /24 prefixes as the ECS client subnets.
We used prefixes from a list of an advertised IPv4 prefixes obtained from
bgp.potaroo.net [7].
17
In our initial experiment, we sent ECS queries from our lab machine to Google
Public DNS using 15 Akamai’s CNAMEs as the query strings. We paired 10 different client
subnets that are widely spread geographically with each of the query strings. We were
assuming that the return IP address from our queries would be located close to the
client subnet we used in the query.
Akamai’s CNAME Resolved IP address IP location

san.revolveclothing.com.edgekey.net 104.70.58.68 Virginia, US
mycareer-akamai.deloitte.com.edgekey.net 104.69.221.160 Virginia, US
www.disney.es.edgesuite.net. 192.232.17.209, 192.232.17.201 Ohio, US
www.gwt-eu.edgekey.net. 104.70.74.130 Virginia, US
san.worldvision.org.edgekey.net. 104.70.61.77 Virginia, US
www.tripwire.com.edgekey.net. 104.70.75.78 Virginia, US
ipv6.cn.louisvuitton.com.edgesuite.net. 192.232.17.194, 192.232.17.178 Ohio, US
www.malaysiaairlines.com.edgesuite.net. 192.232.17.176, 192.232.17.177 Ohio, US
www.ikea.redirects.edgesuite.net. 192.232.17.201, 192.232.17.209 Ohio, US
wwwds.cisco.com.edgekey.net. 104.70.53.180 Virginia, US
itunes.apple.com.edgekey.net. 104.67.92.93, 104.70.59.193 MA, US
www.adobe.com.edgekey.net. 104.70.73.171 Virginia, US
static.ak.facebook.com.edgesuite.net. 192.232.17.203, 192.232.17.209 Ohio, US
gr.voanews.com.edgesuite.net. 192.232.17.193, 192.232.17.192 Ohio, US
www.defense.gov.edgesuite.net. 192.232.17.186, 192.232.17.179 Ohio, US
Table 2: Akamai's CNAMEs and their IP addresses
We geo-located the returned IP addresses using a GeoIP database from
MaxMind [17] and we found that the IP addresses of the edge servers, shown in Table 2,
are all located in the United States. 7 of 15 Akamai’s CNAMEs were directed to edge
servers located in Elyria, OH, roughly 28 miles from our lab location.
We observed that responses to our queries included the ECS option and had scope
prefix-length zero. According to RFC 7871, zero-scope means the client prefix in the ECS
DNS query is not used by the authoritative name server to generate the IP addresses in
18
the answer. Indeed, the edge servers’ IP addresses were not in correlation with the
client subnets we put in the ECS option.
We set up another set of ECS queries that use a large number of prefixes for the
client subnet information. We wanted to make sure the range of prefixes used in our
ECS queries was not the main factor for the unexpected ECS responses from Akamai as
an ECS adopter. We sent the ECS queries from our lab machine and from several
PlanetLab [21] nodes located in South Korea, Australia and Finland. From each location,
we sent a total of 40K queries with query string itunes.apple.com.edgekey.net. Each
query had a client subnet randomly picked from our list of prefixes. The result from this
set of ECS queries had similar characteristics as our initial ECS queries test. All the
responses included an ECS option and had scope prefix-length zero. The IP addresses in
the answers did not show the closeness to the client subnet in the ECS data. It seemed
that the IP addresses were given based on the network information of the queries’
origin.
Table 3 presents the result of our queries. The locations of Akamai’s edge servers
are relatively close to where the queries were sent. For example, queries sent from the
PlanetLab node in Helsinki, Finland received answers with IP addresses all located in
Amsterdam, the Netherlands. The example also shows more convincingly that the
answers were not derived from the ECS option in our queries. 4K queries with various
client subnet information in the ECS data that were sent from Helsinki, Finland only gave
us two Akamai’s IP addresses.
19
Akamai’s CNAME Queries origin Resolved IP IP location
address
itunes.apple.com. Ohio, US 104.104.246.247 New York, US
edgekey.net. 104.70.59.193 Massachusetts, US
104.87.140.245 Massachusetts, US
23.1.24.107 Washington, US
23.197.110.3 Chicago, US
23.197.24.161 Chicago, US
23.209.48.62 New York, US
Victoria, AUS 184.28.8.65 Sydney, AUS
184.84.61.151 Hongkong, HK
23.78.27.89 Buenos Aires, AR
Helsinki, FIN 23.14.4.61 Amsterdam, NED
104.76.38.94 Amsterdam, NED
Daejon, KR 184.25.22.13 Anyang, KR
2.17.49.37 Hongkong, HK
184.26.240.241 Tokyo, JPN
202.43.50.217 Seoul, KR
118.214.66.217 Seoul, KR
23.78.29.254 Texas, US
104.70.154.65 Tokyo, JPN
23.33.123.185 Seoul, KR
Table 3: Itunes.apple.com.edgekey.net.’s resolved IP addresses
The experiment showed that we could not map Akamai’s edge servers using ECS
queries sent via Google Public DNS. We needed to make sure that our use of Google
public DNS in probing Akamai was not the reason we could not to get the result we
hoped for. We wanted to know how Google public DNS forwards an ECS query to an
authoritative name server. In section 5.2 we will present our experiment to address this
question.
As an alternative to using Google in our probing, we used the 83,523 resolvers
obtained by our lab colleague. First, we sent ECS queries through the resolvers to
20
resolve various Akamai CNAMEs. We looked for a resolver that would return our query
with ECS option enabled and scope prefix-length non-zero, and we found eight such
resolvers. These resolvers are all located in China and we noticed the returned IP
addresses are located in/or around China, as shown in Table 4. We inferred that the IP
addresses were given based on the resolver’s IP addresses, not the client subnet we put
in the queries. Regardless, we sent another set of queries through the resolvers. We
sent the additional queries considering the presence of the non-zero scope might lead
to a different result compared to the results from our use of Google. In our queries we
used three different and geographically dispersed client subnets to each of the
resolvers. We saw that for each of the three queries, the resolvers returned IP addresses
that have the same location, if not the same IP address. Apparently we also cannot use
these resolvers to scan Akamai’s edge servers.
Akamai’ CNAMEs Resolver’s IP Scope Returned IP IP address’

addresses addresses location
any.gmo-media.jp.edgekey.net. 124.251.21.95 24 184.27.168.90 Osaka, Japan
www.bfage.com.edgesuite.net. 111.13.113.8 24 23.2.16.25 Hongkong
www.jnj.com.edgekey.net. 111.13.113.9 24 184.84.59.125 Hongkong
lidl.pt.edgesuite.net. 42.236.41.25 24 184.50.87.64 Beijing, China
www.defenselink.mil.edgesuite. 111.206.209.21 24 125.56.201.137 Singapore
net.
s.atgstores.com.edgekey.net. 163.177.8.8 24 23.58.242.51 Kuala Lumpur,
Malaysia
www.fxnowcanada.ca.edgesuite. 120.209.132.157 24 23.2.16.67 Hongkong
net.
www.tuenlinea.com.edgesuite. 125.39.75.110 24 61.213.168.9 Osaka, Japan
net.
Table 4: Queries sent through China-based resolver that returned DNS responses
with non-zero ECS prefixes.
21
5.2. DNS Queries to Our Own ADNS via Google
The previous experiment raised a question on how Google Public DNS handles ECS DNS
queries. Does Google modify the ECS option of queries received, or do they just let it
pass through as is? To answer this question, we sent ECS DNS queries through Google
Public DNS to our own authoritative name servers and observed the forwarded queries.
We set up our authoritative name server in our lab machine. First, we deploy
BIND as our authoritative name server, however at that time BIND had yet to release a
version with ECS support. Since we wanted it to be able to handle ECS DNS queries, our
authoritative name server needed to be ECS ready. Not many authoritative name
servers already support ECS. We find only PowerDNS [19] and gdnsd [13] already
support the extension. We picked PowerDNS as our ECS authoritative name server since
PowerDNS could run using our existing BIND configuration.
We first ran our experiment with our authoritative name server set to be ECS-
disabled. We sent several ECS queries and observed that Google does not forward the
ECS option from our original request to our authoritative name server. According to the
ECS standard, a recursive resolver may or may not include the ECS option from the
incoming query. Passing the ECS data from the client is supportive of recursive resolver
being used as the target of the forwarding resolver. However, it is recommended to only
send the ECS option to authoritative name servers that already support ECS.
22
There are two ways for recursive resolvers to determine which authoritative
name servers already support ECS:
1. Using whitelist
Authoritative name server actively submits a request to recursive resolver, to put
their domain name on the recursive resolver whitelist to send the option to.
2. Using probes
Recursive resolver implements a detection mechanism by sending probes
periodically and remembers which authoritative name server did return ECS option.
At first, we thought that Google does not forward our ECS option because our
authoritative name server was never registered in Google Public DNS’s whitelist and we
would need to send a request Google to be included on their whitelist. However, we
learned that Google no longer uses a whitelist to decide whether to send the ECS
option: it stopped using the whitelist in June 2014 [15]. Google public DNS now
automatically detects authoritative name servers that support ECS options and sends
queries with ECS options to such name servers automatically.
Even though Google employs such mechanisms to avoid sending ECS queries to a
non ECS-compliant authoritative name server, using another set of queries, we observed
Google did not completely exclude ECS option in a query to such name servers. We sent
40K ECS queries to our name server, via Google, with a different client subnet among
the queries. Although our name server was not ECS enabled, Google sent a total of
1,473 queries with the ECS option included to our name server. We also found that the
23
ECS queries still have the client subnet from our original query. Google indeed forwards
the ECS data from the client’s query.
Examining Google’s queries from the above experiment, it appeared that Google
would occasionally forward the ECS option in its queries to probe the ADNS’s ECS
support, and when the ADNS response does not indicate such support, Google would
stop forwarding the ECS option for a while, until the next probe.
To be more certain, we also sent ECS queries to our own authoritative name
server set to ECS enabled. As expected, Google forwarded our ECS option to our
authoritative name server without any modification. With this result, we assumed
Google supposedly would forward our ECS option to Akamai’s authoritative name
server, however the outcome from the previous experiment indicates a different
behavior between ECS DNS queries to Akamai and ECS DNS queries to our authoritative
name server.
Instead of using client subnet from a client, Google may use the client’s IP
address on the outgoing query to Akamai. This is shown by our previous experiment
where the returned IP addresses are close to the client locations. The only publicly
available information supporting our suspicion is the presentation slides of Matt Jansen
in SANOG 24, Delhi, August 2nd 2014 [30]. One of the slides discusses the security
concerns of an attempt to scan/walk the mapping algorithm used by Akamai. One of the
points presented was to enforce replacement of ECS data by Google and OpenDNS
before sending the query to Akamai.
24
5.3. ECS DNS Queries Using Our Own Resolver
Since we were unable to uncover Akamai’s edge servers using queries through Google
Public DNS, we then tried to resolve the Akamai’s CNAMEs using our own resolver. We
sent DNS queries with the ECS option from a single machine in our lab to resolve 17,618
of Akamai’s CNAMEs with ECS information of five arbitrary client subnets that are
geographically dispersed, for each CNAME. We also re-sent the same queries without
the ECS option in order to see whether both types of queries give different answers.
Akamai’s CNAME Client-subnet IP addresses

www.seattletimes.com.edgesuite.net. 196.11.75.0 66.61.167.59, 66.61.167.67
(a) Queries with various client subnet returned same IP addresses

san.maccosmetics.com.edgekey.net. 104.37.58.0 23.196.120.126
(b) Few queries returned with different IP addresses

br.ask.com.edgesuite.net. 192.5.240.0 66.61.167.59, 66.61.167.25
aos.ask.com.edgesuite.net. 153.254.90.0 66.61.167.27, 66.61.167.67
www.citibank.com.ph.edgekey.net. 104.37.58.0 104.112.33.19
(c) Different CNAMEs directed to the same IP addresses
Table 5: An example of Akamai's CNAME and their IP addresses
25
We recorded that Akamai’s authoritative name server responded to all of our
ECS queries without the ECS option. For the majority of the CNAMEs, each directed to
the same set of IP addresses as shown in an example in Table 5 (a). Only 111 of the
17,618 CNAMEs have more than one IP addresses as can be seen in an example in Table
5 (b). We also saw that many of the CNAMES share the same IP addresses. Like the
example shown in Table 5 (c), there are a total of 5,480IP addresses linked to the 17,618
CNAMEs.
The non-present ECS option in the query’s responses indicates that Akamai’s
authoritative name server did not honor the ECS option from our queries. The returned
IP addresses also show that their assignment was not based on the client subnet in the
ECS option. A possible explanation would be that Akamai use a whitelist to limit the ECS
feature to only certain clients.
Even though we could not detect Akamai’s use of ECS from the ECS option and
the returned IP addresses, we observed that Akamai’s authoritative name servers do
treat ECS DNS queries differently from regular DNS queries. Table 6 shows that even
though the query’s answer is the same, there are differences in the DNS query’s trace of
interaction between queries with ECS and those without. In the example we see that
queries with ECS option have an extra round of interaction and a different hostname
compared to queries without ECS option.
1st round 2nd round 3rd round Answers

ECS gr.voanews.com. a1479.b.akamai. a1479.b.akamai.net.0.1. 66.61.167.32
edgesuite.net. net. cn.akamaitech.net. 66.61.167.50
26
No gr.voanews.com. a1479.b.akamai. - 66.61.167.32
ECS edgesuite.net. net. 66.61.167.50
Table 6: Trace of interaction on queries to Akamai with and without ECS.
Same as the result from our previous queries via Google public DNS, sending the
ECS DNS queries directly to Akamai’s authoritative name servers also showed us that we
could not exploit the ECS security vulnerability to map Akamai’s entire edge servers.
6. Exploring ECS Adopters
Moving on from our effort to map Akamai using ECS DNS queries, next we wanted to
explore the ECS adoption by other CDNs and the top websites’ authoritative name
servers. In this section, we are going to discuss how we detected ECS adopters and
which CDNs and hostnames are in our list of ECS adopters.
6.1. Detecting ECS Adopters
Since the RFC 7871 is not an Internet Standard Track specification, its adopters may
have a different implementation on a case-by-case basis. This was shown in our
previous experiments where Google treats ECS queries to Akamai and to our name
server differently. Examining the presence of the ECS option may not allow us to directly
find out if an authoritative name server is ECS enabled or not. Streibelt et al [37]
annotate an authoritative name server as ECS-enabled if the scope in the ECS option is
non-zero in one of the queries. Having non-zero scope however, does not convey
27
whether the client subnet was used in the assigning of the returned IP address – as
demonstrated by the 8 Chinese resolvers (Table 4).
Another thing to consider is that our experiments on Akamai show that even
though ECS option was not present in their answers, we detected different traces of
interactions involved in processing queries with and without ECS. We then decided to
determine that a name server is ECS-enabled by using the following heuristic:
1. If, for multiple queries, the ECS option with non-zero scope is present in the reply
and the IP address in the answer shows a correlation with the queries’ client subnet
in term of their distance, we annotate the authoritative name server as an ECS
adopter.
2. Even when the response to an ECS DNS query arrives without the ECS option, if the
trace of interaction between DNS queries with and without the ECS option shows a
difference in a considerable amount of queries, we also annotate the authoritative
name server as an ECS adopter.
6.2. ECS Adoption by CDNs
To detect CDN ECS adopters, we recursively sent ECS DNS queries from a single machine
to resolve 89,135 CDN’s CNAMEs. For each CNAME, we sent five queries with a different
client subnet in the ECS option plus one query without ECS option for trace of
interaction comparison purposes. Table 7 shows our compilation of CDNs’ ECS adoption.
We marked Instartlogic in our table because from our queries to 348 Instartlogic’s
CNAMEs, we only found one CNAME where the responses included the ECS option.
28
No CDN ECS Scope- Answer Different ECS-
option prefix correlate with trace of enabled
length client subnet interaction
1 Akamai - - - √ √
2 Cloudfront √ non-zero √ - √
3 Azure - - - - -
4 Edgecaset √ non-zero √ - √
5 Cachefly √ non-zero √ - √
6 CDNetworks √ non-zero √ - √
7 Chinacache √ non-zero - - -
8 Cloudflare √ non-zero - - -
9 Incapsula √ non-zero √ - √
10 Instartlogic √* non-zero - - -
11 Internap - - - - -
12 Limelight - - - - -
13 MaxCDN √ non-zero √ - √
14 Mirror Image √ - - - -
15 Level-3 √ zero - - -
16 Onapp √ non-zero √ - √
17 OVH - - - - -
18 Yotta - - - - -
19 Streamzilla √ zero - - -
20 CDN77 √ non-zero √ - √
21 CDNify √ non-zero √ - √
22 CDNSun √ non-zero √ - √
23 CDNVideo √ non-zero √ - √
24 ChinanetCenter √ non-zero √ √
25 Fastly √ zero - - -
26 Highwind - - - - -
27 Isprime - - - - -
28 KeyCDN √ zero √** - √
29 Leaseweb √ non-zero √ - √
30 Ngenix √ - - - -
31 Rackspace √ non-zero √ - √
32 Xcdn √ non-zero √ - √
Table 7: CDNs ECS adoption

* Instartlogic only returned one query with ECS option from 348 queries
** KeyCDN returned answers that correlate with the given client subnets even though
the scope in the returned ECS option is zero
29
We found interesting behavior on KeyCDN. Even though they returned a zero
scope on their responses, we observed that the client subnets in fact correlate to the IP
addresses in the answers. Table 8 presents samples of our queries to KeyCDN that show
a correlation between the client subnet and the returned IP addresses.
According to the ECS specification, scope prefix-length is used to indicate to
which prefix’s family the answer belongs. In caching the answer, a resolver must
examine the scope. If the scope is shorter than source prefix-length, the answer is valid
for all addresses that fall within the scope range. If the scope is longer than source prefix
length, the answer is valid only to addresses within the range of the source prefix
length. That being said, zero scope means the answer covers all network.
However, zero scope could also mean the option is supported but not actually
used for generating a response. The two meanings of the zero scope led to an
ambiguous interpretation in our observations, whether the answer is valid for all
addresses or the option was not used to assign the answer. In the KeyCDN case, since
the client subnets in fact correlate to the IP addresses in the answers, we strongly
believe it was the former rather than the latter and we annotated KeyCDN as an ECS
adopter. With this, we further considered a non-zero scope in our heuristic to detect
ECS adopters. We re-examined other CDNs, now considering the non-zero scope, and no
changes were found from our pervious examination.
The second point in our heuristic involved comparing traces of interactions
involved in processing queries with ECS and without. We only found that Akamai has
differences in the traces of interaction.
30
CNAMEs Client subnets Client subnet Returned IP IP addresses
location addresses location
178.49.132.0 Russia 194.63.141.18 Russia
192.5.240.0 Australia 168.1.24.74 Sydney,
Australia
ecoportal-
5.178.41.0 Italy 95.141.32.94 Marco, Italy
16d9.kxcdn.com.
153.254.90.0 Tokyo, Japan 161.202.72.175 Tokyo, Japan
104.37.58.0 Chandler, 208.77.22.107 Aurora,
United States United
States
153.254.90.0 Tokyo, Japan 161.202.72.175 Japan
217.135.102.0 United 188.227.185.218 United
Kingdom Kingdom
193.29.239.0 Germany 136.243.68.197 Germany
kingbuzz- 5.178.41.0 Italy 95.141.32.94 Italy
14b0.kxcdn.com 183.54.15.0 Guangzhou, 161.202.39.238 United
China States
Table 8: KeyCDN direct client based on the subnet client.
Half of the CDNs in our list have adopted ECS. In fact, their authoritative name
servers returned an ECS DNS query with an IP address that correlated with the client
subnet sent, thus making themselves open to easy scanning. For example, when we
scanned CDN77 using ECS DNS queries with only one query string and a random 100
different client subnets sent from a single machine, we obtained 21 locations of their
edge servers.
CDN77’s website officially stated that they have 32 datacenters around the
globe. What would it take to discover all their datacenters? Our next queries to CDN77
used selected client subnets that represented all regions (provinces/states) in the world
with a total of 2,676 prefixes, obtained from MaxMind database. We geo-located the
returned IP addresses using MaxMind and only found 17 locations of their datacenters.
31
However, using a better geo-location services (we used free geo-ip databases from
IP2Location [16] and DB-IP [10] in addition to MaxMind) we could discover 28 out of 32
CDN77 datacenters. The undiscovered datacenters are four datacenters that have status
‘custom datacenter’ in their description. Table 9 shows samples of our queries output to
CDN77. We see that the return IP addresses have a location correlation with the client
subnets used in the queries.
Query string Client subnet Client subnet Returned IP IP address

location address location
101.100.149.0 Auckland, 103.60.240.125 Chippendale,
New Zealand Australia
189.127.48.0 Barra de Sao 189.1.36.27 Brazil
Joao, Brazil
202.59.159.0 Hong Kong 43.245.63.27 Central District,
Hong Kong
202.176.192.0 Singapore 203.174.85.171 Singapore
223.255.244.0 Rajkot, India 43.252.91.30 India
63790025.r.cdn77.net
27.131.9.0 Tokyo, Japan 188.172.202.23 Tokyo, Japan
185.8.187.0 Brixen im 5.159.233.204 Hungary
Thale, Austria
201.140.194.0 Chihuahua 37.235.108.8 Los Angeles,
City, Mexico United States
202.58.166.0 Jakarta, 203.174.85.173 Singapore
Indonesia
Table 9: CDN77 returned IP addresses correlate with the client subnet.
6.3. ECS Adoption by Authoritative DNS Servers
Next, we wanted to detect which websites have adopted the ECS extension. We used
the same detection heuristic we used on previous experiments on CDNs. In this
experiment, we probed the Alexa’s 1 million hostnames using ECS queries. In our
probing, we sent ECS queries using three different client subnets from distant regions.
We picked one client subnet located in Ukraine to represent Europe, one client subnet
32
located in Philippines to represent the Pacific region, and one client subnet located in
Kansas, United States.
Figure 6: ECS presence in queries' responses from 1 million name servers.
From the query responses, we first categorized the hostnames based on the
presence of the ECS option. Figure 6 shows that we successfully resolved the addresses
from 92% of the hostnames. While the majority of the name servers did not include the
extension, there were roughly 28% (279,468) name servers that included the ECS option
in the response. 57,049 of the name servers returned our queries with non-success
status and 16,861 name servers were inaccessible. We also recorded 23,476 name
servers did not support queries with the DNS extension (EDNS0) and only returned an
answer if the extension was omitted from the message. We previously found such a
name server in our CDNs scanning, which is Azure’s authoritative name server. We look
33
at these servers more closely in Section 6.4.3, to see if they introduce brittleness into
DNS system, when some recursive resolvers include ECS option in their queries by
default, and some authoritative name servers fail to resolve these queries.
We further looked into the ECS responses and checked the scope-prefix length in
the option. We found 120,358 name servers put a non-zero scope in the ECS option and
159,110 name servers have a zero scope. Previous findings showed that a zero scope
does not always mean the client subnet is not used, thus we still include the zero-scope
in the following examination. In the next step, we needed to examine the correlation
between the client subnet and the returned IP addresses.
To measure the correlation between the two addresses, we checked the location
of the IP addresses and the geo-distance between the client subnet and the IP
addresses. If the returned IP addresses were all located in one city, we inferred the
client subnets were not used by the authoritative name server. If the returned IP
addresses located in different cities, we then checked if the returned IP address is the
closest to the client subnet from the corresponding query, among the three client
subnets used. If that is the case, we noted the client subnet correlates with the IP
address. In this experiment, we used database from IP2location and MaxMind to obtain
the addresses’ geographical information. We could not use DB-IP database since the
free version does not have the latitude and longitude information.
We automatically annotated a hostname as an ECS adopter if the client subnets
in all three queries correlated with the returned IP address. We also observed
hostnames with only two queries out of three queries that had client subnet correlating
34
with the returned IP address; these hostnames might also be ECS adopters. Table 10
shows an example for such cases. We present the geographical information of the client
subnets and the returned IP addresses from queries to hostname: 8.xyz. We see that
8.xyz returned two IP addresses on our three queries. Query with client subnet in
Philippines returned with an IP address in Republic of Korea and queries with client
subnet in US and Ukraine returned with the same IP address located in US. Its look like
8.xyz is using two data centers and sending clients to one of the two based on the client
location. Ukraine is roughly equidistant but they may know that their connectivity
between Europe and US is better than between Europe and Asia.
For each of the hostnames with cases like 8.xyz, we re-used the same client
subnets and re-sent each query three times. If the results are consistent across three
tries, we annotated the hostname as an ECS adopter.
Client subnet CS Country Returned IP IP Country Distance Distance

address CS – IP CS – other IP
(miles) (miles)
203.215.117.0 Philippines 14.63.214.48 Republic of 1625.67 7296.18
Korea
50.93.228.0 U.S. 23.244.10.193 U.S. 1053.15 6302.36
195.214.197.0 Ukraine 23.244.10.193 U.S 6296.53 4535.08
Table 10: The client subnets and the returned IP addresses on queries to 8.xyz
Ultimately, from the 120,358 non zero-scope name servers we annotated 3,681
name servers as ECS adopters, and from the 159,110 zero-scope name servers we found
14 ECS adopter name servers, with the total of 3,695 name servers. We especially note
35
the fact that a vast majority of authoritative name servers that return a non-zero scope
ECS prefix in fact appear to be non-adopters. This contradicts the proper use of the ECS
extension, which according to [24], states: “A zone is ECS-enabled if its name servers'
responses to queries with ECS option data that has a non-zero prefix”. Whether to call
these servers “adopters” or “non-adopters”, they don’t appear to leverage the ECS
information from queries in generating their response.
To check for “hidden” ECS adopters, which – like Akamai – only honor ECS
options from white-listed resolvers, we also made a comparison between traces of
interactions involved in the DNS resolution of queries with and without ECS. We did not
find any differences in the trace of interactions.
We plotted the ranking position of the 3,695 websites we determined to be ECS
adopters on Alexa’s Top 1 million websites ranking, shown in Figure 8. We saw that 32%
of the name servers that we annotated as ECS adopters are in the top 100K of Alexa
ranking. In general, indicated by the concave shape of the curve on Figure 7, popular
websites are more likely to adopt ECS since they may also employ multiple web servers
in different locations.
36
Figure 7: ECS-adopters’ position on Alexa's ranking
6.4. ECS Adoption by Resolvers
Finally, we will present our effort to explore resolvers that already support the ECS
extension. We determine resolver’s compliance of ECS by examining the existence of
the ECS option in their outgoing queries.
6.4.1. Resolvers That Initialize the Option
The work our colleague described in section 4.3 provided us with a packet trace of a
large number of DNS queries that were sent to his authoritative name server. Even
though the queries were originally sent to resolvers without an ECS option on them, we
wanted to find resolvers that include the ECS option in their outgoing queries.
The packet capture files have a roughly 30 Gigabytes size in total. To make our
analysis on the packet traces easier, we converted the .pcap files into a readable format
files using tshark tool from Wireshark [22]. In the readable format files, we recognized
37
queries with an ECS option by finding queries with EDNS0 option set as ‘CSUBNET –
Client subnet’. Figure 8 shows a sample of a query with ECS option enabled.
Figure 8: A sample of the EDNS0 section from the trace in a readable format.
From the total of 66,982,397 DSN messages, there were 97,854 queries
containing the ECS option. We examined the source IP addresses of these queries and
counted that they were sent from 973 different IP addresses. In Table 11, we present
the list of organizations of the IP addresses that include ECS option in the outgoing
queries.
No Organization Resolvers
1 Google 764
2 Level 3 Communications 57
3 APPRIVER LLC (APPRI-1) 30
4 Chinanet 30
38
5 RIPE Network Coordination Centre (RIPE) 28
6 China Unicom 22
7 China Mobile Communications Corporation / APNIC 15
8 Beijing Xinsixian Keji Co. 11
9 Shenzhen Tencent Computer Systems Company Limited 4
10 Beijing Teletron Telecom Engineering Co. 3
11 Sufeng Dianxin Keji (Shanghai) Co. 2
12 BeiJing Guoxin bilin Telecom Technology Co. 2
13 Beijing Internet Harbor Technology Co. 1
14 China Tianjin Tengxun / APNIC 1
15 JUNDETONGXUN-LTD / APNIC 1
16 China Taiyuan Shanxi Telecom 1
17 Asia Pacific Network Information Centre 1
Table 11: List of the ECS-enabled resolvers’ organizations
Since the original queries were sent without ECS and we found that some of the queries
reached our colleague’s authoritative name server with an ECS option on them, we
learned that even though a client sent a query without ECS option, some DNS servers at
some point between the client and the authoritative name server must have initialized
the option before forwarding the query to the authoritative name server.
6.4.2. Resolvers Forwarding ECS Option
In Sec. 6.4.1, we analyzed the packet trace that was taken in an experiment involving
sending DNS queries without ECS option. Even without the ECS option from the clients,
some resolvers deliberately initialize the ECS option and forward the queries after
including the option in them. In this experiment we wanted to find more resolvers that
enabled ECS in their DNS queries by sending them queries with the ECS option already
included. We sent ECS queries to our authoritative name server through 83,523
resolvers obtained from our colleague’s works. We presumed our ECS queries would
39
trigger the resolver to send the ECS option to our authoritative name server, indicating
their support for the option. To avoid being detected as a non ECS-compliant name
server, we configured our authoritative name server to be ECS-enabled. We were aware
that some of the ECS-compliant resolvers might still omit the ECS option due to the use
of the whitelist.
We examined the traces that were recorded in our probe’s machine and in our
authoritative name server. Table 12 (a) presents the number of queries grouped based
on their responses. Since we sent one query to each of the resolvers, the number of
queries in the group also represents the number of resolvers. We saw that there are
3,852 resolvers that had both an ECS option and an answer in their responses. However,
when we crosschecked with our records from the authoritative name server, we only
found 153 queries that included an ECS option as shown in Table 12 (b). We have two
possible explanations for the discrepancy. The first is that either the ECS-compliant
resolvers use a whitelist and exclude the ECS option on the outgoing queries or their
DNS caching stop the ECS queries being forwarded to our authoritative name server.
Based on the ECS standard, if the client query did include the option, an ECS-compliant
resolver must include one in its response. The other could be that the resolvers did not
actually support ECS and used normal queries to our name server, but they simply
echoed the ECS option back to our probe.
We observed that many of the queries received by our authoritative name server
were sent from different IP addresses that were not on the list of the resolvers that we
sent queries to. Many of the resolvers that we sent our queries to, acted as forwarding
40
resolvers. We checked the organization of the egress resolvers that sent us the ECS
queries. All of the IP addresses of the resolver are owned by Google, except one IP
address, 103.235.235.235, which belongs to Beijing Internet Harbor Technology Co.,Ltd.
These two organizations are already in our list of resolvers that include an ECS option in
their outgoing queries, presented in Table 11 (Section 6.4.1).
Type of responses Number of

queries
Response has a status = 0 (SUCCESS), an ECS option and an answer 3852
Response has a status = 0 (SUCCESS), an ECS option and no answer 287
Response has a status = 0 (SUCCESS), without ECS option and has an 18267
answer
Response has a status = 0 (SUCCESS), without ECS and has no answer 1390
Response has a status = 5 (REFUSED) 17758
Response has a status = 3 (NXDOMAIN) 17
Response has a status = 2 (SERVER FAIL) 373
Response has a status = 1 (FORMERR) 4499
Queries time out. 36964
Response from unexpected source 116
(a) Recorded queries in our probe’s machine
Type of request Number of

queries
Request without an ECS option 24338
Request has ECS option 153
(b) Recorded queries in our authoritative name server
Table 12: ECS queries sent through the 83,523 resolver to our name server
6.4.3. Can ECS-Compliant Resolvers Break a DNS Query?
From previous experiments, we noticed that some authoritative name servers did not
respond properly to ECS DNS queries. For an example, as shown in Figure 9, we found
41
that MS Azure responded to our ECS DNS query with a response code FORMERR. When
we sent the same query without an ECS option, the name server successfully replied
with an answer. Such name server behavior was unexpected from a big CDN like MS
Azure. According to RFC6891 – EDNS0 [27], regardless of the type of option in the DNS
extension, if the name server does not recognize it, the option must be ignored.
With many authoritative name servers potentially behaving like MS Azure’s, ECS-
compliant resolvers must be able to re-send the query without the ECS option to
complete the DNS resolution. Imagine a client that wants to access content served by
MS Azure uses an ECS-enabled resolver that does not re-send the queries. The resolver
would never resolve any hostname from the Azure’s domain, effectively blocking the
client’s access to the content. In the next experiment, we tried to find an ECS-enabled
resolver that would break access to a name server that refused an ECS query.
(a) An ECS query to MS Azure’s name server returned with status FORMERR
42
(b) The same query sent without ECS returned with an answer
Figure 9: A name server that reject ECS queries and returns status error
First, we needed to find resolvers that are certain to send outgoing queries with
an ECS option regardless of the existence of the ECS option in the incoming query sent
by a client. To collect such resolvers, we sent a series of queries through candidate
resolvers to our authoritative name server. We then checked if the incoming queries
consistently included the ECS option on several occasions. We used the 83,523 resolvers
as our candidate resolvers and sent queries through them to our ECS disabled
authoritative name server. On the first query batch, we recorded 168 ECS queries sent
to our authoritative name server. The number of recorded ECS queries is different with
our previous experiment where we recorded 153 ECS queries. There are two possible
explanations for this:
1. Some of the resolvers that are involved in sending ECS queries to our authoritative
name server were not open on the previous experiment.
43
2. Some of the forwarding resolvers were not constantly forwarding queries to the
same resolvers on both experiments.
As we saw from previous experiment, the queries were received from IP
addresses that were not on the list of 83,523 resolvers and we actually sent our queries
to forwarding resolvers that forwarded our queries to other resolvers and ultimately an
egress resolvers sent the queries to our authoritative name server.
It would be simpler if we could directly use the egress resolvers for our Azure
experiment. However, these resolvers were not open to us. We need to use the
forwarding resolvers to use them. We observed that a majority of the client subnets in
the queries are a prefix of an IP addresses from the 83,523 resolvers. There are 193 IP
addresses that were covered by the /24 prefixes in the ECS queries recorded at our
authoritative name server. We then re-sent our queries, but this time only to the 193
forwarding resolvers that were possibly involved in the sending of previous ECS queries.
We recorded 116 of the forwarding resolvers forward the queries to our name
server. The presence of the ECS option and the client subnet were consistent on three
retries, therefore we (a) assumed these egress resolvers supposedly would also send
ECS queries to Azure and (b) knew how to access these egress resolvers consistently
through open forwarding resolvers.
Next, we sent queries to the forwarding resolvers to resolve Azure’s CNAME
gigabyte1.ec.azureedge.net and check the responses to see if any of the resolvers break
access to Azure. A majority of the resolvers successfully resolved our queries. We found
several responses coming with error status but none of them had status FORMERR, a
44
status we receive if we directly send ECS queries to Azure’s name server. We therefore
checked if these resolvers generally functioned properly, by sending them several
queries for non-Azure hostnames. Indeed, the resolvers also returned the same error
status.
We know that in order to resolve Azure’s hostnames, a resolver that sends an
ECS query to Azure’s authoritative name servers needs to re-send the query without
ECS. Supposedly, the total query time to resolve Azure will be higher than an average
time to resolve other hostnames. To make a comparison of the timing, we sent another
set of queries to forwarding resolvers that successfully resolved Azure (at that time, only
103 of the forwarding resolvers open), to resolve CDN77’ CNAME
955486542.r.cdn77.net and Akamai’s CNAME anywhere.ebay.com.edgekey.net. We then
compared the recorded response time among queries to the three CDNs.
Figure 10: Queries time comparison between Azure, Akamai and CDN77
45
Figure 10 plots the response times of our queries to Azure, Akamai and CDN77. A
majority of the resolvers responded with similar response times between the three.
Presumably for the resolvers that showed little response time differences, the response
times are dominated by the delays from our probing host to the forwarding resolver to
the egress resolver; DNS caching could also mask extra latency in some of the cases
where the resolver need to re-send the query. Multi-second spikes may be caused by
DNS timeouts, when the resolver only re-sends the query after the DNS timeout.
7. Exploring the Use of CDN Federations and Multi-CDNs
7.1. Multi-CDN
In one of our initial tests on sending ECS DNS queries, we performed DNS resolution to
resolve a query string edu.qq.com, one of the hostnames that outsourced their content
delivery to Akamai, using various client subnets. We noticed that not all answers were
directed to an Akamai edge server. We looked into the queries that were not directed to
Akamai and found that these queries had one thing in common; the client subnets
belonged to China. On the other hand, queries that were directed to Akamai had a non-
Chinese client subnet. The authoritative name server of a content provider like qq.com
that actively directs their clients to different sets of servers, may use more than one
CDN to deliver content to their client.
46
The practice of combining multiple CDN providers into one network by the
content provider is called multi-CDN. The main purpose of multi-CDN is to spread the
content across multiple CDNs to further increase global presence. Multi-CDN helps
content providers reduce the latency to deliver content and reduces the content
provider’s reliance of a single CDN. With multiple CDNs to select from, content providers
can direct the client to the CDN that is better represented in the client’s region, to one
that is better suited for specific content, or use any other selection criteria. The use of
y.7
Client y y.8
? example.com y.6 Edge server
CDN A
y.4
y.1
y.5
y.2
Local DNS resolver Authoritative DNS server
CDN A
y.3
x.2
Authoritative DNS server x.4

example.com x.3
x.5
x.1
Authoritative DNS server
Local DNS resolver CDN B
x.6
x.7
Client x x.8
? example.com Edge server
CDN B
Figure 11: DNS resolution on multi-CDN implementation
47
multi-CDN also improves redundancy and -- using automatic failover -- content delivery
could be switched from a failed CDN to another, online, CDN. Figure 11 shows the
general DNS resolution that implements multi-CDN. Suppose client X and client Y both
want to access content from website example.com. The website employs multiple CDNs
including CDN A and CDN B to deliver content to the clients. Using DNS redirection, the
website’s authoritative name server directs client X to be served by CDN B and client Y
by CDN A.
To detect a hostname that employs multi-CDN, we need to send multiple DNS
queries to resolve the hostname. If the hostname is directed to a different CDN on a
different query, the hostname’s authoritative name server then can be said to
implement multi-CDN.
(a) Query from client in Hamburg, Germany
(b) Query from client in Ohio, US
Figure 12: Example of multi-CDN DNS resolution
48
One of the factors that could trigger DNS re-direction by the authoritative name
server is the location of the query’s client. Figure 12 shows an example illustrating a
multi-CDN setup where hostname www.roposo.com uses two CDN providers, Akamai
and Cloudfront. In Figure 12(a), using a query from Germany, we see the hostname
directed to Akamai. On a different query, the hostname directed to Cloudfront when the
query was sent from the United States, shown in Figure 12(b).
The ECS extension can potentially be leveraged to detect multi-CDN
deployments, and we would like to investigate this potential. Suppose there is a
hostname that employs multi-CDN and has an ECS-enabled authoritative name server.
We want to see if the use of ECS DNS queries to such hostnames from a single vantage
point with multiple client prefixes to emulate queries from various location on the globe
could trigger the DNS re-direction to multiple CDNs. In one of our experiments, we
probed 89,135 CDN-directed hostnames using ECS DNS queries with three dispersed
client prefixes for each of the hostnames. Using data from this experiment, we detected
26 hostnames outsourced their content to multiple CDNs. Table 13 presents samples of
the hostnames that employ multiple CDNs while Table 14 shows multiple CDNs which
these hostnames employed together.
No Hostnames CDNs CDN’s CNAMEs

1 res.levexis.com Edgecast wac.9e4d.edgecastcdn.net.
Akamai res.levexis.com.edgesuite.net.
2 img0.etsystatic.com Fastly etsy.map.fastly.net.
Edgecast cs136.wac.handmadecdn.net.
Akamai img0.etsystatic.com.edgekey.net
3 onesearch.svc.primedia CDNetwork onesearch.svc.primedia.com.cdngc.net.
.com Akamai primedia.edgesuite.net.
49
4 cdn-1.metacdn.net Edgecast cs16.wpc.sigmacdn.net.
Cloudfront dnqvbasz3oamr.cloudfront.net.
5 cdn.e96.ru CDNvideo e96.cdnvideo.ru.
Onapp 182073765.r.worldcdn.net.
Table 13: Samples of hostnames detected employ multi-CDN
Cdn 1 Cdn 2 Cdn 3

Akamai Fastly Edgecast
Akamai Limelight
Akamai Cdnetwork
Akamai Cloudfront
Akamai Fastly
Cdnetwork Edgecast
Cdnvideo Onapp
Cloudflare Fastly
Cloudfront Cloudflare
Cloudfront Fastly
Cloudfront Edgecast
Table 14: The combination of CDNs employed in multi-CDNs – single vantage point
We observed that the use of ECS DNS queries with multiple client prefixes to
resolve the hostnames does not seem to affect the DNS re-direction to multiple CDNs.
Repeated trials showed that the CDNs returned were seemingly picked at random,
regardless of the client subnet, and in fact 6 of 26 authoritative name servers did not
return ECS option.
7.2. CDN Federation
Another practice of combining several CDNs – this time by the CDNs themselves – is
called CDN federation. The objectives of CDN federation are to reduce network
50
transport costs by sharing resources between CDN members, allow individual CDN
federation members to capture wider market, and allow content provider to have an
agreement with only one of CDN federation member but benefit from wider footprint
and greater capacity of the combined CDN platforms.
Figure 13 shows common DNS resolution that implements CDN federation.
Suppose example.com is a client of CDN A and CDN A is a member of a CDN federation
Authoritative DNS server Authoritative DNS server

example.com CDN A
3 4
2 5
Edge server
6
CDN A
7
DNS resolver Authoritative DNS server
CDN B
10 Edge server
Client CDN B
? example.com
CDN Federation
Figure 13: DNS resolution on CDN federation implementation
where CDN B also a member. When a client wants to access content from the
example.com website, the name server directs the client to be served by CDN A. CDN
51
A’s authoritative name server detects that the client is located closer to the serving area
of CDN B then decides to re-direct the client to CDN B. In the end the client gets the
content of example.com from CDN B, even though example.com and CDN B may not
have a direct agreement between them. For more detail explanation on CDN federation
see [35].
Unlike multi-CDN detection that requires more than one query, CDN federation
could be seen in the DNS resolution of a single query. If the resolution contains a chain
of CNAMEs that are owned by more than one CDN, we annotate that those CDNs are
members of a CDN federation. In Figure 14 we see an example of a CDN federation that
has ChinaCache and Akamai as its members. DNS resolution of the hostname assets.ray-
ban.com shows it is first directed to assets.ray-ban.com.ccgslb.com, a Chinacache’s
CNAME, before being redirected to an Akamai’s CNAME dsan.scene7.com.edgekey.net.
Figure 14: A CDN federation example
Using the data from our experiment on ECS adoption by CDNs that we discussed
in section 6.2, we discovered 730 hostnames bounced from one CDN to another. Table
15 shows how they are paired in the queries’ trace of interaction. Like previous multi-
CDN detection, the client prefixes on the ECS option seems did not affect the DNS
52
routing to another CDN. Regardless the client prefixes sent to each of the hostnames, all
queries showed the same pair of CDNs.
Akamai Chinanet Cloudflare Cloudfront Edgecast Incapsula

Akamai √ √ √ √
Chinacache √ √ √
Chinanet √
Cloudflare √
Msazure √ √
Rackspace √
Yotta √
Table 15: Pairs of CDNs detected in the federation – single vantage point
(CDNs in the left column redirect to CDNs in the top row)
Our attempt to discover which authoritative name servers implement multi-
CDNs and which CDNs collaborate in a CDN federation using queries from a single
vantage point, while utilizing ECS option to emulate various location on the globe,
produced a limited detection. In the following experiment we further investigate the
detection of multi-CDN and CDN federation using DNS queries sent from various
vantage points around the globe by employing PlanetLab nodes. To ensure our vantage
points were representing the distribution of client prefixes across the world, we
clustered all 272,246 prefixes from the IPv4 advertised prefixes obtained from
bgp.potaroo.net [7] into 50 clusters based on the estimated distance between prefixes.
We then selected PlanetLab nodes that were close to the center of the clusters.
We could do this on the majority of the clusters. However not all clusters have active
nodes on them. Clusters that covered Africa, The Middle East, South Asia and West
53
Australia did not have an active node that we could use. To substitute the vantage
points from the aforementioned clusters, we picked active PlanetLab nodes that are still
widely distant from other nodes that we already picked. In the end we had 47 PlanetLab
nodes as our vantage points. From these nodes, we probed 89,135 CDN-directed
hostnames and captured all the traces of interactions in the DNS resolution.
In this experiment, we also wanted to see if the detection of multi-CDN and CDN
Federation increases with the increasing number of vantage points used. For that
purpose, we clustered the 47 PlanetLab nodes into several smaller clusters to have
clusters of 2, 5, 10, 20 and 47 vantage points. We recorded the number of detections
with each cluster and plotted the difference.
7.3. CDN Federation Footprints
We first reported our detection of CDN federations using the technique we previously
explained. From data on each of the vantage points, we detected hostnames that use
CDNs involved in CDN federations were ranging from 728 to 744 hostnames. As
illustrated in Figure 15, the trend shows more websites are detected outsourced to a
CDN federation with the increase of vantage points used. There was a significant
increase in the detection from the use of 1 vantage point (we used the median value of
all vantage points) to 5 vantage points but the new detections began to gradually drop
with the addition of vantage points.
In each detection, it shows only two CDNs involved in the federation. Table 16
shows how they are paired in the queries’ trace of interaction. In the row headers are
54
Figure 15: CDN Federation detection from several sets of vantage points
CDNs that are directly linked from the content delivery’s authoritative name server and
in the column headers are CDNs that ultimately deliver content to the client. We saw
that Cloudfront and Edgecast served as the end point in a federation while MirrorImage
and Yotta were the front end linked directly with the content providers. It is interesting
to see that Akamai, the biggest CDN, operates as both of the roles. Based on their
massive infrastructure, one could expect that Akamai would be mostly used to deliver
content to clients rather than being the front end. However, being the front end also
indicates that Akamai has a large number of customers.
Chinacache Akamai Cloudflare Chinanet Edgecast Cloudfront Highwind

Rackspace v
Chinacache v v v
Akamai v v v v v
Azure v v
Cloudflare v
Mirrorimage v
Yotta v
Chinanet v
55
Cdnetworks v
Incapsula v
Table 16: Pairs of CDNs detected in the federation – multiple vantage points
(CDNs in the left column redirect to CDNs in the top row)
7.4. Multi-CDN Footprints
Next, we considered hostnames that employ multi-CDN and how the discovery of these
hostnames increased as we increased the number of vantage points. Similar to our
result on CDN federation detection, as shown in Figure 16, we also recorded an increase
in multi-CDN detection when we added the number of vantage points. Data from 2
vantage points showed a total of 136 hostname using multi-CDN. The number of multi-
CDN detection rose significantly on the use of 5 vantage points before gradually leveling
off with further addition of vantage points.
Figure 16: Multi-CDN detection from several set of vantage points
56
Figure 17 presents the CDF of the number of CDNs detected for each of the
hostnames that we probed in this experiment. Even though all the 89,137 hostnames
that we used in this experiment were directed to some CDNs in prior experiments, 15%
of them are not directed to a CDN in this experiment, thus plotted to 0 CDN. 83% of the
hostnames used 1 CDN and only a small number of hostnames employed multi-CDN;
176, 26 and 14 hostnames used 2, 3 and 4 CDNs respectively. Which multiple CDNs
these hostnames employed together shown in Table 17.
Figure 17: The number of CDNs present on CDN-directed hostname resolutions
CDN 1 CDN 2 CDN 3 CDN 4

Akamai Chinanetcenter Edgecast Level3
Akamai Cdnetworks Cloudflare Incapsula
Akamai Edgecast Highwind Maxcdn
Chinacache Chinanetcenter Edgecast Level3
Chinanetcenter Cloudfront Edgecast Limelight
Chinanetcenter Cloudflare Keycdn Maxcdn
57
Akamai Chinacache Chinanetcenter
Akamai Cloudfront Highwind
Akamai Cdnetworks Edgecast
Akamai Limelight Maxcdn
Akamai Edgecast Level3
Akamai Highwind Maxcdn
Cachefly Cloudfront Fastly
Cdnetworks Cloudflare Incapsula
Cdnetworks Edgecast Level3
Cdnetworks Edgecast Highwind
Chinanetcenter Cloudfront Edgecast
Cloudflare Cloudfront Incapsula
Cloudfront Edgecast Limelight
Edgecast Fastly Highwind
Edgecast Highwind Leaseweb
Akamai Cloudfront
Akamai Level3
Akamai Edgecast
Akamai Fastly
Akamai Cdnetworks
Akamai Chinacache
Akamai Limelight
Akamai Highwind
Akamai Chinanetcenter
Akamai Maxcdn
Cdnetworks Edgecast
Cdnetworks Fastly
Cdnetworks Chinanetcenter
Cdnetworks Cloudfront
Cdnetworks Level3
Cdnvideo Onapp
Chinacache Cloudflare
Chinacache Cloudfront
Chinacache Chinanetcenter
Chinanetcenter Keycdn
Chinanetcenter Cloudfront
Chinanetcenter Cloudflare
Chinanetcenter Fastly
Chinanetcenter Msazure
Chinanetcenter Edgecast
Cloudflare Cloudfront
58
Cloudflare Fastly
Cloudflare Incapsula
Cloudfront Incapsula
Cloudfront Edgecast
Cloudfront Fastly
Cloudfront Maxcdn
Cloudfront Limelight
Cloudfront Level3
Edgecast Highwind
Edgecast Level3
Edgecast Fastly
Edgecast Maxcdn
Edgecast Incapsula
Highwind Internap
Highwind Maxcdn
Maxcdn Msazure
Table 17: The combination of CDNs in the multi-CDNs found – multiple vantage points
Aside from the detection of hostnames that employ multi-CDN, we also
discovered hostnames that while outsourced their content to CDN, still direct some
request to their own server. We found 2589 hostnames direct client either to their own
server or to a CDN on multiple queries. We see that content providers now have more
options on how they deliver their content to client, either using their own servers
or/and outsourced the delivery to one or more CDNs.
Overall, we find that the ECS option is not an effective mechanism to detect the
use multi-CDNs. While we identified 26 hostnames as using a multi-CDN setup with ECS,
we were able to detect 216 hostnames using multiple vantage points.
59
8. Conclusion
In this thesis we explored the adoption of the EDNS-Client-Subnet extension (ECS) by
major CDNs, authoritative name servers and resolvers. By using a series of ECS DNS
queries, we inferred that a majority of CDNs have adopted ECS. Our scanning in fact
exploited the unintended uses of this recently proposed DNS extension.
Using CDN77 as example, our experiments show that one can simply uncover the
serving servers of ECS adopters by using a single vantage point. An early adopter of ECS
needs to address this security vulnerability that comes as a consequence of enabling
ECS. We showed that Akamai, the largest CDN, has successfully addressed this potential
vulnerability. The growing adoption of ECS by resolvers needs to be anticipated by
authoritative name servers to make sure they are able to provide an answer to ECS
queries even though they have yet to adopt ECS. Failing to do so may result in
unnecessary re-query and delays.
On the other hand, an ECS-compliant resolver needs to re-send queries without
the option when their ECS query is refused by the target authoritative name server.
Finally we found that many major CDNs are involved in CDN federation, including the
largest CDN, Akamai, which is rather surprising given Akamai’ own extensive footprint.
We also showed that some content providers deliver their content using multiple CDNs.
Our overall conclusion is that ECS has reached a critical mass of adoption by
CDNs but not by individual content providers. We speculate that the reason for the
latter is that most content providers scale their web sites by subscribing to CDNs rather
60
than building up their own distributed platforms. However, many content providers
that have not adopted ECS, instead of just ignoring the ECS option, fail to process ECS
queries altogether, leading to query timeouts and delays.
Furthermore, most CDNs that adopt ECS fail to protect themselves from
exposing their platforms to easy scans by an adversary. Thus, the emergence of ECS
requires action from web platform operators whether or not they decide to make use of
this new DNS feature.
61
9. Bibliography
[1] Akamai. https://www.akamai.com/uk/en/cdn/.
[2] Akamai Community.
https://community.akamai.com/community/web-performance/
blog/2014/12/16/google-public-dns-and-location-sensitive
-dns-responses
[3] Akamai Technologies Wiki.
https://en.wikipedia.org/wiki/Akamai_Technologies.
[4] Alexa top sites.
http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
[5] BIND software. https://www.isc.org/downloads/bind/
[6] Bind’s dig ECS patch. https://www.gsic.uva.es/~jnisigl/dig-
edns-client-subnet.html
[7] BPG IPv4 reports. http://bgp.potaroo.net/index-ale.html
[8] CDN Planet, http://www.cdnplanet.com/cdns/.
[9] ClientSubnetOption.
https://github.com/opendns/dnspython-clientsubnetoption
[10] DB-IP. https://www.db-ip.com/db/download/city
[11] Dipzoom - deep internet performance zoom. http://dipzoom.case.edu.
[12] dnspython. http://www.dnspython.org/
[13] gdnsd. http://gdnsd.org/
[14] Google Public DNS.
62
https://developers.google.com/speed/public-dns.
[15] Google public DNS announcement.
https://groups.google.com/forum/#!topic/public-dns-
announce/67oxFjSLeUM
[16] IP2Location Lite
http://lite.ip2location.com/database/ip-country-region-
city
[17] MaxMind. http://www.maxmind.com.
[18] OpenDNS. http://www.opendns.com.
[19] PowerDNS. https://www.powerdns.com/auth.html
[20] PowerDNS Recursor. https://www.powerdns.com/recursor.html
[21] PlanetLab. https://www.planet-lab.org/
[22] Wireshark software. https://www.wireshark.org/
[23] B. Ager, W. Muhlbauer, G. Smaragdakis, and S. Uhlig. Comparing DNS Resolvers in
the Wild. In ACM IMC, 2010.
[24] M. Calder, X. Fan, Z. Hu, E. Katz-Bassett, J. Heidemann, and R. Govindan. Mapping
the Expansion of Google’s Serving Infrastructure. In ACM IMC, 2013.
[25] F. Chen, R. K. Sitaraman, and M. Torres. End-User Mapping: Next Generation
Request Routing for Content Delivery, Proceedings of the 2015 ACM Conference
on Special Interest Group on Data Communication, August 17-21, 2015, London,
United Kingdom.
63
[26] C. Contavalli, W. van der Gaast, D. Lawrence, and W. Kumari. Client Subnet in DNS
Queries. RFC 7871, DOI 10.17487/RFC7871, May 2016.
[27] J. Damas, M. Graff , and P. Vixie, "Extension Mechanisms for DNS (EDNS(0))", STD
75, RFC 6891, DOI 10.17487/RFC6891, April 2013.
[28] C. Labovitz, S. Lekel-Johnson, D. McPherson, J. Oberheide, and F. Jahanian.
Internet Inter-Domain Traffic. In ACM SIGCOMM, 2010.
[29] C. Huang, A. Wang, J. Li, and K. Ross. Measuring and Evaluating Large-scale CDNs.
In ACM IMC, 2008
[30] M. Jansen. (2014). EDNS0 Client-Subnet for DNS based CDNs [pdf]. Retrieved from
http://www.sanog.org/resources/sanog24/akamai-mj-end-
user-mapping-sanog.pdf
[31] P. Kintis, Y. Nadji, D. Dagon, M. Farrell, and M. Antonakakis. Understanding the
Privacy Implications of ECS. In DIMVA 2016.
[32] B. M. Maggs and R. K. Sitaraman, Algorithmic Nuggets in Content Delivery, ACM
SIGCOMM Computer Communication Review, v.45 n.3, July 2015.
[33] Z. Mao, C. Cranor, F. Douglis, M. Rabinovich, O. Spatscheck, and J. Wang. A Precise
and Efficient Evaluation of the Proximity between Web Clients and Their Local DNS
Servers. In USENIX ATC, 2002.
[34] J. S. Otto, M. A. Sanchez, J. P. Rula, and F. E. Bustamante. Content delivery and the
natural evolution of DNS - Remote DNS Trends, Performance Issues and
Alternative Solutions. In ACM IMC, 2012.
64
[35] S. Puopolo, M. Latouche, F. Le Faucheur, J. Defour. Content delivery network
(CDN) federations: How SPs can win the battle for content-hungry consumers.
Cisco Internet Business Solutions Group (IBSG); 2011.
[36] M. A. Sanchez, J. S. Otto, Z. S. Bischof, D. R. Choffnes, , F. E. Bustamante, B.
Krishnamurthy, and W. Willinger. Dasu: Pushing Experiments to the Internet’s
Edge. In USENIX/ACM NSDI, 2013.
[37] F. Streibelt, J. B¨ottger, N. Chatzis, G. Smaragdakis, and A. Feldmann. Exploring
EDNS-client-subnet adopters in your free time. In IMC, 2013.
[39] S. Triukose, Z. Wen, and M. Rabinovich. Measuring a Commercial Content Delivery
Network. In WWW, 2011.
65

EDNS Masters Thesis

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EDNS Masters Thesis

Uploaded by

Copyright:

Available Formats

THE STATE OF ADOPTION OF DNS ECS EXTENSION

FAJAR UJIAN SUDRAJAT

Submitted in partial fulfillment of the requirements for the degree of

Department of Electrical Engineering and Computer Science

CASE WESTERN RESERVE UNIVERSITY

SCHOOL OF GRADUATE STUDIES

We hereby approve the thesis of

FAJAR UJIAN SUDRAJAT

candidate for the MASTER OF SCIENCE degree*.

Committee Chair: Michael Rabinovich

Committee Member: Vincenzo Liberatore

Committee Member: Andy Podgurski

*We also certify that written approval has been obtained

List of Tables ................................................................................................................... iii

List of Figures ................................................................................................................. iv

2. Related Work .............................................................................................................4

3.1. The Domain Name System ................................................................................ 5

3.2. Content Delivery Networks ............................................................................... 7

3.3. ECS EDNS Extension ........................................................................................ 10

3.4. Enabling ECS DNS Query ................................................................................. 11

4.1. Selected CDNs ................................................................................................ 13

4.2. CDN CNAMEs .................................................................................................. 15

4.3. List of Resolvers .............................................................................................. 16

5. Mapping Akamai Using ECS ......................................................................................17

5.1. ECS DNS Queries via Google Public DNS .......................................................... 17

5.2. DNS Queries to Our Own ADNS via Google ..................................................... 22

5.3. ECS DNS Queries Using Our Own Resolver ...................................................... 25

6. Exploring ECS Adopters ............................................................................................27

6.1. Detecting ECS Adopters .................................................................................. 27

6.2. ECS Adoption by CDNs .................................................................................... 28

6.4. ECS Adoption by Resolvers .............................................................................. 37

6.4.1. Resolvers That Initialize the Option .................................................... 37

6.4.2. Resolvers Forwarding ECS Option ....................................................... 39

6.4.3. Can ECS-Compliant Resolvers Break a DNS Query? ............................. 41

7. Exploring the Use of CDN Federations and Multi-CDNs ............................................46

7.1. Multi-CDN ....................................................................................................... 46

7.2. CDN Federation .............................................................................................. 50

7.3. CDN Federation Footprints ............................................................................. 54

7.4. Multi-CDN Footprints ...................................................................................... 56

Table 1: List of CDNs and their serving domains ........................................................ 15

Table 2: Akamai's CNAMEs and their IP addresses .................................................... 18

Table 3: Itunes.apple.com.edgekey.net.’s resolved IP addresses .............................. 20

non-zero ECS prefixes. ................................................................................. 21

Table 5: An example of Akamai's CNAME and their IP addresses .............................. 25

Table 7: CDNs ECS adoption ...................................................................................... 29

Table 8: KeyCDN direct client based on the subnet client. ........................................ 31

Table 11: List of the ECS-enabled resolvers’ organizations .......................................... 39

Table 13: Samples of hostnames detected employ multi-CDN .................................... 50

Table 17: The combination of CDNs in the multi-CDNs found – ......................................

multiple vantage points ............................................................................... 59

Figure 1: The interaction of DNS resolution process ..................................................... 6

Figure 2: The system model of CDN ............................................................................. 8

Figure 3: An example of a poor DNS redirection......................................................... 10

Figure 4: ECS option structures .................................................................................. 11

Figure 5: An output example of dig with ECS patch .................................................... 12

Figure 7: ECS-adopters’ position on Alexa's ranking ................................................... 37

Figure 11: DNS resolution on multi-CDN implementation ............................................ 47

Figure 12: Example of multi-CDN DNS resolution ......................................................... 48

Figure 13: DNS resolution on CDN federation implementation .................................... 51

Figure 14: A CDN federation example .......................................................................... 52

Fajar Ujian Sudrajat

some content providers employ multiple CDNs to deliver their content.