DNS: Content Blocking at Scale

This article was written based on a talk given by AdGuard's CTO and co-founder Andrey Meshkov at Ad Blocker Dev Summit 2021. So you can either watch the video or read this text to learn about content blocking at scale.

Absolutely everything is connected to the Internet these days, from TV to smart light bulbs, from mobile devices to smart auto. Given that ads and ad trackers are everywhere the Internet is, a browser-based ad blocker seems to be not enough. It provides just a tiny window into the "better Internet" without aggressive and intrusive banners vying for your attention. But what if you want to widen this "window" and have it all?

Then buckle up and get ready for a fascinating journey through the past, present, and future of DNS filtering. Why? Because DNS is the answer!

What DNS-level blocking is

Just to refresh your knowledge, DNS stands for "Domain name system", and its purpose is to translate websites' names into something browsers can understand, i.e. IP addresses. You can think of DNS as the "address book" of the Internet. So each time you go to a website, your browser sends a DNS query to a DNS server (which is usually provided by your Internet Service Provider) to figure out the IP address of the website. And a regular DNS resolver simply returns the IP address of the requested domain.

Your device always uses some DNS server to obtain IP addresses of the domain name apps want to navigate to.

There are also DNS servers that provide DNS-level blocking. Usually, it is done by using the DNS Sinkholing technique, which means handing out non-routable addresses for blocked domains. When your device sends a "bad" request, be it an ad or a tracker, a DNS server responds with the 0.0.0.0 IP address for a blocked domain. So, the app simply cannot connect to this address and we have the desired result — the connection is prevented.

An example of using a DNS server to block access to a domain

Other options are responding with NXDOMAIN (which means that the domain does not exist) or REFUSED (which means that the server refused to process the query). You might not notice the difference between them, but some old devices may misinterpret REFUSED and try to use a fallback DNS server.

DNS predecessor — HOSTS file

Surprisingly, DNS-level blocking is the oldest known approach to ad blocking. How is that possible?

Back in the days when the Internet was young and called ARPANET, there was devised a naming system that mapped a computer's IP address to a unique ASCII-based identifier. With only a few computers linked to ARPANET, the US Defense Department's Network Information Center maintained a file named HOSTS.TXT — a master list of each computer's address and its hostname. System administrators would download hosts.txt periodically. If you knew an ARPANET computer's name, you would simply look up its IP address in the HOSTS file.

This is how the ARPANET HOSTS.TXT file looked like

But it soon became clear that the exploding number of hosts was putting too great a load on the NIC computer. Everyone wanted their computers on ARPANET, but they had to wait for NIC to update HOSTS.TXT. Something else was needed. So in late 1987, Paul Mockapetris outlined the Domain Name System.
But the HOSTS file was not forgotten. Soon after the first Internet ads appeared, people discovered that the HOSTS.TXT can be used as a blocklist to prevent downloading of the ads using pretty much the same DNS sinkholing approach.

Once this was discovered, some people devoted their time to maintaining the first blocklists and sharing them with everyone. There are some notable examples of blocklists with more than 20 years of history that are still actively developed to this day, like Peter Lowe’s blocklists, Steven Black’s blocklists or Dan Pollock’s hosts.

Snippet from Peter Lowe’s blocklist

When the first version of Adblock for Firefox appeared, some people simply couldn’t understand the purpose. They were just happy with what the HOSTS file was doing for them.

Snippet from forum

However, there are some issues with the HOSTS file that aren’t easy to solve.

• The distribution of updates. There is no good way to distribute a HOSTS file update to the users. You either need to have a piece of software do it for you, or you need to be very patient and periodically do it manually.
• HOSTS files tend to be HUGE since you cannot use wildcards and have to list every domain there. The HOSTS file syntax is simply not supposed to be used that way. Just take a look at the picture below where one of the largest blocklists — Energized Unified — is shown. As you can see, it contains over 700K domains and weighs about 21 megabytes. Distributing a 21 Mb file to millions of users daily is already a serious problem.

Energized Unified blocklist

Public DNS servers that block ads appeared as a viable alternative to using a HOSTS file. It was enough to configure the device to use such a server and users didn’t need to deal with HOSTS files updates anymore. One of the first public DNS servers was called Alternate DNS and it appeared in 2015. It was maintained by a single person. Later in 2016, we launched AdGuard DNS.

New life for DNS blockers

Before the last couple of years, DNS as a standard was almost frozen with only slight changes. But then it all exploded with new standards. And the big topic is DNS Encryption: DNS-over-TLS, DNS-over-HTTPS, DNS-over-QUIC, all of which have become mainstream and have had a big impact on content blocking!

Before the rise in popularity of DNS encryption the options were too complex for a casual user:

• Use the HOSTS file and somehow receive updates to it.
• Use a plain DNS server and accept the limitations, such as the inability to set it up for a mobile network on mobile. Or the necessity to configure every network adapter on desktops and dive into rather low-level settings to do that.
• Use a VPN (or a local VPN) software.

After DNS encryption became commonplace, some technical obstacles simply disappeared. Now it is much easier to configure a DNS server and it can be used on any network. Granted, you still need to dive into the device settings, but not as deep as before. DNS encryption is natively supported on Android 9+ (DoT, DoH is coming), iOS 14+ and macOS 15+ (DoT, DoH), Windows 11 (the support is rather limited, though). This alone gave a huge boost to DNS blocking solutions' popularity.

In addition, the development of DNS encryption has allowed public DNS servers to provide users with customization options, in other words, everyone has got an opportunity to choose what to block and what to not block. Earlier applying custom rules based on the DNS server’s domain name was only possible by “linking” the user’s IP address which can hardly be called a good solution.

Using DNS for content blocking: pros and cons

Using DNS for content blocking has some advantages as well as obvious flaws. So let us list them here:

Pros

• Does not require installing additional software.
• Does not depend on the browser or OS vendor.
• No performance cost.
• Running a public DNS server allows you to observe the whole Internet. This is very useful if you maintain a blocklist. You can get rid of unused rules and promptly learn about new threats. DNS has no blind spots since it observes all devices and not just the browsers.
• Centralized solution is better at dealing with some issues.

For example, let’s take a look at CNAME cloaking, a tactic that was employed by some trackers to hide from blockers. By the means of a CNAME DNS record, they were hiding the real third-party domain name disguising it as a first-party domain. With AdGuard DNS we were able to find out all those disguised domains and publish a list of them so that even content blockers with no access to DNS could stop this.

One more example is using proxies for disguising trackers. Basically, you can use CloudFlare or CloudFront to configure a server-less proxy that would hide the original third-party domain. DNS in general cannot help you with this, but it can be a good starting point to detect such proxies.

Cons

• Cannot deal with first-party ads. For instance, you can’t block YouTube video ads because they are hosted on the same domain as legitimate videos.
• No cosmetic filtering. With DNS blocking alone you may not have most of the ads, but you have rather ugly web pages with broken frames and ad placeholders.
• Higher chance of breakage. For instance, some apps or websites may be broken due to blocked Google Analytics, and you can do nothing with that.
• Easier to circumvent. An app may simply choose to use a different DNS server.

What can be improved

DNS filtering has some flaws that might be fixed or at least mitigated.

Access Denied Error Page

Currently, if the domain is blocked on the DNS level, you just see an ugly error page. This hurts the user experience. For instance, they might want to temporarily unblock it and there’s simply no way to do that.

Fortunately, there is a draft RFC (a Request for Comments document) right about that: "DNS Access Denied Error Page" that will allow showing a beautiful error page to the user without resorting to ugly solutions (like requiring a user to install an HTTPS certificate).

Encrypted DNS Discovery Mechanisms

Now you need to configure DNS on every device. Wouldn’t it be better if you could just stick it up in your router? Some new routers already support DNS encryption natively. However, many old routers still accept only plain DNS servers. There is the Adaptive DNS Discovery working group that is committed to improving this situation.

Notable draft RFCs are:

1. "Discovery of Designated Resolvers" that allows discovery of encrypted DNS servers via a special DNS record. Here’s how it is supposed to work. You configure a plain DNS server and then your OS sends a special DNS query to it asking if there’s an encrypted DNS server that it can use.
2. Then the second notable RFC is about a new DHCP option for encrypted DNS servers. This option would allow a router to propagate encrypted DNS settings to the computers connected to that router.

App Detection

Finally, one more thing can be improved. We need to be able to detect what app makes this or that DNS query. There’s no RFC that could help us, though. But we can try to do it ourselves, try to create some kind of "DNS fingerprints" at least for the popular apps.

If we knew what app makes a DNS query, we could be more flexible about what is blocked and what’s not.

1. For instance, we want to block Facebook tracking, but blocking it globally will break all Facebook apps, so we would like to do that in non-Facebook apps only.
2. Another real-life problem is that blocking AppsFlyer (a mobile analytics system) breaks some rather popular apps. We’d prefer to allow it selectively (only for the affected apps) and keep it blocked otherwise.

Creating these DNS fingerprints requires analyzing the network behavior of every popular app and we’re working on a solution that will do it.

What AdGuard DNS is

If you want to get access to the "better Internet", you should definitely use DNS in combination with VPN and ad blocker. And we suggest you take a closer look at AdGuard DNS.

So what is AdGuard DNS? It’s one of the most privacy-friendly DNS services that you can choose. It supports such reliable encryption protocols as DNS-over-HTTPS, DNS-over-TLS, and DNS-over-QUIC. It can identify requests to an ad, tracking, and/or adult domains (optionally), and responds with an empty response. AdGuard has its own base of domain names that serve ads, trackers, and fraud, and it is regularly updated. Furthermore, there’s a great opportunity to add your custom rules to blocklists.

To give you a broader view of what AdGuard DNS is, I will mention some more points:

• It consists of multiple servers located in 14 locations.
• All these servers "announce" the same IP addresses via BGP (Border Gateway Protocol).
• Its current load is about three hundred thousand DNS queries per second.
• AdGuard DNS is written in Golang.
• About 75% of the DNS traffic is encrypted. This is actually what differentiates content blocking DNS servers from others. If you take a look at CloudFlare or Quad9 stats, you’ll see that encrypted DNS is just a small share of all queries. For AdGuard DNS this is very different, most of the traffic is encrypted.

DNS Filtering Rules Syntax

The blocklists of AdGuard DNS are composed using special DNS filtering syntax. We weren’t happy with the limitations of HOSTS files type blocklists so for AdGuard DNS we used familiar adblock-style syntax. It was then gradually extended with DNS-specific modifiers.

Examples of DNS filtering rules. The full spec is available on Github.

Just to demonstrate the difference. On the left side, we have stats for the AdGuard DNS filter (the one that we use by default). It is relatively short, but it actually blocks about 900K domains. And on the left side, we have a different blocklist — Energized Ultimate which is a HOSTS file with 500K domains. And this is what it blocks — 500K domains.

Stats according to AdGuard DNS data

AdGuard DNS Server Node

Each AdGuard DNS server node consists of several parts. On the front end, we have a DNS forwarder (written in Golang) that implements all the logic, i.e. DNS filtering, caching, etc.

On the backend, we have an unbound server instance that provides DNS recursion. A single server node load is from 5 000 to 40 000 responses depending on the node location (some locations are more popular than others).

DNS Filtering Engines Implementations

DNS filtering engine implementations are open source and available on Github. There are two of them. The first is called urlfilter and it is a Golang library that we use in AdGuard DNS and AdGuard Home. Provides relatively high performance and low memory footprint. The second one is AdGuard DnsLibs, which is a C++ library that we use in our client apps. Besides DNS filtering, it also provides DNS encryption implementations.

If you want to protect yourself from ads and trackers with the help of AdGuard DNS, just follow this setup guide.

A glimpse into the future

Browser-based content blockers might be the most popular among casual users, but not among the filter lists maintainers. There are way more hosts files and domains lists maintained by volunteers than the filter lists for browser-based blockers. This is a signal that it has great perspectives.

Stats according to filterlists.com

So, what are the prospects of DNS content blocking? First, the growth of DNS content blocking will accelerate. Windows 11 brings DNS encryption support, new routers also support DNS encryption, this all will make it easier for the users to set it up and to use it.

Circumvention of DNS content blocking rates will increase, but it will still be marginal. It takes a lot of effort from different parties to circumvent it and for that to start happening DNS content blocking should be as popular as browser blockers. And that probably won't happen, despite the growth rate.

DNS will not replace browser-based blockers for obvious reasons, the quality is simply not good enough. But still, it is here to stay.