This article was written based on a talk given by AdGuard's CTO and co-founder Andrey Meshkov at Ad Blocker Dev Summit 2021. So you can either watch the video or read this text to learn about content blocking at scale.
Absolutely everything is connected to the Internet these days, from TV to smart light bulbs, from mobile devices to smart auto. Given that ads and ad trackers are everywhere the Internet is, a browser-based ad blocker seems to be not enough. It provides just a tiny window into the "better Internet" without aggressive and intrusive banners vying for your attention. But what if you want to widen this "window" and have it all?
Then buckle up and get ready for a fascinating journey through the past, present, and future of DNS filtering. Why? Because DNS is the answer!
Just to refresh your knowledge, DNS stands for "Domain name system", and its purpose is to translate websites' names into something browsers can understand, i.e. IP addresses. You can think of DNS as the "address book" of the Internet. So each time you go to a website, your browser sends a DNS query to a DNS server (which is usually provided by your Internet Service Provider) to figure out the IP address of the website. And a regular DNS resolver simply returns the IP address of the requested domain.
Your device always uses some DNS server to obtain IP addresses of the domain name apps want to navigate to.
There are also DNS servers that provide DNS-level blocking. Usually, it is done by using the DNS Sinkholing technique, which means handing out non-routable addresses for blocked domains. When your device sends a "bad" request, be it an ad or a tracker, a DNS server responds with the 0.0.0.0 IP address for a blocked domain. So, the app simply cannot connect to this address and we have the desired result — the connection is prevented.
An example of using a DNS server to block access to a domain
Other options are responding with NXDOMAIN (which means that the domain does not exist) or REFUSED (which means that the server refused to process the query). You might not notice the difference between them, but some old devices may misinterpret REFUSED and try to use a fallback DNS server.
Surprisingly, DNS-level blocking is the oldest known approach to ad blocking. How is that possible?
Back in the days when the Internet was young and called ARPANET, there was devised a naming system that mapped a computer's IP address to a unique ASCII-based identifier. With only a few computers linked to ARPANET, the US Defense Department's Network Information Center maintained a file named HOSTS.TXT — a master list of each computer's address and its hostname. System administrators would download hosts.txt periodically. If you knew an ARPANET computer's name, you would simply look up its IP address in the HOSTS file.
This is how the ARPANET HOSTS.TXT file looked like
But it soon became clear that the exploding number of hosts was putting too great a load on the NIC computer. Everyone wanted their computers on ARPANET, but they had to wait for NIC to update HOSTS.TXT. Something else was needed. So in late 1987, Paul Mockapetris outlined the Domain Name System.
But the HOSTS file was not forgotten. Soon after the first Internet ads appeared, people discovered that the HOSTS.TXT can be used as a blocklist to prevent downloading of the ads using pretty much the same DNS sinkholing approach.
Once this was discovered, some people devoted their time to maintaining the first blocklists and sharing them with everyone. There are some notable examples of blocklists with more than 20 years of history that are still actively developed to this day, like Peter Lowe’s blocklists, Steven Black’s blocklists or Dan Pollock’s hosts.
Snippet from Peter Lowe’s blocklist
When the first version of Adblock for Firefox appeared, some people simply couldn’t understand the purpose. They were just happy with what the HOSTS file was doing for them.
Snippet from forum
However, there are some issues with the HOSTS file that aren’t easy to solve.
Energized Unified blocklist
Public DNS servers that block ads appeared as a viable alternative to using a HOSTS file. It was enough to configure the device to use such a server and users didn’t need to deal with HOSTS files updates anymore. One of the first public DNS servers was called Alternate DNS and it appeared in 2015. It was maintained by a single person. Later in 2016, we launched AdGuard DNS.
Before the last couple of years, DNS as a standard was almost frozen with only slight changes. But then it all exploded with new standards. And the big topic is DNS Encryption: DNS-over-TLS, DNS-over-HTTPS, DNS-over-QUIC, all of which have become mainstream and have had a big impact on content blocking!
Before the rise in popularity of DNS encryption the options were too complex for a casual user:
After DNS encryption became commonplace, some technical obstacles simply disappeared. Now it is much easier to configure a DNS server and it can be used on any network. Granted, you still need to dive into the device settings, but not as deep as before. DNS encryption is natively supported on Android 9+ (DoT, DoH is coming), iOS 14+ and macOS 15+ (DoT, DoH), Windows 11 (the support is rather limited, though). This alone gave a huge boost to DNS blocking solutions' popularity.
In addition, the development of DNS encryption has allowed public DNS servers to provide users with customization options, in other words, everyone has got an opportunity to choose what to block and what to not block. Earlier applying custom rules based on the DNS server’s domain name was only possible by “linking” the user’s IP address which can hardly be called a good solution.
Using DNS for content blocking has some advantages as well as obvious flaws. So let us list them here:
For example, let’s take a look at CNAME cloaking, a tactic that was employed by some trackers to hide from blockers. By the means of a CNAME DNS record, they were hiding the real third-party domain name disguising it as a first-party domain. With AdGuard DNS we were able to find out all those disguised domains and publish a list of them so that even content blockers with no access to DNS could stop this.
One more example is using proxies for disguising trackers. Basically, you can use CloudFlare or CloudFront to configure a server-less proxy that would hide the original third-party domain. DNS in general cannot help you with this, but it can be a good starting point to detect such proxies.
DNS filtering has some flaws that might be fixed or at least mitigated.
Currently, if the domain is blocked on the DNS level, you just see an ugly error page. This hurts the user experience. For instance, they might want to temporarily unblock it and there’s simply no way to do that.
Fortunately, there is a draft RFC (a Request for Comments document) right about that: "DNS Access Denied Error Page" that will allow showing a beautiful error page to the user without resorting to ugly solutions (like requiring a user to install an HTTPS certificate).
Now you need to configure DNS on every device. Wouldn’t it be better if you could just stick it up in your router? Some new routers already support DNS encryption natively. However, many old routers still accept only plain DNS servers. There is the Adaptive DNS Discovery working group that is committed to improving this situation.
Notable draft RFCs are:
Finally, one more thing can be improved. We need to be able to detect what app makes this or that DNS query. There’s no RFC that could help us, though. But we can try to do it ourselves, try to create some kind of "DNS fingerprints" at least for the popular apps.
If we knew what app makes a DNS query, we could be more flexible about what is blocked and what’s not.
Creating these DNS fingerprints requires analyzing the network behavior of every popular app and we’re working on a solution that will do it.
If you want to get access to the "better Internet", you should definitely use DNS in combination with VPN and ad blocker. And we suggest you take a closer look at AdGuard DNS.
So what is AdGuard DNS? It’s one of the most privacy-friendly DNS services that you can choose. It supports such reliable encryption protocols as DNS-over-HTTPS, DNS-over-TLS, and DNS-over-QUIC. It can identify requests to an ad, tracking, and/or adult domains (optionally), and responds with an empty response. AdGuard has its own base of domain names that serve ads, trackers, and fraud, and it is regularly updated. Furthermore, there’s a great opportunity to add your custom rules to blocklists.
To give you a broader view of what AdGuard DNS is, I will mention some more points:
The blocklists of AdGuard DNS are composed using special DNS filtering syntax. We weren’t happy with the limitations of HOSTS files type blocklists so for AdGuard DNS we used familiar adblock-style syntax. It was then gradually extended with DNS-specific modifiers.
Examples of DNS filtering rules. The full spec is available on Github.
Just to demonstrate the difference. On the left side, we have stats for the AdGuard DNS filter (the one that we use by default). It is relatively short, but it actually blocks about 900K domains. And on the left side, we have a different blocklist — Energized Ultimate which is a HOSTS file with 500K domains. And this is what it blocks — 500K domains.
Stats according to AdGuard DNS data
Each AdGuard DNS server node consists of several parts. On the front end, we have a DNS forwarder (written in Golang) that implements all the logic, i.e. DNS filtering, caching, etc.
On the backend, we have an unbound server instance that provides DNS recursion. A single server node load is from 5 000 to 40 000 responses depending on the node location (some locations are more popular than others).
DNS filtering engine implementations are open source and available on Github. There are two of them. The first is called urlfilter and it is a Golang library that we use in AdGuard DNS and AdGuard Home. Provides relatively high performance and low memory footprint. The second one is AdGuard DnsLibs, which is a C++ library that we use in our client apps. Besides DNS filtering, it also provides DNS encryption implementations.
If you want to protect yourself from ads and trackers with the help of AdGuard DNS, just follow this setup guide.
Browser-based content blockers might be the most popular among casual users, but not among the filter lists maintainers. There are way more hosts files and domains lists maintained by volunteers than the filter lists for browser-based blockers. This is a signal that it has great perspectives.
Stats according to filterlists.com
So, what are the prospects of DNS content blocking? First, the growth of DNS content blocking will accelerate. Windows 11 brings DNS encryption support, new routers also support DNS encryption, this all will make it easier for the users to set it up and to use it.
Circumvention of DNS content blocking rates will increase, but it will still be marginal. It takes a lot of effort from different parties to circumvent it and for that to start happening DNS content blocking should be as popular as browser blockers. And that probably won't happen, despite the growth rate.
DNS will not replace browser-based blockers for obvious reasons, the quality is simply not good enough. But still, it is here to stay.