Menu
EN

Partial AdGuard DNS outage on 29 November 2022

Yesterday, at 18:24 UTC AdGuard DNS experienced a serious outage that affected servers in 3 locations: Miami, New York, and London. During the outage the Internet was effectively broken for all customers connected to these 3 locations. This is about 20% of all AdGuard DNS customers, i.e. over 10 million people experienced issues with the Internet. We are truly sorry for this and we are taking steps to avoid having issues like that in the future.

AdGuard DNS outage November 29 2022

What happened

A number of small mistakes led to this issue. Each of these mistakes alone was not critical and would not lead to any disaster. Unfortunately, together they were just enough.

The first mistake was made two days ago when we renewed the TLS certificates for AdGuard DNS. The new certs were RSA and not ECDSA that we always use. The difference might not be obvious even for experienced developers, but it is very important for us. AdGuard DNS is written in Go and in addition to numerous advantages it also imposes some limitations. Initial TLS handshake with RSA is very slow in Go's crypto/tls and this issue is not going to be fixed any time soon. This performance issue mostly affects initial handshakes and not session resumptions so immediately after changing the certificates the CPU load increase was not too huge. But the bomb had been planted and waited for the best moment to detonate.

Note that plain unencrypted DNS is just about 10% of AdGuard DNS traffic. Most of the traffic is different encrypted protocols and performance-wise the heaviest part of any encryption is the handshake.

The wait was not too long. A day later we rolled out a change to the BGP configuration which was supposed to improve the traffic distribution for a couple of small ISPs. This new configuration shaked things pretty heavily, and the servers in Miami, New York, and London started getting much more traffic.

Read more about BGP and how we use and adjust it in AdGuard DNS.

This change tipped the scales and the RSA bomb has finally detonated, the servers weren't able to handle the increased load and started failing one by one. After 10 minutes we rolled back the routing changes to restore the traffic distribution. To our surprise this has not changed anything, the servers still could not cope with the load and the software was constantly killed & restarted.

We ran into a new problem. Now after a downtime all the DNS clients were trying to establish a new TLS connection and not just resume the existing session. This is exactly the case where RSA certs demonstrate the worst possible performance. On top of that, the increased load is created by millions of Android devices. We noticed that when Android's DoT reconnects to the server after a network change or after recovering from an error, it sends multiple "test" queries right away. These additional queries contribute to the server's load.

At first, we didn't realize where the root issue was. We wrongly thought it was a BGP misconfiguration since this was the thing that triggered the outage. In addition to that after several restarts and chaotic configuration changes we were able to recover London and New York. Reproducing these changes on Miami servers didn't help. This is when we calmed down and carefully inspected the software metrics and only then we identified the real cause of the disaster.

Incident timeline

  1. 18:24 UTC: Miami and New York servers go down due to the increased load.
  2. 18:45 UTC: BGP configuration has been rolled back but that doesn't help.
  3. 19:00 UTC: London servers also go down.
  4. 19:00—20:20 UTC: Chaotic configuration changes. London servers are going down without any obvious reason makes us think that the issue is BGP and maybe the routing is not restored.
  5. 20:20 UTC: London servers are recovered and operational again. At this point we don't understand what exactly has helped, but we try to reproduce the same configuration changes in other locations.
  6. 21:00 UTC: New York servers are recovered and operational. We finally realize that the root cause for the issue is the new RSA certs. We start changing the certs back everywhere.
  7. 21:30 UTC: While ECDSA certs are being prepared, we bring back Miami without DNS-over-TLS for the time being.
  8. 22:10 UTC: Miami is fully operational again.

Follow-up steps

In order to prevent this from happening again, we're taking both technical and organizational steps.

  1. We will be automatically checking the types of certificates and send an alert if the type is not ECDSA.
  2. Improve the deployments procedure. We could've noticed this issue earlier if we had analyzed the servers CPU graphs immediately after deploying the new certs. This should be done after any deployment even if the changes seem to be minuscule.
  3. Improve the internal incident response documentation. We are not happy with how the troubleshooting was conducted and it was possible to recover much, much faster.
Liked this post?
By downloading the comments you agree the terms and policies

AdGuard
for Windows

AdGuard for Windows is more than an ad blocker. It is a multipurpose tool that blocks ads, controls access to dangerous sites, speeds up page loading, and protects children from inappropriate content.
User Reviews: 18601
4.7 out of 5
By downloading the program you accept the terms of the License agreement
Read more

AdGuard
for Mac

AdGuard for Mac is a unique ad blocker designed with macOS in mind. In addition to protecting you from annoying ads in browsers and apps, it shields you from tracking, phishing, and fraud.
User Reviews: 18601
4.7 out of 5
By downloading the program you accept the terms of the License agreement
Read more

AdGuard
for Android

AdGuard for Android is a perfect solution for Android devices. Unlike most other ad blockers, AdGuard doesn't require root access and provides a wide range of app management options.
User Reviews: 18601
4.7 out of 5
By downloading the program you accept the terms of the License agreement

AdGuard
for iOS

The most advanced ad blocker for Safari: it makes you forget about pop-up ads, speeds up page loading, and protects your personal data. A manual element-blocking tool and highly customizable settings help you tailor the filtering to your exact needs.
User Reviews: 18601
4.7 out of 5
By downloading the program you accept the terms of the License agreement

AdGuard Browser extension

AdGuard is the fastest and most lightweight ad blocking extension that effectively blocks all types of ads on all web pages! Choose AdGuard for the browser you use and get ad-free, fast and safe browsing.
User Reviews: 18601
4.7 out of 5

AdGuard for Safari

Ad blocking extensions for Safari are having hard time since Apple started to force everyone to use the new SDK. AdGuard extension is supposed to bring back the high quality ad blocking back to Safari.
User Reviews: 18601
4.7 out of 5
Available on the
App Store
Download
By downloading the program you accept the terms of the License agreement

AdGuard Home

AdGuard Home is a network-wide software for blocking ads & tracking. After you set it up, it’ll cover ALL your home devices, and you don’t need any client-side software for that. With the rise of Internet-Of-Things and connected devices, it becomes more and more important to be able to control your whole network.
User Reviews: 18601
4.7 out of 5

AdGuard Content Blocker

AdGuard Content Blocker will eliminate all kinds of ads in mobile browsers that support content blocker technology — namely, Samsung Internet and Yandex.Browser. While being more limited than AdGuard for Android, it is free, easy to install and still provides high ad blocking quality.
User Reviews: 18601
4.7 out of 5
By downloading the program you accept the terms of the License agreement
Read more

AdGuard Assistant

A companion browser extension for AdGuard desktop apps. It offers an in-browser access to such features as custom element blocking, allowlisting a website or sending a report.
User Reviews: 18601
4.7 out of 5
Assistant for Chrome Is it your current browser?
Install
By downloading the program you accept the terms of the License agreement
Assistant for Firefox Is it your current browser?
Install
By downloading the program you accept the terms of the License agreement
Assistant for Edge Is it your current browser?
Install
By downloading the program you accept the terms of the License agreement
Assistant for Opera Is it your current browser?
Install
By downloading the program you accept the terms of the License agreement
Assistant for Yandex Is it your current browser?
Install
By downloading the program you accept the terms of the License agreement
Assistant for Safari Is it your current browser?
If you can't find your browser, try the old legacy Assistant version, which you can find in AdGuard extension settings.
Downloading AdGuard To install AdGuard, click the file indicated by the arrow Select "Open" and click "OK", then wait for the file to be downloaded. In the opened window, drag the AdGuard icon to the "Applications" folder. Thank you for choosing AdGuard! Select "Open" and click "OK", then wait for the file to be downloaded. In the opened window, click "Install". Thank you for choosing AdGuard!
Install AdGuard on your mobile device