Category: High availability
-
Enabling highly available global anycast DNS modifications with Galera
(Read more...)Last time, we set up a global anycast PowerDNS cluster with geolocation and health checks. This enables it to provide high availability for other services, always sending the user to a node that’s up.
However, there was one flaw—there is a single master node, which is the single point of failure for writes. When it goes down, it becomes impossible to make changes to the DNS, even though the “slave”1 nodes in the anycast will continue to serve the zone, including performing availability checks.
Last time, I also hinted at MariaDB Galera clusters becoming important. This time, we’ll leverage Galera, a multi-master solution for MariaDB, to eliminate the dependency on the single master. We’ll also make poweradmin highly available so that we can edit DNS records through a nice UI, instead of running API requests through
curlor executing SQL queries directly.Without further ado, let’s dive in.
-
Building highly available services: global anycast PowerDNS cluster
(Read more...)As I’ve written about before, this blog has multiple geographically distributed backend servers serving the content, with my anycast PowerDNS cluster selecting the geographically closest backend server that’s up and returning it to the user.
Due to various outages I’ve experienced recently, I’ve been thinking a lot more about making my self-hosted services highly available (HA), staying up even if a few servers go down. This is mostly for the sake of my sanity, so that I could just shrug if a server goes down and wait for the provider to bring it back up, instead of panicking. Of course, the added availability also helps, but it’s probably a bigger concern in the enterprise space than it is for hobbyists. As a bonus, if you have nodes spread out across multiple locations, you can also route the user to the geographically closest one for lower latency and faster response times.
Either way, I thought it was time to start a series about building highly available services. We begin with the most important building block—DNS, which is basically required to make any other service highly available.
The stack I’ve chosen for this is MariaDB and PowerDNS, mostly because these are fairly easy to set up and I already have experience with them. Many other alternative tech stacks are probably equally viable, but that’s left as an exercise for the reader. The general idea should apply anyway. Note that anycast isn’t really required, since you can still follow along and deploy two unicast DNS servers for redundancy.
Without further ado, let’s dive in.