High availability k3s with only two nodes

I’ve been running k3s at home for years now. I use it to drop random Raspberry PIs, old laptops, and VMs into my cluster to automatically pick up workloads and easily utilize their compute. But one thing that has always bothered me about it is the geography of it. All these nodes are in my own house and if the power goes out, or the internet goes down then it doesn’t matter how fancy the setup is, it’s offline. And that is no good for my users (me).

So the next logical step is to convince a friend with a server that they also need high availability and that they also have a geography problem. Surprisingly they agree, but then you realize for real geographical high availability, you need three sites because if you understand quorum systems particularly those around the data stores in Kubernetes (etcd, mariadb, postgres) then you know that they need (n/2)+1 nodes to achieve quorum. 1 2 3.

At this point you give up.

As homelabbers, we don’t give up when the technology doesn’t work for us. We over engineer a hacked together solution that costs us way more time and money than it’s worth.

The Third Site

Get a third friend. Well we don’t have a third friend, that would have made everything simpler right? But what can you do, you only know so many people willing to waste their time building servers and over-engineering services that you can get for free.

If you do have a third friend, add them to your cluster and stop here, you’ve achieved geographical high availability.

Cloud providers are always your friend.

But doesn’t that defeat the purpose of self hosting?

Well yes, and no. I managed to strike a balance by using a cloud provider as an “arbiter” for the k3s datastore, not a node itself. This keeps all the workloads self hosted, while providing the geographical diversity required to keep the cluster operation if either of our two self hosted nodes goes down. This has the added benefit of requiring such little compute that it falls in the free tier of most large VPS providers.

Let me explain.

The Arbiter

In my case the arbiter node lives on Google Cloud Platform (GCP) under a free tier e2-micro (2 vCPUs, 1 GB Memory) instance. The arbiter is a MariaDB MaxScale instance.

MaxScale provides two things critical to achieving high availability in this case:

Monitoring
Routing

Monitoring

MaxScale can act both as a monitor and as a proxy for database traffic. We will utilize both. On the VPS, we will install MaxScale as a monitor. It will monitor the status of each site’s datastore and determine which site is the master node, which is the replica, and automatically switch the replica if the master goes down.

MaxScale has a setting to automatically promote a replica to master if the master goes down.

auto_failover=true

maxscale.cnf.j2

But when the old master comes back from the depths of internet outage, it is out of sync so it needs to be rebuilt.
The monitor script detects these events and commands it to rebuild as a replica:

if [ $event = 'server_up' ] || [ $event = 'lost_slave' ]; then
   maxctrl call command mariadbmon async-rebuild-server MainMonitor $server_name
   echo "Server Up Event -- Rebuilding..."
fi

monitor_script.sh

Routing

The final piece of this puzzle is for each k3s master node to be configured to point to its local MaxScale instance, not the MariaDB instance itself. This is so that MaxScale will handle routing to the Master database, which may actually be on remote site. The local instance may be the replica.

MaxScale solve this simply with the configuration option:

router_options=master

The final setup looks something like this:

architecture-beta
    group siteA(cloud)[SiteA]
    group siteB(cloud)[SiteB]
    group siteC(cloud)[Arbiter]

    service max_a(database)[MaxScale] in siteA
    service db_a(database)[MariaDB] in siteA
    service k3s_master_a(server)[k3s Master] in siteA

    service max_b(database)[MaxScale] in siteB
    service db_b(database)[MariaDB] in siteB
    service k3s_master_b(server)[k3s Master] in siteB

    service max_arbiter(database)[MaxScale] in siteC

    max_arbiter:R -- L:max_a
    max_arbiter:L -- R:max_b
    max_a:R -- L:db_a
    max_a:B -- T:k3s_master_a

    max_b:L -- R:db_b
    max_b:B -- T:k3s_master_b

Notes

High Availability k3s