Posts from 2025

  • A whirlwind tour of systemd-nspawn containers

    Mar 22, 2025
    Sysadmin
    17 minutes
    qt.ax/snc

    In the last yearly update, I talked about isolating my self-hosted LLMs running in Ollama as well as Open WebUI in systemd-nspawn containers and promised a blog post about it. However, while writing that blog post, a footnote on why I am using it instead of Docker accidentally turned into a full blog post on its own. Here’s the actual post on systemd-nspawn.

    Fundamentally, systemd-nspawn is a lightweight Linux namespaces-based container technology, not dissimilar to Docker. The difference is mostly in image management—instead of describing how to build images with Dockerfiles and distributing prebuilt, read-only images containing ready-to-run software, systemd-nspawn is typically used with a writable root filesystem, functioning more similarly to a virtual machine. For those of you who remember using chroot to run software on a different Linux distro, it can also be described as chroot on steroids.

    I find systemd-nspawn especially useful in the following scenarios:

    1. When you want to run some software with some degree of isolation on a VPS, where you can’t create a full virtual machine due to nested virtualization not being available1;
    2. When you need to share access to hardware, such as a GPU (which is why I run LLMs in systemd-nspawn);
    3. When you don’t want the overhead of virtualization;
    4. When you want to directly access some files on the host system without resorting to virtiofs; and
    5. When you would normally use Docker but can’t or don’t want to. For reasons, please see the footnote-turned-blog post.

    In this post, I’ll describe the process of setting up systemd-nspawn containers and how to use them in some common scenarios.

    (Read more...)

  • Docker considered harmful

    Mar 18, 2025
    Sysadmin
    9 minutes
    qt.ax/dch

    In the last yearly update, I talked about isolating my self-hosted LLMs running in Ollama, as well as Open WebUI, in systemd-nspawn containers. However, as I contemplated writing such a blog post, I realized the inevitable question would be: why not run it in Docker?

    After all, Docker is super popular in self-hosting circles for its “convenience” and “security.” There’s a vast repository of images that exist for almost any software you might want. You could run almost anything you want with a simple docker run, and it’ll run securely in a container. What isn’t there to like?

    This is probably going to be one of my most controversial blog posts, but the truth is that over the past decade, I’ve run into so many issues with Docker that I’ve simply had enough of it. I now avoid Docker like the plague. In fact, if some software is only available as a Docker container—or worse, requires Docker compose—I sigh and create a full VM to lock away the madness.

    This may seem extreme, but fundamentally, this boils down to several things:

    1. The Docker daemon’s complete overreach;
    2. Docker’s lack of UID isolation by default;
    3. Docker’s lack of init by default; and
    4. The quality of Docker images.

    Let’s dive into this.

    (Read more...)

  • On ECC RAM on AMD Ryzen

    Feb 17, 2025
    Sysadmin
    9 minutes
    qt.ax/ecc

    Last time, I talked about how a bad stick of RAM drove me into buying ECC RAM for my Ryzen 9 3900X home server build—mostly that ECC would have been able to detect that something was wrong with the RAM and also correct for single-bit errors, which would have saved me a ton of headache.

    Now that I’ve received the RAM and ran it for a while, I’ll write about the entire experience of getting the RAM working and my attempts to cause errors to verify the ECC functionality.

    Spoilers: Injecting faults was way harder than it appeared from online research.

    (Read more...)