• Cloning Proxmox with LVM Thin Pools

    During Black Friday last year, I got tempted by a super good offer of a dedicated server in Kansas City with the option of connecting it to the Kansas City Internet Exchange (KCIX). Here are the specs:

    • Intel Xeon E5-2620 v4 (8 cores, 16 threads)
    • 64 GB DDR4 RAM
    • 500 GB SSD
    • 1 Gbps unmetered bandwidth

    It was such the perfect thing for AS200351 (if a bit overkill), so I just had to take it. I set it up during the winter holidays, having decided to install Proxmox to run a bunch of virtual machines, and all was well. Except for one thing—the disk.

    You see, the server came with a fancy SAN, with exactly 500 GiB of storage mounted over iSCSI via 10 Gbps Ethernet, backed by a highly reliable ZFS volume (zvol). While this all sounds good on paper, in practice I am barely able to hit over 200 MB/s when doing I/O, even at large block sizes. Nothing I did seemed to help, so I asked the provider to switch it to physical drives.

    Having configured Proxmox just the way I wanted it, I opted against reinstalling it from scratch, instead choosing to clone the disk. The provider suggested using Clonezilla, which should be able to do this sort of disk cloning very quickly. So we found an agreeable time, took the server down, and booted Clonezilla over PXE. All should be good, right?

    As it turns out, this ended up being a super painful experience.

    Editorial note: This story is based on my memory and incomplete console output. While the broad story is correct, the commands provided may not be correct.

    (Read more...)
  • 2023: Year in Review

    Last year, I started writing year-end reviews to look back upon the past year and reflect on what has happened. I thought I might as well continue to do the same this year.

    In January, I decided to finally create that stratum 1 NTP server that I had wanted ever since I heard about people doing it with Raspberry Pis. Instead of using Pis though, I ended up doing the ancient (but superior) approach of using a serial port. Along the way, I ran into various issues, but that tale is told in its own blog post.

    (Read more...)
  • On BGP Route Selection and High Availability via Anycast

    Earlier, we discussed how IP addresses and route authorizations work, before we took a break to talk about how the RIRs issue ASNs. As promised, I’ll now cover BGP route selection, how it enables anycasting, and how we can use it to achieve low latency and high availability. We’ll also cover some of the pitfalls of this approach and how it led to an infamous outage.

    For those not familiar with the concept, anycasting means the same IP address is shared by devices in multiple locations, with routers sending packets to the “nearest” location. This can result in latency lower than that is possible with the speed of light limitation from a single location1. Although, as you will see later, the routers’ concept of “nearest” may not necessarily be what we expect.

    Now, if one location stops announcing the IP address via BGP, then routers will select the next best location, enabling high availability as long as there is one location still available. Somewhat morbidly, I’ve claimed that this website will stay up even if Yellowstone erupts, which is theoretically true since my servers in Europe would still be able to serve traffic to the rest of the world even if every server in North America is down, but I’ve not tested this (and hope it will never be tested).

    Side note: AS200351 turns one year old today! 🎂

    (Read more...)
  • What I wish I knew when I got my ASN

    As you may know, I am currently writing a series on BGP and how the Internet works, from my perspective as the operator of a small autonomous system, AS200351. While we haven’t really exhausted the theoretical material, I think I’ve covered enough to enable readers to set up their own basic autonomous system. Rather than forcing you to do your own research based on outdated and potentially incorrect information on the Internet, or allowing you to fall victim to scams, I think it would be wise to talk about the process of getting your own ASN.

    For readers who haven’t read the previous parts of the series and are unfamiliar with why one might want an ASN, here’s a brief explanation:

    An autonomous system (AS) is a constituent part of the Internet that can define its own routing to the remainder of the Internet, and ASes exchange routes with each other over Border Gateway Protocol (BGP) to form the Internet itself. By receiving a globally unique identifier, an AS number (ASN), which in my case is 200351, I can exchange routes over BGP with other ASes, announce my own IP addresses to the Internet, and control how traffic flows in and out of my network, as opposed to simply exchanging traffic from a default gateway to reach the Internet with an IP address assigned by my ISP. This comes with several advantages, such as being able to switch upstream ISPs at will (or when such an ISP fails) without changing my IP addresses or breaking a single connection; or to advertise the same IP addresses from multiple locations (anycasting) to allow users to reach my services with lower latency than otherwise permissible by the speed of light with automatic failover.

    I will now share what I wish I knew when I impulsively decided to apply for an ASN at 3 a.m. on a cold December night last year, now that I’ve been doing this for a while. I’ll walk through the process as objectively and thoroughly as possible, demystifying the role of any player in this space. I would like you to go into this with full knowledge of the risks and a full understanding of where your money is going. In the end, I will offer some subjective suggestions on providers, but those can be ignored if you would rather do your own research.

    (Read more...)
  • Diving into IP Addresses and Route Authorization

    In the first part of this series, we had a brief overview of what BGP is. Then last time, we dived into what autonomous systems (ASes) are and the relationships that can exist between them, as well as the existence of Tier 1 networks and Internet Exchange Points (IXPs). That provided a broad overview of the Internet’s structure.

    However, so far in this series, we’ve talked about IP addresses—or really, IP prefixes and CIDRs—as if they are something that just exists. This mental model is no longer sufficient. Before we can truly understand routing, we must first understand how IP addresses really work—how they are assigned and who is authorized to announce them.

    (Read more...)
  • Diving into Autonomous Systems of the Internet

    Last time, we introduced BGP as a protocol, but introduced the concept of autonomous systems (AS for short) and gave some examples. This time, we’ll dive deeper into the concept, which hopefully will give you a better understanding of the various types of ASes and how the Internet is organized on a global level.

    But first, let’s start with how the numerical identifiers—the autonomous system numbers (ASNs)—work.

    (Read more...)
  • An Introduction to BGP... from the operator of a small AS

    Border Gateway Protocol (often abbreviated BGP) is a critical protocol that makes the modern Internet possible, yet remains one of its most poorly understood parts even among its long-time users. At the same time, it has played a significant role in several high-profile outages on the Internet. As someone who has been running my “own piece of the Internet”—AS200351—for half a year now, I think the time has come to write a piece explaining exactly what BGP is, what AS200351 is, and how the Internet truly functions behind the scenes. We’ll start with the basics.

    To understand BGP, we must first understand why it is called the “Internet” in the first place. To simplify greatly, the Internet is called that because it’s an interconnected network of networks (more precisely, autonomous systems, or ASes) glued together by BGP. Since this sounds like a nonsensical sequence of words, let’s dive a bit deeper.

    (Read more...)
  • Introducing my own mirroring service: mirror.quantum5.ca

    In January, I upgraded my home Internet connection to 3 Gbps symmetric, because, strangely enough, it was cheaper than the package I already had at the time (1500 Mbps down, 940 Mbps up). This was connected to the second port on my ConnectX-3, allowing my home server to achieve the full speed where 2.5 Gbps Ethernet would have failed. Unfortunately, nothing I was doing could have harnessed the full speed of this Internet connection, or anywhere near it, so I started thinking…

    In February, I realized that I could run a mirroring service for open-source software to serve the community at basically no additional cost—I am already paying for this 3 Gbps Internet connection and I have some spare disk space on my SSD. So I decided to do exactly that.

    Today, I am happy to announce that this mirror, mirror.quantum5.ca, has been tested for a few months and is fully ready for production. If you find the service helpful, please feel free to support me via GitHub Sponsors, Ko-fi, Liberapay, or directly with credit card or bank through Stripe (CAD), though this is of course strictly optional.

    If you are interested in how it’s all set up, please read on:

    (Read more...)
  • Microsecond Accurate Time Synchronization on LAN with PTP

    Last time, I built a stratum 1 NTP server with a PPS signal from a GPS receiver, synchronizing my server’s clock to within 10 microseconds of UTC. However, NTP was designed to synchronize clocks within a few tens of milliseconds over the Internet, and I’d be lucky to achieve millisecond accuracy on a LAN. I mentioned that PTP was the alternative that could achieve accuracy in the sub-microsecond range. Well, this time I’ll be setting up PTP between my server and my PC with the hardware timestamping on the ConnectX-3s.

    If you are following along at home, don’t despair if your hardware can’t do timestamping or PTP. I will also attempt to set up PTP with software timestamping later for my other devices.

    Naturally, I first turned to the gpsd documentation, since that was a decent reference for setting up NTP with the PPS signal. Well, this is what it says for PTP with hardware timestamping:

    Sadly, theory and practice diverge here. I have never succeeded in making hardware timestamping work. I have successfully trashed my host system clock. Tread carefully. If you make progress please pass on some clue.

    That didn’t sound encouraging at all. “Oh well, I guess I am on my own here,” I thought to myself. “How bad could digging through a few man pages and random online documentation be? Worst case, there is the source code, right?”

    (Read more...)
  • DIY a Stratum 1 NTP Server with a Serial Port

    These days, it seems like everyone is posting about turning Raspberry Pis into a stratum 1 NTP server by hooking up a cheap GPS module, most often the GT-U7 u-blox 7 clone with a PPS (pulse-per-second) signal output, whose rising edge indicates exactly the start of a second.

    While this seems like a cool idea, it suffers from one flaw—while the Raspberry Pi itself almost certainly has very accurate time, getting accurate time to the rest of the network would be problematic. This is because the Ethernet adapter on Raspberry Pis before the Pi 4 was hooked up via USB, and the polling nature of USB introduces jitter, preventing the accurate signal from reaching the rest of the network. Unfortunately, I only have a Raspberry Pi 3 model B in my possession, which suffers from the problem.

    Now, I could have gotten a Raspberry Pi 4, but those aren’t priced sanely at the moment and it would be just an exercise in copying. Instead, I looked at the various alternatives. The traditional way of doing this kind of thing involves hooking up a GPS receiver into a serial port, which generates an interrupt. If the PPS signal is delivered to the DCD (data carrier detect) signal (as described in RFC 2783), then the in-tree Linux driver pps_ldisc is able to do the timestamping in kernel mode for the highest possible accuracy.

    I found out that my server’s X570 motherboard came with a serial port header (labelled COM). This meant that I could buy some fancy GPS receiver with a serial port and hook it up. Unfortunately, those aren’t priced sanely either, so I decided to build my own with the GT-U7 module and a driver module for RS-232 (the common serial port standard).

    This was late last year. I ordered the components on AliExpress and they all arrived in January, so I finally started this project.

    (Read more...)