Installing Debian (and Proxmox) by hand from a rescue environment
Normally, installing Debian is a simple process: you boot from the installer CD
image and follow the menu options in debian-installer. Simple, right? Or even
easier, just use the Debian image provided by your server vendor, since Debian
is quite popular and an image is bound to be available. Given the simplicity of
this, you might have idly wondered: what’s actually going on behind
debian-installer’s pretty menus? Well, you are about to find out.
You see, recently, I got this cheap headless dedicated server without IPMI1—really, just an Intel N100 mini PC. To cut costs, there was no video feed, as that would require separate hardware to receive and stream the screen. Instead, there’s only the ability to power cycle and boot from PXE, which is used to perform a variety of tasks, such as booting rescue CDs or performing automated installation of operating systems. This shouldn’t be a problem for my use case, since there is a Proxmox 8 image right there, and I just set it to install automatically.
Of course that didn’t work, because I wouldn’t be writing about it if it did! As it turns out, the Proxmox 8 image (and also the Debian 12 image) didn’t have the firmware for the Realtek NICs on the mini PC, which prevented them from working. I thought that I just needed to install the firmware package, but when I booted into the included Finnix rescue system, it appeared that Debian wasn’t installed at all! Clearly, the PXE installer failed to start due to the missing firmware.
What now? Well, I’ve already done some pretty sketchy Debian installs in the past, so I thought I might as well just go all out and install a full Debian system through the rescue system. Unlike last time though, I’ll do a complete clean install, instead of keeping the partition scheme.
Step 0: The setting
I booted by PXE into a Finnix 125 rescue system on an Intel N100 mini PC, which I was accessing via SSH.
I ran lsblk to find that the main system drive is /dev/sda. Ubuntu was
installed on an LVM volume group called ubuntu-vg, though that doesn’t really
matter given that we are about to wipe the entire disk. The drive was using the
GPT partition scheme and it seems to be doing UEFI booting. This determines what
we will do when partitioning, and later, when installing the bootloader.
For the purposes of this post, I’ve redacted the actual IP address of the server
in question. Instead, we pretend the server is running on the IPv4 address
192.0.2.5 with the gateway 192.0.2.1.
Step 1: Partitioning
We start by running gdisk /dev/sda and to see the existing partition scheme:
root@0:~# gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.9
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): p
Disk /dev/sda: 1000215216 sectors, 476.9 GiB
...
Number Start (sector) End (sector) Size Code Name
1 2048 2000895 976.0 MiB 8300
2 2000896 3076095 525.0 MiB EF00
3 3076096 1000214527 475.5 GiB 8E00
As you can see, there are three partitions:
- a 976 MiB
/bootpartition; - a 525 MiB EFI system partition; and
- the rest of the space is partitioned into LVM.
Since we are going to start completely from scratch, we’ll delete all these partitions:
Command (? for help): d
Partition number (1-3): 1
Command (? for help): d
Partition number (2-3): 2
Command (? for help): d
Using 3
You can use whatever partition scheme you want, but this is what I ended up using:
- a 512 MiB EFI system partition to install the bootloader; and
- the rest of the disk is used for LVM, on which I would put the root filesystem and create a thin pool for all the Proxmox VMs.
I decided against a separate /boot partition since grub2 is perfectly
capable of loading everything from LVM and I don’t want to waste space.
Time to create them:
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-1000215182, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-1000215182, default = 1000214527) or {+-}size{KMGTP}: +512M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): EF00
Changed type of partition to 'EFI system partition'
Command (? for help): n
Partition number (2-128, default 2):
First sector (34-1000215182, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-1000215182, default = 1000214527) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8e00
Changed type of partition to 'Linux LVM'
Command (? for help): p
Disk /dev/sda: 1000215216 sectors, 476.9 GiB
...
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 1000214527 476.4 GiB 8E00 Linux LVM
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Why is the kernel using the old partition table? Oh right, because the old
Ubuntu install had an LVM volume group. Let’s unload it so that partprobe can
delete the device:
# vgchange -a n ubuntu-vg
# partprobe
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sda 8:0 0 476.9G 0 disk
├─sda1 8:1 0 512M 0 part
└─sda2 8:2 0 476.4G 0 part
Now, the new partitions are loaded, so it’s time to create the LVM. I randomly
decided to use 20 GiB to the root filesystem, which is fine for now. I can
always expand it later, which is the point of LVM. I chose to use the name
debian for the volume group, but you could of course call it anything you’d
like. Just remember to change subsequent commands.
# vgcreate debian /dev/sda2
Physical volume "/dev/sda2" successfully created.
Volume group "debian" successfully created
# lvcreate -n root -L 20G debian
Logical volume "root" created.
Of course, just creating the partitions isn’t enough—we’ll need to create
filesystems on them. The EFI system partition has to be FAT32, which we create
with mkfs.vfat. For the root filesystem, I just decided to use the traditional
ext4:
# mkfs.vfat /dev/sda1
mkfs.fat 4.2 (2021-01-31)
# mkfs.ext4 /dev/debian/root
mke2fs 1.46.6 (1-Feb-2023)
Discarding device blocks: done
Creating filesystem with 5242880 4k blocks and 1310720 inodes
Filesystem UUID: ec47d17e-42cd-4a12-b65a-dc89d7344d64
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
# mount /dev/debian/root /mnt
Now, the new root filesystem is available on /mnt.
Step 2: debootstrap onto the new root filesystem
We naturally use our good friend debootstrap to install the base Debian
system. Feel free to change bookworm to whatever version you prefer. You can
use this to install most Debian-based distros too by replacing the repository
URL.
# debootstrap --arch amd64 bookworm /mnt http://deb.debian.org/debian
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id A7236886F3CCCAAD148A27F80E98404D386FA1D9)
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
...
I: Configuring libc-bin...
I: Base system installed successfully.
At this point, the base Debian system is ready. However, the system isn’t
bootable. Hell, it’s missing a kernel! Let’s chroot into the new system to
install more stuff:
# for dir in /proc /sys /sys/firmware/efi/efivars /dev /dev/pts; do
> mount --bind "$dir" "/mnt$dir"
> done
# chroot /mnt /bin/bash
root@finnix:/#
Now we are in the chroot. The prompt, confusingly, is root@finnix:/ because
that’s the system hostname, but for the rest of this post, I’ll use debian# as
the prompt to make it obvious.
Step 3: Setting up Debian in a chroot
First, we need to configure the package manager, APT. Currently, we have the
very basic /etc/apt/sources.list generated by debootstrap:
deb http://deb.debian.org/debian bookworm main
This wouldn’t do. We don’t even have security updates, so any further package
installation would result in potentially vulnerable packages being installed.
Instead, replace the file with something debian-installer might generate:
deb http://deb.debian.org/debian bookworm main non-free-firmware
deb-src http://deb.debian.org/debian bookworm main non-free-firmware
deb http://security.debian.org/debian-security bookworm-security main non-free-firmware
deb-src http://security.debian.org/debian-security bookworm-security main non-free-firmware
# bookworm-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
deb http://deb.debian.org/debian bookworm-updates main non-free-firmware
deb-src http://deb.debian.org/debian bookworm-updates main non-free-firmware
Now, let’s update the repositories and install updates:
debian# apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
...
8 packages can be upgraded. Run 'apt list --upgradable' to see them.
debian# apt upgrade
...
Do you want to continue? [Y/n]
...
Now, let’s install the packages we need:
debian# apt install linux-image-amd64 grub-efi openssh-server lvm2 wget
...
After this operation, 506 MB of additional disk space will be used.
Do you want to continue? [Y/n]
...
We naturally install the kernel and the UEFI version of Grub so that the system
can boot. We also install the OpenSSH server to ensure we can access it
headlessly, lvm2 to allow Debian to work while it’s installed on LVM, and
wget to be able to download files later.
Installing missing firmware
While the installation completes, we see warnings like:
...
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168g-3.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168g-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8106e-2.fw for module r8169
...
This is the root cause of the server not working with the images, so let’s install the firmware too:
debian# apt install firmware-realtek
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
firmware-realtek
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 1491 kB of archives.
After this operation, 7046 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/non-free-firmware amd64 firmware-realtek all 20230210-5 [1491 kB]
Fetched 1491 kB in 0s (49.7 MB/s)
Selecting previously unselected package firmware-realtek.
(Reading database ... 23512 files and directories currently installed.)
Preparing to unpack .../firmware-realtek_20230210-5_all.deb ...
Unpacking firmware-realtek (20230210-5) ...
Setting up firmware-realtek (20230210-5) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.1.0-23-amd64
This time, there are no warnings.
Step 4: Configuring system
For the system to function properly, several configuration steps are required.
/etc/network/interfaces
First, you need to configure network connectivity, or you won’t be able to SSH
in after it boots. This means creating /etc/network/interfaces. You should run
ip a to see how the network is currently configured:
debian# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether dc:13:f9:98:05:c8 brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether dc:13:f9:98:05:c9 brd ff:ff:ff:ff:ff:ff
inet 192.0.2.5/24 metric 1024 brd 192.0.2.255 scope global dynamic enp3s0
valid_lft 1208684sec preferred_lft 1208684sec
inet6 fe80::de13:f9ff:fe98:05c9/64 scope link
valid_lft forever preferred_lft forever
4: wlp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 34:3e:0e:fd:96:33 brd ff:ff:ff:ff:ff:ff
In this case, I opted to just use static IPs, resulting in the following
/etc/network/interfaces:
auto lo
iface lo inet loopback
allow-hotplug enp3s0
iface enp3s0 inet static
address 192.0.2.5/24
gateway 192.0.2.1
You can try something like iface enp3s0 inet dhcp if you want to use DHCP to
configure IPs.
/etc/fstab
The next thing to do is configure /etc/fstab, so that after the system boots,
all the partitions get mounted. I decided to use the following:
UUID=8F1E-2F2E /boot/efi vfat errors=remount-ro 0 0
/dev/debian/root / ext4 errors=remount-ro 0 1
Run blkid to get the UUIDs for the partitions, e.g. for the EFI system
partition. Since I am using LVM, I just used the LVM name for the root
partition, but if you aren’t, I would highly recommend that you use the UUID for
it also.
root user password and SSH keys
Finally, to ensure you won’t be locked out of your new system, you should configure the password and SSH keys for the root user.
Setting a password is simple:
debian# passwd
New password:
Retype new password:
passwd: password updated successfully
However, by default, Debian won’t let root login over SSH with a password, and
it’s good practice to use SSH keys for login anyway, so let’s just download SSH
keys:
debian# wget -O /root/.ssh/authorized_keys https://github.com/quantum5.keys
--2024-08-11 20:42:44-- https://github.com/quantum5.keys
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 162 [text/plain]
Saving to: ‘/root/.ssh/authorized_keys’
/root/.ssh/authorized_keys 100%[=============================================>] 162 --.-KB/s in 0s
2024-08-11 20:42:44 (3.74 MB/s) - ‘/root/.ssh/authorized_keys’ saved [162/162]
(Naturally, you should not use my keys, unless you want to give me your server for free, in which case I appreciate the gesture.)
Step 5: Install bootloader
With everything already configured, this is easy:
debian# mkdir -p /boot/efi
debian# mount /dev/sda1 /boot/efi
debian# grub-install /dev/sda
Installing for x86_64-efi platform.
Installation finished. No error reported.
debian# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.1.0-23-amd64
Found initrd image: /boot/initrd.img-6.1.0-23-amd64
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
This should install grub and add Debian to the UEFI firmware as a boot option.
However, if any previous operating systems are installed, they might create weird boot entries that would render the system unbootable. Let’s check for it:
debian# efibootmgr
BootCurrent: 0002
Timeout: 1 seconds
BootOrder: 0001,0002,0003,0004,0006,0000,0005
Boot0000 Windows Boot Manager
Boot0001* UEFI: PXE IPv4 Realtek PCIe GBE Family Controller
Boot0002* UEFI: PXE IPv4 Realtek PCIe GBE Family Controller
Boot0003* UEFI: PXE IPv6 Realtek PCIe GBE Family Controller
Boot0004* UEFI: PXE IPv6 Realtek PCIe GBE Family Controller
Boot0005 ubuntu
Boot0006* debian
Oh no, it seems like Ubuntu is still around. Let’s kill it with fire:
debian# efibootmgr -B -b 0005
BootCurrent: 0002
Timeout: 1 seconds
BootOrder: 0001,0002,0003,0004,0006,0000
Boot0000 Windows Boot Manager
Boot0001* UEFI: PXE IPv4 Realtek PCIe GBE Family Controller
Boot0002* UEFI: PXE IPv4 Realtek PCIe GBE Family Controller
Boot0003* UEFI: PXE IPv6 Realtek PCIe GBE Family Controller
Boot0004* UEFI: PXE IPv6 Realtek PCIe GBE Family Controller
Boot0006* debian
That’s much better. I am leaving the PXE boot options as is, since the provider relies on those to implement stuff like rescue booting. But let’s reboot straight into Debian:
debian# efibootmgr -n 0006
BootNext: 0006
...
debian# exit
# reboot
Note that you can’t reboot directly from a chroot, so we exit it and run
reboot on Finnix.
After a few minutes, the server is up:
$ ssh 192.0.2.5
Linux finnix 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Aug 10 04:12:09 2024 from 142.198.98.109
root@finnix:~#
As you see, Debian is indeed up, even if the hostname is finnix. You can
change it in /etc/hostname and then run hostname -F /etc/hostname.
If Debian is what you are after, then you are done here.
Step 6: Installing Proxmox (optional)
This basically requires you to follow the official instructions for installing Proxmox on Debian 12. I’ll quickly summarize what I did:
First, I changed the hostname to proxmox, and then set up /etc/hosts so that
the server’s public IP resolves to the desired FQDN:
root@proxmox:~# cat /etc/hostname
proxmox
root@proxmox:~# cat /etc/hosts
127.0.0.1 localhost
192.0.2.5 proxmox.example.com proxmox
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
root@proxmox:~# hostname --ip-address
192.0.2.5
Then, we follow Proxmox’s instructions to add its APT repositories;
root@proxmox:~# echo "deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
root@proxmox:~# wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
--2024-08-10 04:15:06-- https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg
Resolving enterprise.proxmox.com (enterprise.proxmox.com)... 212.224.123.70, 2a01:7e0:0:424::249
Connecting to enterprise.proxmox.com (enterprise.proxmox.com)|212.224.123.70|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1187 (1.2K) [application/octet-stream]
Saving to: '/etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg'
/etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg 100%[=============================================>] 162 --.-KB/s in 0s
2024-08-10 04:15:07 (10.8 MB/s) - '/etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg' saved [1187/1187]
root@proxmox:~# apt update && apt full-upgrade
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
...
10 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
grub-common grub-efi grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub2-common shim-helpers-amd64-signed shim-signed shim-signed-common shim-unsigned
10 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 7288 kB of archives.
After this operation, 24.6 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-efi amd64 2.06-13+pmx2 [2376 B]
...
Installation finished. No error reported.
No DKMS packages installed: not changing Secure Boot validation state.
Then, we install Proxmox’s kernel and reboot:
root@proxmox:~# apt install proxmox-default-kernel
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
proxmox-kernel-6.8 proxmox-kernel-6.8.12-1-pve-signed pve-firmware
Suggested packages:
linux-image
The following packages will be REMOVED:
firmware-linux-free firmware-realtek
The following NEW packages will be installed:
proxmox-default-kernel proxmox-kernel-6.8 proxmox-kernel-6.8.12-1-pve-signed pve-firmware
0 upgraded, 4 newly installed, 2 to remove and 0 not upgraded.
Need to get 233 MB of archives.
After this operation, 860 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-firmware all 3.13-1 [134 MB]
...
Adding boot menu entry for UEFI Firmware Settings ...
done
Setting up proxmox-kernel-6.8 (6.8.12-1) ...
Setting up proxmox-default-kernel (1.1.0) ...
root@proxmox:~# systemctl reboot
Then after rebooting, we install Proxmox’s packages:
root@proxmox:~# apt install proxmox-ve postfix open-iscsi chrony
During the installation of postfix, the following screen appears:
┌───────────────────────────┤ Postfix Configuration ├───────────────────────────┐
│ Please select the mail server configuration type that best meets your needs. │
│ │
│ No configuration: │
│ Should be chosen to leave the current configuration unchanged. │
│ Internet site: │
│ Mail is sent and received directly using SMTP. │
│ Internet with smarthost: │
│ Mail is received directly using SMTP or by running a utility such │
│ as fetchmail. Outgoing mail is sent using a smarthost. │
│ Satellite system: │
│ All mail is sent to another machine, called a 'smarthost', for │
│ delivery. │
│ Local only: │
│ The only delivered mail is the mail for local users. There is no │
│ network. │
│ │
│ General mail configuration type: │
│ │
│ No configuration │
│ Internet Site │
│ Internet with smarthost │
│ Satellite system │
│ Local only │
│ │
│ │
│ <Ok> <Cancel> │
│ │
└───────────────────────────────────────────────────────────────────────────────┘
Select Local only if you don’t know what to do, and then everything should
just work afterwards. You should be able to go to https://192.0.2.5:8006 to
access the Proxmox web console.
Finally, to clean stuff up, you should remove the Debian stock kernel and
os-prober:
root@proxmox:~# apt remove linux-image-amd64 'linux-image-6.1*' os-prober
The following packages will be REMOVED:
linux-image-6.1.0-23-amd64 linux-image-amd64 os-prober
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 408 MB disk space will be freed.
Do you want to continue? [Y/n]
...
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
There, Proxmox is now installed.
Conclusion
As you can see, it’s not that difficult to install Debian (or Proxmox)
completely manually, without using a prebuilt image or debian-installer. I
hope the instructions and explanations here prove useful to you when you run
into a similar situation. Thank you for reading.
Notes
-
Those of you who have read the previous post on cloning Proxmox with LVM thin pools would remember how much we managed to accomplish with IPMI on the server despite all the curveballs that Clonezilla threw at us, including loading firmware from a virtual CD drive. This time, we have none of this, and boy did it suck. ↩