While building my Traefik reverse proxy (Hostnames and Let’s Encrypt for My Homelab with Traefik), I discovered my Proxmox host was still on an old 8.x release. Proxmox VE 8 reached end of life in August 2026, so the hypervisor pieces (kernel, QEMU, LXC tooling) had stopped getting fixes. It had also just bitten me: Docker refused to run inside an LXC container on that version, which is why Traefik ended up as a native binary.

This post documents the in-place upgrade to Proxmox VE 9.2: the prep, the checker findings, how I avoided typing long commands at the server’s console, the two scary-looking errors that turned out to be harmless, how I verified everything afterwards, and a follow-up fix: converting my Twingate connector from a privileged to an unprivileged container.

This post is anonymized. The domain is example.com and the IP addresses are illustrative, not my real ones.

My setup, and why it mattered

• Single Proxmox node, booting with BIOS/GRUB, on the free no-subscription repository
• Open vSwitch networking with one physical NIC (enp0s31f6, the onboard Intel port)
• pfSense VM running my VLANs (an internal network plus pen-testing ranges)
• Pi-hole (my primary DNS), Traefik, Kimai (time tracking), Home Assistant and Calibre, all as guests on this one box
• Twingate for remote access
• A Synology NAS mounted in Proxmox over NFS

Because so much depends on this one host, I wrote down the risks before touching anything:

Phase 1: Back up everything

Proxmox has a built-in backup tool (vzdump). For a one-off “back up everything”, the easiest route is a backup job you run by hand.

Choosing the target

pvesm status

My options were the root disk (local), two LVM-thin pools, and the Synology over NFS. The NAS was the clear choice:

• LVM-thin pools can’t store backups at all. They only hold VM disks and container volumes, not backup files.
• My guests live on the big thin pool. A backup on the same disk as the originals protects against very little.
• The NAS survives a full rebuild. In the worst case (reinstall the host), recovery is: reinstall, re-add the NFS share, restore.

Check that the target accepts backups:

pvesm status --content backup

Creating and running the job

Datacenter → Backup → Add:

Then select the job and click Run now. The CLI equivalent:

vzdump --all --mode stop --compress zstd --storage NAS-STORE --notes-template '{{guestname}} pre-PVE9'

About 465 GB of guest data took 40 minutes over gigabit, much faster than I expected. vzdump skips empty space quickly and ZSTD roughly halves the rest (one 40 GB VM became an 18.7 GB archive). Each guest is only down during its own turn, so the outages were a few minutes each.

Checking the backups

1. The task log ends with Backup job finished successfully and TASK OK. If any guest fails, it says “finished with errors” and TASK ERROR instead.
2. Protect them. In the NAS storage’s Backups view, select each new backup and click Change Protection, so nothing can prune or delete them.
3. Test-restore the one that matters most (Kimai, for me): Restore to an unused VM ID with Unique ticked (new MAC address) and Start after restore unticked, so it can’t clash with the live copy. When it finishes with TASK OK, delete the test copy.

Phase 2: Update to the latest 8.4, then reboot

The upgrade to 9 requires the latest 8.4 packages first:

apt update && apt dist-upgrade
pveversion

pveversion showed 8.4.x, but the running kernel was still the old one. The update had installed a newer 6.8 kernel that wasn’t in use yet. I rebooted before going further, so I’d know the box boots cleanly on fully updated 8.4. If anything broke later, it would be the 9 upgrade’s fault, not this update’s.

Phase 3: The pre-upgrade checker

Proxmox ships a checker that only reports, and changes nothing:

pve8to9 --full

Mine found one failure and two warnings.

FAIL: Resolved node IP '192.168.1.164' not configured

/etc/hosts still mapped the hostname proxmox to an address from an old network layout. Proxmox’s cluster filesystem expects the hostname to resolve to an IP the host actually has; a stale entry can stop /etc/pve from mounting after a reboot. The fix was one line:

cp /etc/hosts /etc/hosts.bak
nano /etc/hosts      # change 192.168.1.164 to the host's real IP, keep the names
hostname --ip-address

WARN: running guests

Expected. I shut them down on upgrade day.

WARN: intel-microcode not found

Worth fixing. I added the non-free-firmware component to the new Debian sources during the upgrade and installed the package before the reboot.

Two more things I checked by hand

• Repositories. cat /etc/apt/sources.list.d/*.list showed four files, but only the no-subscription Proxmox repo was enabled. The Ceph, enterprise and pvetest lines were all commented out. No Ceph means no Ceph repo is needed on 9.
• Network config. cat /etc/network/interfaces and ip -br link confirmed a single physical NIC. The only bridge with a physical port was the main one; the pfSense internal switch has no physical port at all. So a NIC rename would mean fixing one name in two lines.

Phase 4: Avoiding typos at the console

I’d been pasting commands into the web UI’s Shell, which Proxmox says not to use during the upgrade, and I didn’t want to retype long commands on the server’s keyboard. The solution:

1. Run the upgrade over SSH from my Mac, inside tmux. Copy-paste works as usual. If the network drops, the upgrade keeps running on the server, and I can reattach from the Mac or from the console with tmux attach -t upgrade.
2. Save the long commands on the server as a script beforehand, so the most I’d type at the console is a short command.

Install tmux while still on the 8.4 repositories:

apt install -y tmux

The repository-switch script

Pasting this only creates the file. It has a guard against running twice, and it stops if any Debian 12 (bookworm) repository is still listed:

cat > /root/pve9-repos.sh << 'SCRIPT'
#!/bin/bash
# Switch apt repositories from PVE 8 (bookworm) to PVE 9 (trixie)
set -e

if grep -q trixie /etc/apt/sources.list; then
  echo "sources.list already mentions trixie - has this already run? Stopping."
  exit 1
fi

echo "== Backing up /etc/apt to /root/apt-backup-pve8"
cp -a /etc/apt /root/apt-backup-pve8

echo "== Stopping audit-log noise"
systemctl disable --now systemd-journald-audit.socket || true

echo "== Debian base: bookworm -> trixie, adding non-free-firmware"
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
sed -i 's/main contrib$/main contrib non-free-firmware/' /etc/apt/sources.list

echo "== Adding Proxmox VE 9 no-subscription repo"
cat > /etc/apt/sources.list.d/proxmox.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

echo "== Moving old bookworm .list files to /root/old-apt-lists"
mkdir -p /root/old-apt-lists
mv /etc/apt/sources.list.d/*.list /root/old-apt-lists/

echo "== Resulting sources"
cat /etc/apt/sources.list /etc/apt/sources.list.d/proxmox.sources

echo "== apt update"
apt update

if apt policy 2>/dev/null | grep -q bookworm; then
  echo "!! bookworm repositories are still listed - STOP and check before upgrading."
  exit 1
fi

echo
echo "Repositories switched cleanly. Next: apt dist-upgrade"
SCRIPT
chmod +x /root/pve9-repos.sh
bash -n /root/pve9-repos.sh && echo "Script syntax OK"

The cheat sheet

A plain text file with the remaining steps and the rescue notes, so they’re on screen if I end up at the console (cat /root/pve9-notes.txt):

PVE 8 -> 9 UPGRADE CHEAT SHEET

Reattach to the upgrade session:   tmux attach -t upgrade

1. bash /root/pve9-repos.sh
2. apt dist-upgrade
     /etc/issue = N | lvm.conf = Y | sshd_config = D, then decide
     grub = N | chrony.conf = Y | other = D, then N if unsure
     changelog = q | STOP if it wants to remove proxmox-ve
3. apt install intel-microcode
4. pve8to9 --full    (remove systemd-boot ONLY if it says so)
5. reboot
6. pveversion   |   ip -br link   |   ping -c 3 <gateway-ip>

NO NETWORK AFTER REBOOT:
  ip -br link                    (find the NIC's new name, starts with "en")
  nano /etc/network/interfaces   (replace enp0s31f6 in BOTH places:
                                  its iface line AND vmbr0's ovs_ports line)
  ifreload -a

UNDO THE REPO SWITCH (only BEFORE running apt dist-upgrade):
  rm /etc/apt/sources.list.d/proxmox.sources
  cp /root/apt-backup-pve8/sources.list /etc/apt/sources.list
  mv /root/old-apt-lists/*.list /etc/apt/sources.list.d/
  apt update

Phase 5: Upgrade day

1. Web UI: select the node, Bulk Actions → Bulk Shutdown, all guests.
2. Mac Terminal: ssh root@192.168.10.164, then tmux new -s upgrade.
3. bash /root/pve9-repos.sh: the bookworm check came back empty.
4. apt dist-upgrade

The prompts

The changelog pager surprised me: before installing, apt shows notes about changes in the new package versions (OpenSSH, adduser and so on) in a pager with a : prompt at the bottom. It’s waiting for q, and the upgrade continues once you press it.

The hard stop: if apt ever says it wants to remove proxmox-ve, answer n. It means a repository is misconfigured.

My mistake: pasting the answer guide

When apt finished, the shell printed a series of command not found errors: lvm.conf, sshd_config, grub, STOP, changelog, plus /etc/issue: Permission denied. I’d accidentally pasted the cheat sheet’s answer guide along with a command, while apt was still running. Once apt exited, bash ran each line as a command. Nothing changed, since none of those words are real commands, and the “permission denied” was bash refusing to execute /etc/issue.

The real question was whether apt’s prompts had read some of that pasted text as answers. These checks settled it:

dpkg --audit            # prints nothing if every package is fully configured
apt dist-upgrade        # should report 0 upgraded, 0 installed, 0 to remove
sshd -t && sshd -T | grep -Ei 'permitrootlogin|passwordauthentication'
find /etc \( -name '*.dpkg-*' -o -name '*.ucf-*' \) -exec ls -l {} \;

• sshd -t checks the SSH config is valid, and sshd -T shows the settings that will apply. Root login was still allowed, so I’d be able to reconnect after the reboot.
• The find reveals each config-file decision. A .dpkg-old file means the new version replaced mine; a .dpkg-dist file means mine was kept. issue, lvm.conf and ca-certificates.conf had taken new versions, which is what I’d intended, and nothing else had been touched.

Lesson: copy only the command lines, one at a time.

Two errors that the reboot fixed

Next came the microcode and a second checker run:

apt install intel-microcode
pve8to9 --full | grep -E 'WARN|FAIL'

Two warnings remained:

• “a suitable kernel (proxmox-kernel-7.0) is installed, but an unsuitable (6.8…) is booted”: simply a reboot reminder.
• “No (active) time synchronisation daemon detected”: chrony had been running before the upgrade and now wouldn’t start.

Earlier in the upgrade output, postfix had also failed to restart with Transaction contains conflicting jobs 'restart' and 'stop'. And the chrony log showed:

Fatal error : Could not send notification to $NOTIFY_SOCKET

The config was fine: chrony loaded it normally, then failed at the final step, telling systemd “I’ve started” and getting no answer. Both errors have the same cause. After dist-upgrade, the new Debian 13 packages are on disk, but the old systemd (PID 1) keeps running until the reboot. Services restarted in that window, like the new chrony 4.8 or postfix, can fail to talk to it. Neither matters on a single node for the length of a reboot, so:

reboot

Phase 6: After the reboot

pveversion              # pve-manager/9.2.x (running kernel: 7.0.x-pve)
ip -br link             # enp0s31f6 and vmbr0_mgmt UP
chronyc tracking        # reference server and a microsecond-level offset
pve8to9 --full | grep -E 'WARN|FAIL'    # WARNINGS: 0, FAILURES: 0

The NIC kept its name, chrony started normally, and the checker was clean. The $NOTIFY_SOCKET error was indeed only the pre-reboot mismatch.

Counting running guests from ip -br link

Proxmox creates a host-side network interface for every running guest, named after the guest’s ID:

So tap = VM, veth = container, then the guest ID, i and the NIC number. These only exist while a guest runs. Counting the distinct IDs gave the same eight guests as before the upgrade, so start-at-boot had brought everything back.

That shows each guest started, not that its service works. The real test was opening, through Traefik:

• https://kimai.home.example.com (with my recent entries present)
• Pi-hole, Home Assistant, Calibre and pfSense by hostname
• the Proxmox web UI, after a force-reload (⌘ + Shift + R)

All worked, which confirmed DNS, Traefik, the certificate and the guests in one go.

One thing that looks wrong but isn’t

PVE 9 accounts for VM memory differently. Guests that don’t report their own memory usage, including pfSense (FreeBSD), can show over 100% in the UI. It’s only a display change.

A hook to remember

During the upgrade I also noticed Removing subscription nag from UI.... That’s an apt hook from a community post-install script I ran at some point, and it edits the web UI’s JavaScript after every update. It caused no trouble here, but if the web UI ever comes up blank after an update, that hook is the first suspect.

Cleanup (a few days later)

apt autoremove

This removes the old 6.8 and 5.15 kernels and the leftover Debian 12 libraries. I waited a few days first, because the old kernels are the fallback boot option if the new one misbehaves. The most recent 6.8 kernel isn’t on the autoremove list, so one fallback remains.

Optional extras:

• apt modernize-sources converts the Debian repo file to the new deb822 format, matching the Proxmox one. It previews the changes and asks before applying.
• /usr/share/pve-manager/migrations/pve-lvm-disable-autoactivation clears the checker’s LVM autoactivation notice. Optional for local storage.
• The pre-upgrade backups: keep them protected for a few weeks, then unprotect and delete.

A bonus of being on 9.1+: Docker inside LXC containers works again, if I ever want it.

Bonus: making the Twingate connector unprivileged

The warning that started it

A day after the upgrade, starting my Twingate connector container showed:

WARN: Systemd 249 detected. You may need to enable nesting.
TASK WARNINGS: 1

Nesting lets a container create its own namespaces, the walled-off views of processes, mounts and networks that containers are built from. That’s what lets Docker run inside an LXC, and newer systemd versions use the same mechanism to sandbox individual services inside the container. Without it, some services start slowly or fail. In an unprivileged container, nesting is safe, and Proxmox turns it on by default for new ones.

What the check actually found

To see every container’s features and privilege level at once:

for id in $(pct list | awk 'NR>1{print $1}'); do echo "$id: $(pct config $id | grep -E '^(features|unprivileged)' | tr '\n' ' ')"; done

The connector already had nesting=1, so the warning itself didn’t matter. But the output showed something more important: every container had unprivileged: 1 except the connector. That line only appears in the config of unprivileged containers, so the connector was privileged.

In a privileged container, root inside the container is real root on the host. If it were ever compromised, the attacker would be much closer to owning the Proxmox host. Of all my guests, the Twingate connector is the one that talks to an outside service for remote access, so it’s the one I most want tightly contained.

Converting it with a backup and restore

Proxmox can convert a container to unprivileged while restoring a backup. The plan, done from home since remote access drops during the switch:

1. Back up the old container (Stop mode, to the NAS).
2. Stop it and turn off Start at boot, so two copies of the same connector never run at once.
3. Restore the backup to a new ID with Unprivileged container ticked, leaving Unique unticked so the network settings stay the same.

The restore from the web UI failed:

tar: ./var/spool/postfix/dev/urandom: Cannot mknod: Operation not permitted
tar: ./var/spool/postfix/dev/random: Cannot mknod: Operation not permitted
tar: Exiting with failure status due to previous errors
TASK ERROR: unable to restore CT ...

Those are device files in the mail server’s sandbox (postfix keeps a mini /dev folder). An unprivileged container isn’t allowed to create device files, which is the whole point of it being unprivileged. Tar extracted everything else, then exited with an error because of those two, and Proxmox treated that as a failed restore and cleaned up.

The connector doesn’t need those files, and postfix works without them. The shell version of the restore has an option to accept unpack errors:

pct restore 211 NAS-STORE:backup/vzdump-lxc-210-<date>.tar.zst \
  --storage <thin-pool> --unprivileged 1 --ignore-unpack-errors 1

From the shell, pct restore doesn’t print TASK OK. Success is the last line, merging backed-up and given configuration.., followed by the prompt with no error.

Switching over and checking

pct shutdown 210 && pct set 210 --onboot 0     # old, privileged
pct start 211 && pct set 211 --onboot 1        # new, unprivileged
pct config 211 | grep -E '^(features|unprivileged)'
pct exec 211 -- systemctl status twingate-connector

Then three real checks: the connector shows online in Twingate’s admin console, https://kimai.home.example.com loads from my phone with Wi-Fi off and Twingate on, and the service is active (running). All three passed. If they hadn’t, going back would have been one stop and one start.

Two loose ends:

• The connector doesn’t need a mail server, so I turned postfix off: pct exec 211 -- systemctl disable --now postfix.
• perl: warning: Setting locale failed appears when running pct exec from my Mac. It passes my SSH session’s language setting (en_US.UTF-8) into a container that doesn’t have that locale installed. It’s harmless; locale-gen en_US.UTF-8 inside the container silences it.

I’m keeping the old container stopped for a week as a fallback, then deleting it. Now every container on the host is unprivileged.

Lessons learned

• The checker is the plan. pve8to9 --full found the one real problem (the stale /etc/hosts entry) before it could stop the host from booting properly. Run it before and after.
• Reboot after the 8.4 update, so problems from the update and from the major upgrade don’t get mixed up.
• Backups on the NAS, not on the thin pool. LVM-thin can’t hold backup files anyway, and a backup beside the originals barely counts.
• Test-restore at least one backup. “TASK OK” on the backup proves it was written; a restore proves it can be read back.
• SSH + tmux beats typing at the console, and a pre-written script plus cheat sheet beats both.
• Paste commands one at a time. Pasting a block while apt is prompting is how I fed the shell my own notes.
• Errors between dist-upgrade and the reboot can be transitional. New services restarting under the old systemd fail in odd ways; check the logs, and if the config is fine, the reboot clears them.
• ip -br link tells you which guests are running, at a glance.
• Read warnings for what they reveal, not just what they say. A harmless nesting warning led me to the one privileged container on the host, which happened to be the remote-access entry point.