Recovering from a failed Proxmox upgrade

Today I’ll look at how I recovered from a failed Proxmox 8 to 9 upgrade on a small form factor Dell that I use. This system is one of two in the cluster, and it failed during the upgrade process. This failure was not caught by the pve8to9 utility, which gave me a clean report prior to the upgrade. However, during the last few steps of the upgrade, I got the message below, which made me think I would be doing some extra work on reboot.

Installing for x86_64-efi platform.
grub-install.real: error: cannot find EFI directory.
Failed: grub-install --target=x86_64-efi
WARNING: Bootloader is not properly installed, system may not be bootable

I could have investigated the situation right there but decided to send it and reboot the system anyway. The system failed to boot, instead showing the Welcome to GRUB message for a moment before restarting the system which then went to the BIOS setup screen. My issue matches one reported on the Proxmox forums quite well so I followed what OP did. You can find the post at https://forum.proxmox.com/threads/pve-9-grub-stops-the-machine-immediately.169366/.

Using balenaEtcher on macOS, and the systemrescue image at https://www.system-rescue.org/Download/, I copied the image to a USB drive and then booted the Dell system from it. From there, I mounted the disks as described in the forum post and ran the necessary commands as well as some others that I found were necessary for my situation. Here is the list of commands I ran on my system after ensuring /dev/sda was my root/boot device:

mount /dev/mapper/pve-root /mnt
mount /dev/sda2 /mnt/boot/efi
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
mount --bind /dev/pts /mnt/dev/pts
chroot /mnt
mount -t efivarfs efivarfs /sys/firmware/efi/efivars
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox /dev/sda
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox --removable /dev/sda
echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u
apt install --reinstall grub-efi-amd64
update-initramfs -u -k all

Once this finished I did the following

umount /mnt/dev/pts
umount /mnt/proc
umount /mnt/dev
umount /mnt/sys/firmware/efi/efivars
umount /mnt/sys
umount /mnt/boot/efi
umount /mnt/
reboot

As it rebooted I removed the USB stick from the system. My system was now able to boot without issue and I went back to ensuring the upgrade was successful and I could migrate my VMs back to it.

I am not certain what step I might have missed that could have prevented this, so if anyone knows I would love to hear about it.

Leave a Reply

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.