Upgrade from 3 to 4

Introduction

Proxmox Backup Server 4 is based on Debian 13 Trixie, a new major release, and introduces several new major features and changes. You should plan the upgrade carefully, make and verify backups before beginning, and test extensively. Depending on the existing configuration, several manual steps — including some downtime — may be required.

Note: A valid and tested backup is always required before starting the upgrade process. You can test the backup beforehand, for example, in a (virtualized) test lab setup.

In case the system is customized and/or uses additional packages or any other third party repositories/packages, ensure those packages are also upgraded to and compatible with Debian Trixie.

In-place Upgrade

Prerequisites

• Perform these actions via SSH, a physical console or a remote management console like iKVM or IPMI.
  • If you use SSH, you should use a terminal multiplexer (for example, tmux or screen) to ensure the upgrade can continue even if the SSH connection gets interrupted.
  • Do not carry out the upgrade via the web UI console directly, as this will get interrupted during the upgrade.

• Upgraded to the latest version of Proxmox Backup Server 3.4, see the roadmap for potential important changes in the stable release.

  Use apt update and apt dist-upgrade (still with Debian Bookworm repos setup) to upgrade to latest 3.4

  • Verify version: The command proxmox-backup-manager versions should print:

    proxmox-backup-server 3.4.2-1 running version: 3.4.2 (or higher)

  • If you do not get updates check correct package repository configuration.

• Make a backup of /etc/proxmox-backup to ensure that in the worst case, any relevant configuration can be recovered:

tar czf "pbs3-etc-backup-$(date -I).tar.gz" -C "/etc" "proxmox-backup"

• Ensure that you have at least 10 GB free disk space on the root mount point:

df -h /

In-place upgrades are carried out via APT. Basic familiarity with APT is required to proceed with this upgrade mechanism.

Installed alongside Proxmox VE

For systems with Proxmox VE and Proxmox Backup Server installed together, you should also read the Proxmox VE upgrade from 8 to 9 how-to carefully.

You can upgrade both in one go, by syncing the steps in which the APT repositories are changed.

Actions Step-by-Step

Before starting the upgrade process, ensure that your Proxmox Backup Server 3.x host is up-to-date.

Continuously use the pbs3to4 checklist script

A small checklist program named pbs3to4 is included in the latest Proxmox Backup Server 3.4 packages. The program will provide hints and warnings about potential issues before, during and after the upgrade process. You can call it by executing:

pbs3to4

To run it with all checks enabled, execute:

pbs3to4 --full

Make sure to run the full checks at least once before the upgrade.

This script only checks and reports things. By default, no changes to the system are made and thus, none of the issues will be automatically fixed. You should keep in mind that Proxmox Backup Server can be heavily customized, so the script may not recognize all the possible problems with a particular setup!

It is recommended to re-run the script after each attempt to fix an issue. This ensures that the actions taken actually fixed the respective warning.

Optional: Enable Maintenance Mode

Enabling the read-only maintenance mode on all datastores ensures that no new backup can be started during the upgrade, while keeping existing ones available to read. The read-only maintenance mode allows you to enforce a known and stable datastore state and reduces the I/O and general load of the Proxmox Backup Server during the upgrade, making that faster.

You can enable and disable the maintenance mode either via the web UI, in the Options tab of each datastore menu entry, or using the command line interface (CLI):

# enable read-only mode (replace DATASTORE-ID with actual value)
proxmox-backup-manager datastore update DATASTORE-ID --maintenance-mode read-only
# disable read-only mode
proxmox-backup-manager datastore update DATASTORE-ID --delete maintenance-mode

Update the Configured APT Repositories

First, make sure that the system is using the latest Proxmox Backup Server 3.4 packages:

apt update
apt dist-upgrade
proxmox-backup-manager versions

The last command should report at least 3.4.2-1 or newer.

Update Debian Base Repositories to Trixie

Update all repository entries to Trixie:

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list

Ensure that there are no remaining Debian Bookworm specific repositories left. You can place a # symbol at the start of the respective line to comment such a repository out, disabling it. Check all files in the /etc/apt/sources.list.d/ folder (like pbs-enterprise.list) and also the top-level /etc/apt/sources.list file. See Package Repositories section in the reference docs for the correct Proxmox Backup Server 4 / Debian Trixie repositories.

Add the Proxmox Backup Server 4 Package Repository

Update the enterprise repository to Trixie in the new deb822 format with the following command:

cat > /etc/apt/sources.list.d/pbs-enterprise.sources << EOF
Types: deb
URIs: https://enterprise.proxmox.com/debian/pbs
Suites: trixie
Components: pbs-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

After you added the new enterprise repository as above, check that apt picks it up correctly. You can do so by first running apt update followed by apt policy. Make sure that no errors are shown and that apt policy only outputs the desired repositories. Then you can remove the old /etc/apt/sources.list.d/pbs-enterprise.list file. Run apt update and apt policy again to be certain that the old repo has been removed. If using the no-subscription repository, see Package Repositories. You should be able to add the Proxmox Backup Server 4 no-subscription repository with this command:

cat > /etc/apt/sources.list.d/proxmox.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/pbs
Suites: trixie
Components: pbs-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

As with the enterprise repository, make sure that apt picks it up correctly with apt update followed by apt policy. Then remove the previous Proxmox Backup Server 8 no-subscription repository from either the /etc/apt/sources.list, /etc/apt/sources-list.d/pbs-install-repo.list or any other .list file you may have added it to. Run apt update and apt policy again to be certain that the old repo has been removed.

Make sure that apt picks it up correctly with apt policy. Then remove the previous Proxmox Backup Server 3 repository from either the /etc/apt/sources.list, /etc/apt/sources-list.d/pbs-install-repo.list or any other .list file you may have added it to. You can use apt policy again to make sure the old repository is not being used anymore.

Instead of removing older repositories, you can also disable them. In .list simply comment them out by adding a # to the beginning of the line. In .sources files, you can add the line Enabled: false to any stanza you want to disable.

Make sure to check that all the .list files you added in /etc/apt/sources.list.d/ got switched over to Trixie correctly.

After you checked that all repositories get picked up correctly with apt policy, update the repositories' package index:

apt update

Note that this command does not start the upgrade itself, it only refreshes the package index and must not return any error.

Upgrade the System

Note that the time required for finishing this step heavily depends on the system's performance, especially the root filesystem's IOPS and bandwidth. A slow spinner can take up to 60 minutes or more, while for a high-performance server with SSD storage, the dist-upgrade can be finished in less than 5 minutes.

Note: While the packages are being upgraded certain operations and requests to the API might fail (for example logging in as system user in the pam realm)

To get the initial set of upgraded packages, run:

apt update
apt dist-upgrade

During the above step, you will be asked to approve changes to configuration files and some service restarts, where the default config has been updated by their respective package.

You may also be shown the output of apt-listchanges, you can simply exit there by pressing "q". If you get prompted for your default keyboard selection, simply use the arrow keys to navigate to the one applicable in your case and hit enter.

For questions about service restarts (like Restart services during package upgrades without asking?) use the default if unsure, as the reboot after the upgrade will restart all services cleanly anyway.

For questions about (default) configuration changes, it's suggested to check the difference for each file in question and choose the answer accordingly to what's most appropriate for your setup.

Common configuration files with changes, and the recommended choices are:

• /etc/issue -> Proxmox Backup Server will auto-generate this file on boot, and it has only cosmetic effects on the login console.

  Using the default "No" (keep your currently-installed version) is safe here.

• /etc/ssh/sshd_config -> If you have not changed this file manually, the only differences should be a replacement of ChallengeResponseAuthentication no with KbdInteractiveAuthentication no and some irrelevant changes in comments (lines starting with #).

  If this is the case, both options are safe, though we would recommend installing the package maintainer's version in order to move away from the deprecated ChallengeResponseAuthentication option. If there are other changes, we suggest to inspect them closely and decide accordingly.

• /etc/default/grub -> Here you may want to take special care, as this is normally only asked for if you changed it manually, e.g., for adding some kernel command line option.

  It's recommended to check the difference for any relevant change, note that changes in comments (lines starting with #) are not relevant.

  If unsure, we suggested to selected "No" (keep your currently-installed version)

Check Result & Reboot Into Updated Kernel

If the command exits successfully, you can reboot the system in order to enable the new kernel.

systemctl reboot

Please note that you should reboot even if you already used the 6.2 kernel previously, through the opt-in package on Proxmox Backup Server 3.

Following the Proxmox Backup Server upgrade

Empty the browser cache and/or force-reload (CTRL + SHIFT + R, or for MacOS ⌘ + Alt + R) the Web UI.

Check Status of Services

Check that the statuses of the main services are active (running)

systemctl status proxmox-backup-proxy.service proxmox-backup.service

Optional: Disable Maintenance Mode Again

If you enabled the maintenance mode before the upgrade, don't forget to disable it again. You can do it via the web UI, in the Options tab of each datastore menu entry, or using the command line interface (CLI):

# disable read-only mode (replace DATASTORE-ID with actual value)
proxmox-backup-manager datastore update DATASTORE-ID --delete maintenance-mode

Optional: Modernize apt Repository Sources

You can migrate existing repository sources to the recommended deb822 style format, by running:

apt modernize-sources

By answering the following prompt with "n" you can check the changes the command would make before applying them. To apply them simply run the command again and respond to the prompt with "Y".

The command will also keep the old .list files around by appending .bak to them. So you will have the new .sources files and the old repository configurations in the .list.bak files. You can remove the leftover backup files once you verified that everything works smoothly with the new format.

Potential Issues

General

As a Debian based distribution, Proxmox Backup Server is affected by most issues and changes affecting Debian. Thus, ensure that you read the upgrade specific issues for Debian Trixie.

Please also check the known issue list from the Proxmox Backup Server 4.0 changelog: https://pbs.proxmox.com/wiki/index.php/Roadmap#4.0-known-issues

Older Hardware and New 6.14 Kernel

Compatibility of old hardware (released >= 10 years ago) is not as thoroughly tested as more recent hardware. For old hardware we highly recommend testing compatibility of Proxmox Backup Server 4 with identical (or at least similar) hardware before upgrading any production machines.

We will expand this section with potential pitfalls and workarounds once they arise.

Network

Network Interface Name Change

Due to the new kernel recognizing more features of some hardware, like for example virtual functions, and interface naming often derives from the PCI(e) address, some NICs may change their name, in which case the network configuration needs to be adapted.

In general, it's recommended to either have an independent remote connection to the Proxmox Backup Server's host console, for example, through IPMI or iKVM, or physical access for managing the server even when its own network doesn't come up after a major upgrade or network change.

The latest version of Proxmox Backup Server 3.4 and 4.0 provide a package called proxmox-network-interface-pinning that you can install. This package offers a CLI tool that helps you pin all network interfaces to NIC-based names and update the network configuration simultaneously.

Systemd-boot meta-package changes the bootloader configuration automatically and should be uninstalled

With Debian Trixie the systemd-boot package got split up a bit further into systemd-boot-efi (containing the EFI-binary used for booting), systemd-boot-tools (containing bootctl) and the systemd-boot meta-package (containing hooks which run upon upgrades of itself and other packages and install systemd-boot as bootloader). As Proxmox Systems usually use systemd-boot for booting only in some configurations (ZFS on root and UEFI booted without secure boot), which are managed by proxmox-boot-tool, the meta-package systemd-boot should be removed. The package was automatically shipped for systems installed from the PBS 3.0 to PBS 3.4 ISOs, as it contained bootctl in bookworm. If the pbs3to4 checklist script suggests it, the systemd-boot meta-package is safe to remove unless you manually installed it and are using systemd-boot as a bootloader. Should systemd-boot-efi and systemd-boot-tools be required, pbs8to9 will warn you accordingly. The pbs3to4 checklist script will change its output depending on the state of the upgrade, and should be run continuously before and after the upgrade. It will print which packages should be removed or added at the appropriate time. The only situation where you should keep the meta-package systemd-boot installed is if you manually setup systemd-boot for your system. See also the filed bug for systemd-boot.