DPM Rollup Update Installation Fails - Failed to execute SQL string, error: BACKUP DATABASE is terminating abnormally

Recently we decided to deploy Update Rollup 5 for Data Protection Manager 2019, but were running into issues getting the update to install.

In the installation log we spotted lots of:

MSI (s) (BC!10) [14:34:02:258]: Transforming table Error.

But most importantly we saw:

ExecuteSqlStrings:  Error 0x80040e14: failed to execute SQL string, error: BACKUP DATABASE is terminating abnormally., SQL key: KB5024231_BackupDpmDb.sql.97DDF5B3_3770_4C3E_8673_52BD081E1EFD SQL string: BACKUP DATABASE DPMDB_SERVERNAME080fe81c_748b_403a_b6ca_fa1d3a042da9 TO DISK = 'QFEDPMDB_SERVERNAME080fe81c_748b_403a_b6ca_fa1d3a042da9.bak' WITH COPY_ONLY, INIT

Well why would that be an issue - it usually works...

The answer is annoyingly simple:

If you delete the file:

QFEDPMDB_SERVERNAME080fe81c_748b_403a_b6ca_fa1d3a042da9.bak

(which will be located in your normal backup directory for SQL Databases and named with the guid and machine/database name you actually have rather than the above)

It will work. But why does it care, the file usually exists from the last update rollup.

The answer is in the command - if you run that in SSMS you'll see this error appear:

System.Data.SqlClient.SqlError: Cannot use the backup file because it was originally formatted with sector size 4096 and is now on a device with sector size 512. (Microsoft.SqlServer.Smo)

Ah ha! Indeed - the last update rollup we installed we were indeed using a disk with 4096 sector size, and the new ones are back to 512 for the system volume (long story for another day!)

Deleting the old backup is the easiest fix.

DPM 2019 - Upgrade leaves Secondary Protection Servers Inaccessible

It is now 2022. We're still using DPM (Data Protection Manager) as our primary backup solution. It continues to be the hidden gem of the Microsoft System Center suite and a product I really hope they never screw around with because overall it is fantastic. It might have a simple interface, it might not have all the bells and whistles, but at the core, it does backup, and most importantly, restore workloads.

We've recently upgraded our DPM 2016 suite to DPM 2019 and after the upgrades almost all was well. It took a few jumps as we had to first update SQL Server on each of the DPM servers as DPM2019 didn't support the older version we used, but equally DPM 2016 didn't support a new enough version for us to have bothered until now.

I strongly recommend you backup the DPM database via SQL Enterprise Manager before you begin using a "Copy Only" backup. Do this on every DPM Server. Then find the SQL server version that is supported by both the current DPM version you use, and the one you intend using (in our case supported by DPM 2016 UR10 and 2019 UR4)

The only problem we found remained was that Secondary Protection didn't work - the Primary DPM Server insisted that there was a problem with the agent version of the Secondary DPM Server but we knew that to be bogus.

DPM itself would say:

"Data Protection Manager ID: 296"

"The protection agent is incompatible with the version of DPM that is installed on this computer. All subsequent protection and recovery tasks will fail on this computer until you update the protection agent"

This is a bogus error - and indeed if you refresh the status of the secondary server you'll briefly see another error that comes/goes - indicating something else is amiss.

The "Recommended action" is a bit misleading in the case of secondary protection servers as it ultimately advises you should reinstall the agent or update the agent but that isn't something you can do on a Secondary DPM Server - assuming your Secondary Protection DPM server already runs the same release of DPM as your Primary (eg if it was DPM 2019 so too must the Secondary be - but also they should be up to the same Update Rollup), then this isn't the problem.

More likely is that the Primary server has not got permissions setup properly to allow the Secondary to talk to it. To fix this:

(1)

On the PRIMARY DPM Server, go into the Local Users and Groups section, and find the group named "DPMRADmTrustedMachines" - make sure the Secondary Server name is listed in this group - and if not, add it. 

In our case, adding the machine back to this group (it was clearly lost during the upgrade), then refreshing the agent status twice on the Secondary server corrected the issue and DPM now thinks all is well and has started providing secondary protection.

(2)

On the PRIMARY DPM Server, go to the "DPMDRTrustedMachines" group and ensure the Secondary DPM Server is listed there - if you don't do this your Agent Status is likely to say "OK" but you'll find you get error 33119 and issues enumerating the contents of the Primary DPM Server and/or a Protected Server it has already protected that you're trying to setup or modify the Secondary protection of.