Sometimes you want Backup Exec back...

So some weeks on and DPM on the whole is everything a backup product should be. There are a few annoyances and a few things that need radical improvement, but on the most important factor - backup reliability, DPM wins hands down.

One area though that Backup Exec was MUCH better at is E-Mail Alerting. Firstly, it was more flexible - any SMTP server was OK, and that worked great for us. DPM however only seems to work if it's pointed at an Exchange based environment - which was a bit annoying since that's not really how I wanted it done. I guess that's the side effect of the "optimal for microsoft based workloads" strategy, but nonetheless...

The other bit though is the alerting capability. You can have alerts for 3 categories "Informational" "Warning" and "Critical", and a list of e-mail addresses to send to. You get one list of e-mail addresses and ALL of those addresses can receive the alerts you enable.

You can't set any thresholds, you can't customise the alerts and most annoyingly, it alerts you to both "Problems" and "Resolved". Given it tries to self resolve I was hoping "Critical" would only alert you to issues it has tried to resolve and failed at or cannot resolve because it needs our intervention.

All in all a bit poor and makes me want BEWS back, just for that bit anyhow...

Another Top Tip for DPM - "replica is inconsistent" on System State and Bare Metal Recovery...

I thought I'd share this Top Tip. It is reasonably well documented, but a classic thing you'd often miss.

If you're finding that "System State" and "Bare Metal Recovery" items are frequently sitting in a "replica is inconsistent" state ...which happens a lot on Windows 2008 system... then the chances are you've not got "Windows Server Backup" as an installed feature on the server you're backing up (the protected server).

It's dead simple to sort, run Server Manager, click Add Feature, check "Windows Server Backup" and wait for it to install - job done - run a consistency check and they'll be sorted and work thereafter.

Of course, why the DPM installer doesn't just install this (or at least prompt) as part of the roll out since it is basically a dependency is anyones guess...

Common DPM Errors...

Since we've now got most of the Data Protection Manager 2010 installations done, I thought I'd share a few common issues we've come across, and the fixes. Maybe this'll save you a LOT of hassle...

"Access Denied (0x80070005)"
Common causes are listed all over the place, suggesting Firewalls as the issues and DCOM Permissions. All entirely possible. One other thing to consider, especially if you've setup Forest Trusts etc, just make sure you've made sure the AD Network holding your DPM Server(s) is fully accessible - and that this traffic isn't restricted either! In our case, we had a Cluster with 2 servers, one in a Subnet (we'll call this Subnet A), another in a different subnet (Subnet B) and our DPM Servers (and the DPM AD Network) in another (Subnet C).

While Subnet A and B could talk without restriction, and A could quite happily talk to C, for historical reasons, B and C weren't completely open for communication. So my tip - make sure you've considered Active Directory Authentication and not just "DPM to Protected Server" issues!

Agents are "unavailable" and "VssError: Invalid value for registry"

This ia bit of an odd one and just "happened" on a previously perfectly happy server. We resolved this by simply removing the account used to push out the agents in the DCOM Config (run "dcomcnfg.exe"), find the "DPM RA" in the list and remove/readd the user. No idea what caused that mind!

Replica is inconsistent with System State and repeatedly so...

Especially if you're on a Windows 2003 SP-2 32-bit system? Yep, thought so. You've probably just not got enough space on the system drive (normally C:\). You should move the normally hidden "DPM_SYSTEM_STATE" folder to another drive, ideally with +10GB free, and then update the data source...

\Microsoft Data Protection Manager\DPM\datasources\PSdataSourceConfig.xml

change:

%SystemDrive%\DPM_SYSTEM_STATE\*

so it points to wherever you put it... easily sorted.


Hopefully they'll help you for now, more tips later!

DPM and "Secondary Protection" and "Chaining"

So first, the good news. Having rolled out DPM 2010 to our production environments, by and large all seems well, backups are completing, using less time, hassle and bandwidth overall than the previous Backup Exec solution.

It does seem to consume much high amounts of storage - but it isn't yet sufficiently clear if this is worthwhile yet (eg. if the space is pre-allocated so it can meet retention policies and then fills it, or it simply over-estimates likely requirements resulting in lots of unused capacity). We'll find out once we've run it a few weeks in a full production environment with realistic changes and replicas - and if needbe we'll tweak things a little.

Anyhow, I digress, so back to the purpose of this post... The next part of our rollout is to enable the "off site" capabilities - specifically making sure we have a second copy of each servers data at another site - you know for "total disasters".

This is called "DPM Chaining", "Secondary Protection" and various other things depending on the version of DPM, the documentation you read etc and what you are trying to achieve.

Basic steps are simple (after doing the normal DPM setup):

(a) On the second DPM server, push the protection agent to the first.

(b) On the first DPM server, push the protection agent to the second.

(c) On the second server, create protection groups, selecting the first dpm server as the data source, expanding "protected servers" and then treating it as if it was the first server.

(d) Complete the wizard, wait (a long time possibly) for replication to complete the first time.

We'll see how our trial run goes...

DPM 2010 - "Replica is Inconsistent" on 2008 Servers for Statem State

It would be fair to say DPM is proving to be far better than Backup Exec on most things, but occasionally there are some short sighted decisions or stupid issues that could have been better handled.

One small example is where you find System State and Bare Metal Recovery Replicas keep becoming inconsistent on a Windows 2008 system that's being backed up with DPM 2010.

The fix is pretty simple. On the 2008 server you're backing up, go to "Server Manager", load features, choose "Add Feature" and ensure "Windows Server Backup" is an allowed feature (this won't need a reboot).

Given DPM seems to check loads of other pre-requisites you'd expect it would either alert you to this at install time, or just enable it as part of the install (even if there was an option which said "If Windows Server Backup features are not enabled on the source for protection, enable it automatically" or something.

A silly oversight and one that just takes a tiny bit of the sparkle of clueful implementation away I think.

DPM - File Restores in Seconds, not minutes

As part of our deployment of Data Protection Manager (DPM) 2010, we decided we wanted to do as much restore testing as we could. So having contacted our usual customers who help us test and prove anything (call it a focus group if you want), we asked them all to delete random sets of files from the various servers we're backing up using DPM for them.

Obviously we asked them to make sure the files were not critical or important (just in case, safety first naturally!) - and then just tell us what files they wanted back. The theory being we should be able to do this without knowing in advance whats being deleted (ensuring nobody here could take extra backups or look out for anything etc).

Guess what, it worked... first time, and it is very fast. By comparison to Backup Exec, which took a minimum of 3-4 minutes even for a single 100KB Word Document (because of the whole loading media nonsense...), it did the job quickly, very quickly.

Where Backup Exec is more flexible however is if you want to restore a random set of files from a single file in different folders - DPM doesn't appear to let you do this - so I'd have to select files in a single folder, run "Recover..." then repeat for each folder (well through the UI anyhow). However, given the restore takes literally a few seconds, I'm not sure we care too much - and in reality doing this is pretty rare - normally we want a whole folder or a group of files in a folder or similar, rather than completely random odd and sods files from across a server.

Microsoft DPM - A breath of Fresh Air (almost!)

Having finally reached the end of our patience with Backup Exec and its never ending failures to simple requests, the terrible performance issues it suffers and all the other problems we hear about and witness every day, we decided to give Microsoft's Data Protection Manager a whirl.

There are a few important things to think about though if you are looking to switch, since Microsoft DPM is really only about Windows, SQL, Exchange and Sharepoint. If that's what you're running, and you're on 2003 SP-2 or 2008 and above, you should be fine. If you need other platforms and apps which Backup Exec supports you're probably out of luck using this.

Microsoft DPM is a very different product. One of the key differences is that it is truely snapshot based. Backup Exec still does far too much by using file by file methods - this has terrible scaling consequences.

It is mostly about Disk backup, whereas Backup Exec has a wider range of support for traditional tape backup. DPM can do it (it calls this "Long Term" Storage, and uses Disk for "Short Term" (you define what short/long term is...)

So in a nutshell (kind of) here's the story so far:

1) Installation of DPM failed because the install folder was "C:\!Software\DPM2010" whereas the installer ignored the existance of ! and tried to load "C:\Software\DPM2010" and couldn't find its own files. So we just put up with that and put DPM2010 in the c:\ folder root so we could get started.

2) Installation takes a while as it also rolls out SQL 2008 (you can get it to use an existing Database but we opted not to - and this is the recommended approach).

3) Take time to read the pre-req's and understand how DPM works. For example, make sure you have a huge volume on each DPM server (the best scenario) you have left unformatted so it can claim this for itself.

With those basics covered, the initial installation was completely succesful and our first DPM server appeared.

Microsoft Data Protection Manager

So we've reached the point after many years where we want to reduce our use of Backup Exec. Mainly because it is stupidly expensive and just not reliable enough.

So we figured we'd give Microsoft Data Protection Manager a go. Full of optimism, we began the install. It failed at the first hurdle.

You see the software was in a folder "C:\!Software\DPMServer2010"

Except the installer decided that is actually "C:\Software\DPMServer2010"

So although ! is a perfectly valid File System Character, the DPM Installer failed.

Folder renamed and it worked.

It isn't a good start... this is the sort of stupidity Backup Exec had!