Thursday 23 August 2012

DPM 2012 - no reports on backup jobs for you!

This morning we came across an interesting issue with DPM 2012, something which is really trivial, but really annoying if you don't know what is wrong!

Recently I moved our DPM servers over to DPM 2012 - and was amazed by how smoothly it went (more because I'm used to upgrading Backup Exec and watching the earth cave in), and all seemed to be working well.

However, I'm VERY paranoid when it comes to backups - and so my colleague has responsiblity for making sure backups happen - reviewing reports and so on. We'd been using the reporting in DPM 2010 quite happily, he received and reviewed the reports on a regular basis.

Since 2012 he'd not been getting them... it seems that the reporting gets broken on an upgrade, and the SMTP Server settings were not right anymore - bizarrely each server we had seemed to have different states - one had no SMTP Server details anymore, one had them but complained they weren't right, and the other had half of the settings. All very strange.

In theory this wasn't an issue - back into SMTP Settings, repopulate and reconfigure the reporting part.

Or not...

Despite having all the details in the "SMTP Server" setting, and having those details set correctly (validated by the send test option and the receipt of a test e-mail etc) we couldn't setup any of the reports.

Why?

Error 3010
"DPM cannot setup an e-mail subscription for this report"
(and then advises you to go and setup your SMTP Server!)

It turns out that the issue is in fact that SQL Reporting Services doesn't actually have the details you entered - from what I can see, the system still looks in the DPM 2010 instance of SQL (because it doesn't remove it or the instance of old SQL it made) at upgrade, so it updates that instead. D'oh!

Whilst that's clearly a bug and should be fixed, the good news is that there is a quick fix.

Go into SQL Reporting Services Configuration, log into the DPM2012 instance, choose the "E-Mail Settings" and fill in the Sender Address and SMTP Server. Save that and you're golden.



Tuesday 14 August 2012

DPM 2010 to DPM 2012 upgrade...

Today we decided it was time to upgrade to DPM 2012. I had written a really good post about our experience and what happened. Annoyingly, just as I went to save/publish it, the app I was using (BlogJet) decided to die and take the draft with it. So sadly you won't see that post yet…
But the good news is that unlike the blog post, the actual install of DPM 2012 went well :-)

I'll try and re-write our experience of setting it up properly later!

Removing orphaned/old Agents from DPM Management View

Just a quick tip - we're preparing our upgrades to DPM 2012 and have decided it's about time we cleaned up some bits and pieces in DPM.

One of the "to clear up" items was to remove old no-longer-in-use Protected Computers from the Agents List in Management - for various reasons we'd ended up with some machines who still show up despite being long since removed and thrown away.

There's no way to do this from the main UI, but with a quick bit of DPM powershell you can nuke those old systems...

Run this script:

Remove-ProductionServer.ps1 -DPMServerName DPMSERVERNAMEHERE -PSName SERVERNAMEHERE

NOTE: You'll generally need to do "SERVERNAME.fullyqualified.com" if it is a domain joined server.

Once you've closed/reopened the UI you'll have a much tidier list :-)

Thursday 9 August 2012

Symantec Backup Exec 2012 - Hell or Sane?

As those familiar with this blog will know, we used to use Symantec Backup Exec to look after backups for many many servers. We ran into all sorts of problems, and that's how we ended up creating this blog –  known originally just as Backup Exec Hell. Today we use the Microsoft Data Protection Manager product since our business is *almost* entirely Windows based, and since we did that we spend a lot less time dealing with backup issues.

As a result we've not spent all that much time paying attention to Backup Exec's development since we moved off it, and our core experiences of it ended with version 10d. For reasons better known to someone else, I recently decided I was a little curious to see how the product had moved on. Popping onto the Symantec Forums, I soon saw a good number of posts complaining about the latest release –  Backup Exec 2012.

Obviously I haven't used the product, but one thing that sticks out is the general noise about one change…

They've made major changes to the way it is designed to operate –  switching the core method of backup from the old “Selection Lists” and “Resources” to a “Server Centric” view.

This seems to be causing lots of complaints from long term users. On first glance it does sound like Symantec have done a bad thing. But actually (and I hate to say it) I see why they've made the change. It's just that the execution is lousy… (so nothing new for our chums at Symantec).

To understand why they've changed the way you handle things to be server centric, you only have to look as far as Data Protection Manager (which is sort of server centric). Today's backup systems take advantage of new technologies (certainly compared to the original tech available when backup exec's current methods were written), plus the nature of server technology has changed.

Back in the day you had server which you could and did backup in a “files on the disk” manner. You couldn't backup files in use, and so on. This meant for some tools –  like a database, you either had to take the database offline, or the database system had to have some backup utility, spitting files out to the disk, so you could back them up. Eventually we ended up with things like “System State” in Windows, and then alongside the crazy growth in disk storage needs, some tech got really smart –  like Microsoft Exchange or SharePoint. The problem with the “way it used to be” is that it was a slow process, the inability to backup stuff in use/always open was becoming a pain, and having to have this crazy do one type of backup, do another was asking for trouble. Plus taking a system offline to do backups was horrible and as technology is more and more critical and 24/7 in nature, unfeasible.

We now have technologies like VSS (for snapshots) and this “virtual machine” thing has happened –  thanks to VMWare, Hyper-V etc. With virtual machines in particular, the hypervisors now offer direct support for backing up systems without having Backup Software agents on every machine and all running unaware of each other etc).

As a result of this, it's more important than it used to be to get a “snapshot” of a servers state with EVERYTHING –  files, exchange, sql, system state etc in a consistent manner. After all, if a server fails, it's no use having a copy of the windows files, but not the exchange information stores. And for straightforward recovery, the information stores are most useful when the rest of the server is there too. So actually backing up “whole servers” makes sense –  and if you're using tools like Hyper-V, backing up all the guests and the host in one hit makes a lot of sense. What symantec thus appear to have done is move to this model, where you backup “a server” and not “the C drive” etc. That's royally hacked users off.

The main complaint seems to be that the upgrade process messes up the existing setup, and they end up with many more jobs and nothing makes sense. That's fair enough. It's interesting though that a couple of posts I read were from users NEW to Backup Exec 2012, and they didn't see what the fuss was because they never had a legacy setup to migrate.

From my perspective, I think Backup Exec users need to re-think how they do backups, and look at the new model as genuine progress and sensible long term. It does mean rethinking how you setup your backups sure –  and if you have a lot of servers and an existing setup, I appreciate that upgrading to Backup Exec 2012 seems to be causing pain –  and Symantec probably could have done a better job at making the changes clear (but from experience most people just dive in with an upgrade anyhow…)

Ignoring the other pains and issues, if you start thinking of things from a “I need to backup my server so I can recover it no matter what” –  there's no reason a “server centric” approach can't and won't work. If you think still in terms of “files” and “drives” and so on, then you're not going to get on with the new version. But really you need to re-think. Backup Exec has used the method it has today for a long long time, and it's time to put it out to pasture. Who knows, maybe in the next version they can jettison more old thinking and give it half a chance of being a credible product again.

As someone who has moved to DPM, we had to get our heads around having “Protection Groups” and “Server Centric” concepts –  coming from Backup Exec this was an interesting experience I admit, but would I want to go back to the crazy polices, templates and selection lists stuff…. no thanks! Today we have “Protection Groups” based on “Location or Customer –  Server Product Family” –  which is what suits us. So a company “Acme Ltd” might have a Protection Group “Acme Ltd – Web and SQL Servers” and another “Acme Ltd –  Internal Network Servers” while we may have “London –  Hosted E-Mail Service” and so on…  In each group we setup each server to be backed up, with all its resources. We can still exclude a drive, folder etc if we want to… although we rarely do since just having a complete image makes sense –  and thinking about it, the only reason we did that with Backup Exec was because it was slow at backing up, and the promised “Synthetic Backups” never worked, so we had to keep doing “full” backups of data that hardly ever changed and this was an issue.

With DPM, it backs up everything ONCE, then just keeps getting snapshots so we can recover. It's much better on network usage, it's far faster and because it doesn't take forever we just let it back up everything without question.

So in conclusion, we think you should go with these changes, understand there is some likely sound reasoning behind it, bite the bullet and reconfigure your setup to work the way they want you to work from now on. (Or just move to DPM if you've got Microsoft-only workloads…)

PS –  I know there are plenty of other issues with Backup Exec 2012 –  I just don't think that this change is the real issue!

Wednesday 8 August 2012

DPM: Error 10048 0x02740

So it's been a while since we posted, and generally that's because DPM is a godsend. It almost works without attention, and apart from a few quirks we've discovered, and the odd thing that happens but isn't ACTUALLY documented anywhere sane, things are good.

Compared to Backup Exec (we stopped using it on BEWS 10d), DPM 2010 seems to "just get it" when it comes to backups. Once a backup is setup, it knows how to backup, it knows when to backup and it actually backs up.

If there's an issue it attempts to fix minor issues via consistency checks etc, and if there's really an issue, it is glaringly obvious for you to fix it. That's pretty good news when you're used to Backup Exec being hell on earth, randomly dropping jobs to "hold" status because of an issue and so on.

However yesterday a random issue cropped up which I hadn't seen before.

One of our Protected Servers Agent Status in the DPM console was "unavailable" - and the error logged was "10048 0x02740". The protected server in question is running Exchange Server 2007.

This appears to have happened because the IIS process on the protected server was suddenly using TCP Ports 5718 and 5719. This prevents the DPM Remote Agent from starting.

To fix this, you can simply:

(a) Stop IIS (iisreset /stop)
(b) Run the DPM Agent
(c) Start IIS (iisreset /start)

...or in our case, do nothing - by this morning it had cleared itself (we hadn't restarted IIS as we didn't want to drop the live users connected via OWA and OutlookAnywhere on this box).

Wednesday 29 February 2012

Where have we been?

Hello

We figured we should drop a quick post in to say hello - and so you know we haven't abandoned this blog! ... The truth is we've been busy, but have had some problems with Data Protection Manager too... and we'll post some updates soon.

Until then, good luck fighting Backup Exec if you're still stuck with it, and hope you avoid a few of the pitfalls we found in DPM which can be irritating if not fatal...