Another Top Tip for DPM - "replica is inconsistent" on System State and Bare Metal Recovery...

I thought I'd share this Top Tip. It is reasonably well documented, but a classic thing you'd often miss.

If you're finding that "System State" and "Bare Metal Recovery" items are frequently sitting in a "replica is inconsistent" state ...which happens a lot on Windows 2008 system... then the chances are you've not got "Windows Server Backup" as an installed feature on the server you're backing up (the protected server).

It's dead simple to sort, run Server Manager, click Add Feature, check "Windows Server Backup" and wait for it to install - job done - run a consistency check and they'll be sorted and work thereafter.

Of course, why the DPM installer doesn't just install this (or at least prompt) as part of the roll out since it is basically a dependency is anyones guess...

Common DPM Errors...

Since we've now got most of the Data Protection Manager 2010 installations done, I thought I'd share a few common issues we've come across, and the fixes. Maybe this'll save you a LOT of hassle...

"Access Denied (0x80070005)"
Common causes are listed all over the place, suggesting Firewalls as the issues and DCOM Permissions. All entirely possible. One other thing to consider, especially if you've setup Forest Trusts etc, just make sure you've made sure the AD Network holding your DPM Server(s) is fully accessible - and that this traffic isn't restricted either! In our case, we had a Cluster with 2 servers, one in a Subnet (we'll call this Subnet A), another in a different subnet (Subnet B) and our DPM Servers (and the DPM AD Network) in another (Subnet C).

While Subnet A and B could talk without restriction, and A could quite happily talk to C, for historical reasons, B and C weren't completely open for communication. So my tip - make sure you've considered Active Directory Authentication and not just "DPM to Protected Server" issues!

Agents are "unavailable" and "VssError: Invalid value for registry"

This ia bit of an odd one and just "happened" on a previously perfectly happy server. We resolved this by simply removing the account used to push out the agents in the DCOM Config (run "dcomcnfg.exe"), find the "DPM RA" in the list and remove/readd the user. No idea what caused that mind!

Replica is inconsistent with System State and repeatedly so...

Especially if you're on a Windows 2003 SP-2 32-bit system? Yep, thought so. You've probably just not got enough space on the system drive (normally C:\). You should move the normally hidden "DPM_SYSTEM_STATE" folder to another drive, ideally with +10GB free, and then update the data source...

\Microsoft Data Protection Manager\DPM\datasources\PSdataSourceConfig.xml

change:

%SystemDrive%\DPM_SYSTEM_STATE\*

so it points to wherever you put it... easily sorted.


Hopefully they'll help you for now, more tips later!

DPM and "Secondary Protection" and "Chaining"

So first, the good news. Having rolled out DPM 2010 to our production environments, by and large all seems well, backups are completing, using less time, hassle and bandwidth overall than the previous Backup Exec solution.

It does seem to consume much high amounts of storage - but it isn't yet sufficiently clear if this is worthwhile yet (eg. if the space is pre-allocated so it can meet retention policies and then fills it, or it simply over-estimates likely requirements resulting in lots of unused capacity). We'll find out once we've run it a few weeks in a full production environment with realistic changes and replicas - and if needbe we'll tweak things a little.

Anyhow, I digress, so back to the purpose of this post... The next part of our rollout is to enable the "off site" capabilities - specifically making sure we have a second copy of each servers data at another site - you know for "total disasters".

This is called "DPM Chaining", "Secondary Protection" and various other things depending on the version of DPM, the documentation you read etc and what you are trying to achieve.

Basic steps are simple (after doing the normal DPM setup):

(a) On the second DPM server, push the protection agent to the first.

(b) On the first DPM server, push the protection agent to the second.

(c) On the second server, create protection groups, selecting the first dpm server as the data source, expanding "protected servers" and then treating it as if it was the first server.

(d) Complete the wizard, wait (a long time possibly) for replication to complete the first time.

We'll see how our trial run goes...

DPM 2010 - "Replica is Inconsistent" on 2008 Servers for Statem State

It would be fair to say DPM is proving to be far better than Backup Exec on most things, but occasionally there are some short sighted decisions or stupid issues that could have been better handled.

One small example is where you find System State and Bare Metal Recovery Replicas keep becoming inconsistent on a Windows 2008 system that's being backed up with DPM 2010.

The fix is pretty simple. On the 2008 server you're backing up, go to "Server Manager", load features, choose "Add Feature" and ensure "Windows Server Backup" is an allowed feature (this won't need a reboot).

Given DPM seems to check loads of other pre-requisites you'd expect it would either alert you to this at install time, or just enable it as part of the install (even if there was an option which said "If Windows Server Backup features are not enabled on the source for protection, enable it automatically" or something.

A silly oversight and one that just takes a tiny bit of the sparkle of clueful implementation away I think.

DPM - File Restores in Seconds, not minutes

As part of our deployment of Data Protection Manager (DPM) 2010, we decided we wanted to do as much restore testing as we could. So having contacted our usual customers who help us test and prove anything (call it a focus group if you want), we asked them all to delete random sets of files from the various servers we're backing up using DPM for them.

Obviously we asked them to make sure the files were not critical or important (just in case, safety first naturally!) - and then just tell us what files they wanted back. The theory being we should be able to do this without knowing in advance whats being deleted (ensuring nobody here could take extra backups or look out for anything etc).

Guess what, it worked... first time, and it is very fast. By comparison to Backup Exec, which took a minimum of 3-4 minutes even for a single 100KB Word Document (because of the whole loading media nonsense...), it did the job quickly, very quickly.

Where Backup Exec is more flexible however is if you want to restore a random set of files from a single file in different folders - DPM doesn't appear to let you do this - so I'd have to select files in a single folder, run "Recover..." then repeat for each folder (well through the UI anyhow). However, given the restore takes literally a few seconds, I'm not sure we care too much - and in reality doing this is pretty rare - normally we want a whole folder or a group of files in a folder or similar, rather than completely random odd and sods files from across a server.

Microsoft DPM - A breath of Fresh Air (almost!)

Having finally reached the end of our patience with Backup Exec and its never ending failures to simple requests, the terrible performance issues it suffers and all the other problems we hear about and witness every day, we decided to give Microsoft's Data Protection Manager a whirl.

There are a few important things to think about though if you are looking to switch, since Microsoft DPM is really only about Windows, SQL, Exchange and Sharepoint. If that's what you're running, and you're on 2003 SP-2 or 2008 and above, you should be fine. If you need other platforms and apps which Backup Exec supports you're probably out of luck using this.

Microsoft DPM is a very different product. One of the key differences is that it is truely snapshot based. Backup Exec still does far too much by using file by file methods - this has terrible scaling consequences.

It is mostly about Disk backup, whereas Backup Exec has a wider range of support for traditional tape backup. DPM can do it (it calls this "Long Term" Storage, and uses Disk for "Short Term" (you define what short/long term is...)

So in a nutshell (kind of) here's the story so far:

1) Installation of DPM failed because the install folder was "C:\!Software\DPM2010" whereas the installer ignored the existance of ! and tried to load "C:\Software\DPM2010" and couldn't find its own files. So we just put up with that and put DPM2010 in the c:\ folder root so we could get started.

2) Installation takes a while as it also rolls out SQL 2008 (you can get it to use an existing Database but we opted not to - and this is the recommended approach).

3) Take time to read the pre-req's and understand how DPM works. For example, make sure you have a huge volume on each DPM server (the best scenario) you have left unformatted so it can claim this for itself.

With those basics covered, the initial installation was completely succesful and our first DPM server appeared.

Microsoft Data Protection Manager

So we've reached the point after many years where we want to reduce our use of Backup Exec. Mainly because it is stupidly expensive and just not reliable enough.

So we figured we'd give Microsoft Data Protection Manager a go. Full of optimism, we began the install. It failed at the first hurdle.

You see the software was in a folder "C:\!Software\DPMServer2010"

Except the installer decided that is actually "C:\Software\DPMServer2010"

So although ! is a perfectly valid File System Character, the DPM Installer failed.

Folder renamed and it worked.

It isn't a good start... this is the sort of stupidity Backup Exec had!

72 Hours with 100%

Yes, in what can be described as some sort of Backup Exec miracle, it has managed to run for 72 hours with 100% of jobs completing successfully across multiple servers via CASO. Only took 5 years of fiddling to get it to run smoothly for a bit, what a result.

I still really don't like Backup Exec much.

Follow your own advice...

Recently I had an issue.

CASO couldn't talk to all of the managed media servers (sort of making them unmanaged then...) Much muttering and fiddling later and nothing.

So I try and think how to describe the exact issue, and google for it (naturally you google last after messing around because that would be akin otherwise to actually reading the manual that comes with stuff you buy)...

First match... er, this blog.

I refer myself to my own sodding blog to fix an issue. The post in question, June 8th 2009...

Backup Exec Licensing

Am I the only person who thinks the licensing scheme of Backup Exec is insane.

Why must there be so many options which become increasingly expensive for features that should just be included, and why exactly must we pay for enhancements which are often essentially fixes for features they never implemented properly the first time.

Grrrrrrrrr

Reporting in Backup Exec ... no more pain and misery...

If, like me you manage a largeish Backup Exec installation with several media servers, hundreds of backups and lots of clients, you'll probably be pretty frustrated with the half assed nature of the Backup Exec logging and reporting capabilities.

For a long time, I've wanted a simple, but powerful way to do things like "show me backups that are consistently failing over 'x' period, or show me the most likely time of day for backup jobs to fail etc.

So, having looked everywhere and found no sane solution, I've just started writing one. Now I have a great little interface where I can review my backups, see what jobs are failing constantly, review the issue, fix it and then mark it as resolved so it can start being checked again.

I'm thinking of adding lots of features and eventually making it something I can sell for a reasonable (read: not outrageous) fee to others who feel the pain...

Any suggestions welcomed...

Error E000FE30 every day on one server...

...for months. For months I've struggled with a problem on ONE server, that happens to be at a remote site on a different subnet, connected via a WAN VPN Link.

Every day, one or more jobs would fail with Backup Exec Errors, mainly E000FE30 - with the useful and generic messages about "communications failure has occured" and sometimes the "connection lost to the remote agent".

Needless to say, I've spent some time working on this, and tried all sorts. Reconfiguring the system to use a different WAN link to ensure the fault isn't with the WAN. Nothing. Checking to ensure the issue isn't with the server, reinstalling agents, trying all sorts.

I've updated network drivers, checked all sorts of patches etc - but nothing, Still this error - consistently failing jobs.

I even got a colleague to look at it for a fresh pair of eyes and he too tried all sorts. Given the error, we suspected "something" to do with comms, but never found any issue, and in hundreds of tests conducted could never replicate the issue - transferring large files to/fro the server worked fine etc.

Today I found the answer. The "Large TCP Offload" feature on the Network Card. While I've seen plenty of issues with this feature before, you normally see it with terrible throughput on the system in general and so on - but this machine is solid as a rock for everything else.

Still, the setting is off, and first complete, full backups in a few weeks... voila!

Top tip for anyone else facing this problem - don't just check the network drivers, but try turning off these features, even if you cannot see this issue at any other time on the machine.

Is this a Backup Exec issue? I'm not sure, but I'm happy to blame it since everything else works just fine.

Debugging Mode for Remote Agent...

Right now I have a problem where a customers server just doesn't backup. Not for love nor money. The connection is over a WAN type connection, so not your average setup, but none the less, working fine previously.

First of all we suspected the comms - e.g. VPN, the Routers, the provider connection, and did the usual testing to be sure that isn't the cause. We saw a couple things that made us "think" the isuse was there but nothing too worrying.

Not a problem though, we've got multiple WAN links, and another way to get to the other end. A couple config changes, and the route now uses a different WAN link at both ends (e.g. BEWS Media Server and RAWS Enabled Server being backed up).

Problem still exists. So it's not comms then (given the other links switched to backup another server at the same site daily (40-60Gb/day without complaint).

So the issue is likely something with the server being backed up.

The point of the post though, is to let you know how to temporarily enable the debug backup logging on the Remote Agent (RAWS).

Stop the service, add "-debug" (no quotes) to the "startup parameters" in the services management in Admin Tools (or start > run > services.msc and hit enter...)

Start the service. You're good for logs until the next restart of services/server reboot.

Logs go in your backup exec install directory on the server being backed up, in the Logs subfolder named beremote... something...

(Oh yeah, and a warning, the debug logs are HUGEEEEEE)

Backup saves your ass, Microsoft kicks it hard

Today has been an interesting day where I learned that our backup platform is joy.

The basic scenario is pretty simple. Customer server dies and our on call guy says we need to restore.

Drop a clean Win2003 box onto our virtual platform, run a restore, bingo, first time success. Reboot the machine and...


windows product activation kicks in begging for reactivation. Many hours of hassle
later and no help from the so called 'parter critical support line', which seemed bothered
only about making statistics up about why I was calling, I get the issue resolved myself.

the moral of the story? backup exec works fine and Microsoft Product Activation smacks. Again.

So it just died....

About a week ago our Backup Exec CASO box decided it had had enough of talking to Managed Media Servers. Randomly declaring known working servers to be "unavailable".

The usual checks started, and nothing. Patches checked, removed etc. Nothing.

The solution. Search for any msgq*.dat files on your servers. Delete them.

Voila. Everything works... I'm not happy...

Still here, still managing Backup Exec every day...

It's been some months since we've postted here, and the foolish (and those who never actually deal with Backup Exec) would probably believe that's because we've eithe forgotten about the blog, or we've got everything working.

You'd be wrong.

It's true to say that we've got a little more success, and now have 5 10d boxes and 1 12d box, all running, and, most of the time it tends to play well. Which is lovely, but when it does go wrong, it tends to lose it completely.

Here are a few problems we've currently got:

a) An old "Managed Media Server" that is long since departed just won't go away from all parts of the UI - most of it knows it has gone, but some parts still show it, as if it may somehow come back one day. It won't.

b) 2-3 Jobs are stuck in an external status where they're on "On Hold, Running" according to the status. That's not true. In fact, they've been stuck there for a year. Meaning I can't delete the now-redundant Policies, Templates or Selection Lists for those jobs. They're just stuck there forever.

c) Sometimes a job fails claiming the cause to be a Communications Failure. Communications Failure is Backup Exec speak for "most problems". There is naff all wrong with the communications, and normally we resolve it by deleting/recreating the job.

d) Synthetic jobs, well let's see. They suck. They only work in exacting circumstances, and the minute you step out of line of one of those or a job is missed, well that's your life made hell. They start failing, come up with lots of silly errors and you end up re-creating them, waiting for a full again. So you tend to not bother, and just do an old-style Full/Incremental set, since they normally work.

In the case of 12d, it does tend to be a little better, particularly with Exchange Backups. Except it STILL doesn't properly manage media, so the IMG foldes it creates don't always get deleted (although it reckons they will). Still no joy on having the B2D Files reused or deleted. Hell no, that'd make sense.

So yeah, I'm still here, managing Backup Exec, 7 days a week, doing what it should do for it, and going mental every time I come in to find it's just collapsed without warning. Quality it is not.

Sadly I've still yet to find a better solution at a price point that is sane.

Why are error messages not unique

That's what I want to know.

Why is it that you get an error message in Backup Exec, and it spits out an error for you, and so you click on it, which takes you to a web page with a description of that error, right?

Wrong. With Symantec Backup Exec, it just takes you to a list of issues which may or may not be remotely close to what you have issues with, and rarely has any useful answer.

Here I am today again trying to find out why certain jobs keep failing without any sort of sane error reason. Another day fighting Backup Exec.

Server Paused. In a not-actually-paused-at-all kinda way

For the past 2-3 weeks our CASO Backup Exec Server has been INSISTING that the "Job Status" for every single job is "Server Paused". While it isn't actually uncommon for that to be the case, and certainly I've seen a server have this status from time to time, it isn't the case for days and not on all of our media servers.

It seems the answer is simple (but annoying)... here's what a Symantec Forum post says:

----------------
I have had this problem many times. Like a plague. The fix most times is simple. Go to the devices tab. Highlght your media server. Right click and pause it. Then unpause it. This is caused by an interruption which corrupts a file. Backup Exec shows nothing paused. Just fixed this today after 10 days following a problem with a tape drive. Symantec support knows about it but hasn't published the fix.
----------------

A few months later...

Backup Exec 10d continues to offer variable results. For straightforward jobs it tends to do a reasonable job, most of the time.

However, each time we add a set of Synthetic jobs to the mix, things start going horribly wrong, and the backup exec engine regularly dies.

While I currently refuse to pay any money for upgrades since they promised these features were part of 10d, and they simply don't work, I have had more luck with an eval of 12d. Of course until we've been running it as long as 10d and it has the same sort of load we won't know for sure, but I don't hold out much hope!

Event numbers

So, now that we've managed *touch wood* to iron out the big issues that have been plaguing us for ages, perhaps it's time to look at some of the more annoying niggles we have to contend with.

I've always thought that many event numbers produced by most applications were simply too short and simple, after all, how can you hope to cover all the eventualities with only 4/5 characters. With the change in Backup Exec in recent versions to much longer event numbers you would think they could tie things down much more specifically, but apparently not.

Looking at an error we've been seeing recently, we're getting this when running a backup :

V-79-57344-33938 - An error occurred on a query to database .

So you might think with a possible 1,000,000,000,000 event numbers available that this error would be specific to the problem I'm seeing, and allow me to find more information directly relating to it... of course you'd be wrong.

Clicking on the link provided by the Job History shows me 9 different articles all apparently relating to this one error. 6 of them refer to restore jobs rather than backup jobs, 2 relate to SQL 2005 not 2000 as is the case in this instance, 2 relate to running SQL 7 alongside 2000 which we're not doing and of the 2 which do relate to backups in their titles, the contents either refer to SQL 7 or to you having an incomplete restore operation preventing the backup from running.

Now I realise that producing written content for up to a trillion error numbers would be a mammoth task, but personally if I'm searching for a specific event number and no content exists for it yet I'd prefer to know that, rather than trawl through mountains of knowledge base articles in the hope that one might actually be relevant to my situation! I can always then search using words from the error message to find articles that are similar, and try things from there!