Microsoft Data Protection Heaven and Backup Exec Hell

Sometimes you want Backup Exec back...

2011-06-13T10:38:00.001+01:00

So some weeks on and DPM on the whole is everything a backup product should be. There are a few annoyances and a few things that need radical improvement, but on the most important factor - backup reliability, DPM wins hands down.

One area though that Backup Exec was MUCH better at is E-Mail Alerting. Firstly, it was more flexible - any SMTP server was OK, and that worked great for us. DPM however only seems to work if it's pointed at an Exchange based environment - which was a bit annoying since that's not really how I wanted it done. I guess that's the side effect of the "optimal for microsoft based workloads" strategy, but nonetheless...

The other bit though is the alerting capability. You can have alerts for 3 categories "Informational" "Warning" and "Critical", and a list of e-mail addresses to send to. You get one list of e-mail addresses and ALL of those addresses can receive the alerts you enable.

You can't set any thresholds, you can't customise the alerts and most annoyingly, it alerts you to both "Problems" and "Resolved". Given it tries to self resolve I was hoping "Critical" would only alert you to issues it has tried to resolve and failed at or cannot resolve because it needs our intervention.

All in all a bit poor and makes me want BEWS back, just for that bit anyhow...

Another Top Tip for DPM - "replica is inconsistent" on System State and Bare Metal Recovery...

2011-05-12T14:09:00.000+01:00

I thought I'd share this Top Tip. It is reasonably well documented, but a classic thing you'd often miss.

If you're finding that "System State" and "Bare Metal Recovery" items are frequently sitting in a "replica is inconsistent" state ...which happens a lot on Windows 2008 system... then the chances are you've not got "Windows Server Backup" as an installed feature on the server you're backing up (the protected server).

It's dead simple to sort, run Server Manager, click Add Feature, check "Windows Server Backup" and wait for it to install - job done - run a consistency check and they'll be sorted and work thereafter.

Of course, why the DPM installer doesn't just install this (or at least prompt) as part of the roll out since it is basically a dependency is anyones guess...

Common DPM Errors...

2011-05-06T20:50:00.003+01:00

Since we've now got most of the Data Protection Manager 2010 installations done, I thought I'd share a few common issues we've come across, and the fixes. Maybe this'll save you a LOT of hassle...

"Access Denied (0x80070005)"
Common causes are listed all over the place, suggesting Firewalls as the issues and DCOM Permissions. All entirely possible. One other thing to consider, especially if you've setup Forest Trusts etc, just make sure you've made sure the AD Network holding your DPM Server(s) is fully accessible - and that this traffic isn't restricted either! In our case, we had a Cluster with 2 servers, one in a Subnet (we'll call this Subnet A), another in a different subnet (Subnet B) and our DPM Servers (and the DPM AD Network) in another (Subnet C).

While Subnet A and B could talk without restriction, and A could quite happily talk to C, for historical reasons, B and C weren't completely open for communication. So my tip - make sure you've considered Active Directory Authentication and not just "DPM to Protected Server" issues!

Agents are "unavailable" and "VssError: Invalid value for registry"

This ia bit of an odd one and just "happened" on a previously perfectly happy server. We resolved this by simply removing the account used to push out the agents in the DCOM Config (run "dcomcnfg.exe"), find the "DPM RA" in the list and remove/readd the user. No idea what caused that mind!

Replica is inconsistent with System State and repeatedly so...

Especially if you're on a Windows 2003 SP-2 32-bit system? Yep, thought so. You've probably just not got enough space on the system drive (normally C:\). You should move the normally hidden "DPM_SYSTEM_STATE" folder to another drive, ideally with +10GB free, and then update the data source...

\Microsoft Data Protection Manager\DPM\datasources\PSdataSourceConfig.xml

change:

%SystemDrive%\DPM_SYSTEM_STATE\*

so it points to wherever you put it... easily sorted.

Hopefully they'll help you for now, more tips later!

DPM and "Secondary Protection" and "Chaining"

2011-05-05T13:16:00.003+01:00

So first, the good news. Having rolled out DPM 2010 to our production environments, by and large all seems well, backups are completing, using less time, hassle and bandwidth overall than the previous Backup Exec solution.

It does seem to consume much high amounts of storage - but it isn't yet sufficiently clear if this is worthwhile yet (eg. if the space is pre-allocated so it can meet retention policies and then fills it, or it simply over-estimates likely requirements resulting in lots of unused capacity). We'll find out once we've run it a few weeks in a full production environment with realistic changes and replicas - and if needbe we'll tweak things a little.

Anyhow, I digress, so back to the purpose of this post... The next part of our rollout is to enable the "off site" capabilities - specifically making sure we have a second copy of each servers data at another site - you know for "total disasters".

This is called "DPM Chaining", "Secondary Protection" and various other things depending on the version of DPM, the documentation you read etc and what you are trying to achieve.

Basic steps are simple (after doing the normal DPM setup):

(a) On the second DPM server, push the protection agent to the first.

(b) On the first DPM server, push the protection agent to the second.

(c) On the second server, create protection groups, selecting the first dpm server as the data source, expanding "protected servers" and then treating it as if it was the first server.

(d) Complete the wizard, wait (a long time possibly) for replication to complete the first time.

We'll see how our trial run goes...

DPM 2010 - "Replica is Inconsistent" on 2008 Servers for Statem State

2011-04-28T16:51:00.003+01:00

It would be fair to say DPM is proving to be far better than Backup Exec on most things, but occasionally there are some short sighted decisions or stupid issues that could have been better handled.

One small example is where you find System State and Bare Metal Recovery Replicas keep becoming inconsistent on a Windows 2008 system that's being backed up with DPM 2010.

The fix is pretty simple. On the 2008 server you're backing up, go to "Server Manager", load features, choose "Add Feature" and ensure "Windows Server Backup" is an allowed feature (this won't need a reboot).

Given DPM seems to check loads of other pre-requisites you'd expect it would either alert you to this at install time, or just enable it as part of the install (even if there was an option which said "If Windows Server Backup features are not enabled on the source for protection, enable it automatically" or something.

A silly oversight and one that just takes a tiny bit of the sparkle of clueful implementation away I think.

DPM - File Restores in Seconds, not minutes

2011-04-27T17:11:00.002+01:00

As part of our deployment of Data Protection Manager (DPM) 2010, we decided we wanted to do as much restore testing as we could. So having contacted our usual customers who help us test and prove anything (call it a focus group if you want), we asked them all to delete random sets of files from the various servers we're backing up using DPM for them.

Obviously we asked them to make sure the files were not critical or important (just in case, safety first naturally!) - and then just tell us what files they wanted back. The theory being we should be able to do this without knowing in advance whats being deleted (ensuring nobody here could take extra backups or look out for anything etc).

Guess what, it worked... first time, and it is very fast. By comparison to Backup Exec, which took a minimum of 3-4 minutes even for a single 100KB Word Document (because of the whole loading media nonsense...), it did the job quickly, very quickly.

Where Backup Exec is more flexible however is if you want to restore a random set of files from a single file in different folders - DPM doesn't appear to let you do this - so I'd have to select files in a single folder, run "Recover..." then repeat for each folder (well through the UI anyhow). However, given the restore takes literally a few seconds, I'm not sure we care too much - and in reality doing this is pretty rare - normally we want a whole folder or a group of files in a folder or similar, rather than completely random odd and sods files from across a server.

Microsoft DPM - A breath of Fresh Air (almost!)

2011-04-26T15:43:00.002+01:00

Having finally reached the end of our patience with Backup Exec and its never ending failures to simple requests, the terrible performance issues it suffers and all the other problems we hear about and witness every day, we decided to give Microsoft's Data Protection Manager a whirl.

There are a few important things to think about though if you are looking to switch, since Microsoft DPM is really only about Windows, SQL, Exchange and Sharepoint. If that's what you're running, and you're on 2003 SP-2 or 2008 and above, you should be fine. If you need other platforms and apps which Backup Exec supports you're probably out of luck using this.

Microsoft DPM is a very different product. One of the key differences is that it is truely snapshot based. Backup Exec still does far too much by using file by file methods - this has terrible scaling consequences.

It is mostly about Disk backup, whereas Backup Exec has a wider range of support for traditional tape backup. DPM can do it (it calls this "Long Term" Storage, and uses Disk for "Short Term" (you define what short/long term is...)

So in a nutshell (kind of) here's the story so far:

1) Installation of DPM failed because the install folder was "C:\!Software\DPM2010" whereas the installer ignored the existance of ! and tried to load "C:\Software\DPM2010" and couldn't find its own files. So we just put up with that and put DPM2010 in the c:\ folder root so we could get started.

2) Installation takes a while as it also rolls out SQL 2008 (you can get it to use an existing Database but we opted not to - and this is the recommended approach).

3) Take time to read the pre-req's and understand how DPM works. For example, make sure you have a huge volume on each DPM server (the best scenario) you have left unformatted so it can claim this for itself.

With those basics covered, the initial installation was completely succesful and our first DPM server appeared.

Microsoft Data Protection Manager

2011-04-18T23:35:00.001+01:00

So we've reached the point after many years where we want to reduce our use of Backup Exec. Mainly because it is stupidly expensive and just not reliable enough.

So we figured we'd give Microsoft Data Protection Manager a go. Full of optimism, we began the install. It failed at the first hurdle.

You see the software was in a folder "C:\!Software\DPMServer2010"

Except the installer decided that is actually "C:\Software\DPMServer2010"

So although ! is a perfectly valid File System Character, the DPM Installer failed.

Folder renamed and it worked.

It isn't a good start... this is the sort of stupidity Backup Exec had!

72 Hours with 100%

2010-10-23T21:14:00.003+01:00

Yes, in what can be described as some sort of Backup Exec miracle, it has managed to run for 72 hours with 100% of jobs completing successfully across multiple servers via CASO. Only took 5 years of fiddling to get it to run smoothly for a bit, what a result.

I still really don't like Backup Exec much.

Follow your own advice...

2010-10-06T09:10:00.002+01:00

Recently I had an issue.

CASO couldn't talk to all of the managed media servers (sort of making them unmanaged then...) Much muttering and fiddling later and nothing.

So I try and think how to describe the exact issue, and google for it (naturally you google last after messing around because that would be akin otherwise to actually reading the manual that comes with stuff you buy)...

First match... er, this blog.

I refer myself to my own sodding blog to fix an issue. The post in question, June 8th 2009...

Backup Exec Licensing

2009-12-29T16:00:00.001Z

Am I the only person who thinks the licensing scheme of Backup Exec is insane.

Why must there be so many options which become increasingly expensive for features that should just be included, and why exactly must we pay for enhancements which are often essentially fixes for features they never implemented properly the first time.

Grrrrrrrrr

Reporting in Backup Exec ... no more pain and misery...

2009-11-10T12:02:00.002Z

If, like me you manage a largeish Backup Exec installation with several media servers, hundreds of backups and lots of clients, you'll probably be pretty frustrated with the half assed nature of the Backup Exec logging and reporting capabilities.

For a long time, I've wanted a simple, but powerful way to do things like "show me backups that are consistently failing over 'x' period, or show me the most likely time of day for backup jobs to fail etc.

So, having looked everywhere and found no sane solution, I've just started writing one. Now I have a great little interface where I can review my backups, see what jobs are failing constantly, review the issue, fix it and then mark it as resolved so it can start being checked again.

I'm thinking of adding lots of features and eventually making it something I can sell for a reasonable (read: not outrageous) fee to others who feel the pain...

Any suggestions welcomed...

Error E000FE30 every day on one server...

2009-09-30T11:00:00.004+01:00

...for months. For months I've struggled with a problem on ONE server, that happens to be at a remote site on a different subnet, connected via a WAN VPN Link.

Every day, one or more jobs would fail with Backup Exec Errors, mainly E000FE30 - with the useful and generic messages about "communications failure has occured" and sometimes the "connection lost to the remote agent".

Needless to say, I've spent some time working on this, and tried all sorts. Reconfiguring the system to use a different WAN link to ensure the fault isn't with the WAN. Nothing. Checking to ensure the issue isn't with the server, reinstalling agents, trying all sorts.

I've updated network drivers, checked all sorts of patches etc - but nothing, Still this error - consistently failing jobs.

I even got a colleague to look at it for a fresh pair of eyes and he too tried all sorts. Given the error, we suspected "something" to do with comms, but never found any issue, and in hundreds of tests conducted could never replicate the issue - transferring large files to/fro the server worked fine etc.

Today I found the answer. The "Large TCP Offload" feature on the Network Card. While I've seen plenty of issues with this feature before, you normally see it with terrible throughput on the system in general and so on - but this machine is solid as a rock for everything else.

Still, the setting is off, and first complete, full backups in a few weeks... voila!

Top tip for anyone else facing this problem - don't just check the network drivers, but try turning off these features, even if you cannot see this issue at any other time on the machine.

Is this a Backup Exec issue? I'm not sure, but I'm happy to blame it since everything else works just fine.

Debugging Mode for Remote Agent...

2009-08-18T20:02:00.002+01:00

Right now I have a problem where a customers server just doesn't backup. Not for love nor money. The connection is over a WAN type connection, so not your average setup, but none the less, working fine previously.

First of all we suspected the comms - e.g. VPN, the Routers, the provider connection, and did the usual testing to be sure that isn't the cause. We saw a couple things that made us "think" the isuse was there but nothing too worrying.

Not a problem though, we've got multiple WAN links, and another way to get to the other end. A couple config changes, and the route now uses a different WAN link at both ends (e.g. BEWS Media Server and RAWS Enabled Server being backed up).

Problem still exists. So it's not comms then (given the other links switched to backup another server at the same site daily (40-60Gb/day without complaint).

So the issue is likely something with the server being backed up.

The point of the post though, is to let you know how to temporarily enable the debug backup logging on the Remote Agent (RAWS).

Stop the service, add "-debug" (no quotes) to the "startup parameters" in the services management in Admin Tools (or start > run > services.msc and hit enter...)

Start the service. You're good for logs until the next restart of services/server reboot.

Logs go in your backup exec install directory on the server being backed up, in the Logs subfolder named beremote... something...

(Oh yeah, and a warning, the debug logs are HUGEEEEEE)

Backup saves your ass, Microsoft kicks it hard

2009-06-27T17:58:00.002+01:00

Today has been an interesting day where I learned that our backup platform is joy.

The basic scenario is pretty simple. Customer server dies and our on call guy says we need to restore.

Drop a clean Win2003 box onto our virtual platform, run a restore, bingo, first time success. Reboot the machine and...

windows product activation kicks in begging for reactivation. Many hours of hassle
later and no help from the so called 'parter critical support line', which seemed bothered
only about making statistics up about why I was calling, I get the issue resolved myself.

the moral of the story? backup exec works fine and Microsoft Product Activation smacks. Again.

So it just died....

2009-06-08T18:03:00.002+01:00

About a week ago our Backup Exec CASO box decided it had had enough of talking to Managed Media Servers. Randomly declaring known working servers to be "unavailable".

The usual checks started, and nothing. Patches checked, removed etc. Nothing.

The solution. Search for any msgq*.dat files on your servers. Delete them.

Voila. Everything works... I'm not happy...

Still here, still managing Backup Exec every day...

2009-03-08T16:56:00.002Z

It's been some months since we've postted here, and the foolish (and those who never actually deal with Backup Exec) would probably believe that's because we've eithe forgotten about the blog, or we've got everything working.

You'd be wrong.

It's true to say that we've got a little more success, and now have 5 10d boxes and 1 12d box, all running, and, most of the time it tends to play well. Which is lovely, but when it does go wrong, it tends to lose it completely.

Here are a few problems we've currently got:

a) An old "Managed Media Server" that is long since departed just won't go away from all parts of the UI - most of it knows it has gone, but some parts still show it, as if it may somehow come back one day. It won't.

b) 2-3 Jobs are stuck in an external status where they're on "On Hold, Running" according to the status. That's not true. In fact, they've been stuck there for a year. Meaning I can't delete the now-redundant Policies, Templates or Selection Lists for those jobs. They're just stuck there forever.

c) Sometimes a job fails claiming the cause to be a Communications Failure. Communications Failure is Backup Exec speak for "most problems". There is naff all wrong with the communications, and normally we resolve it by deleting/recreating the job.

d) Synthetic jobs, well let's see. They suck. They only work in exacting circumstances, and the minute you step out of line of one of those or a job is missed, well that's your life made hell. They start failing, come up with lots of silly errors and you end up re-creating them, waiting for a full again. So you tend to not bother, and just do an old-style Full/Incremental set, since they normally work.

In the case of 12d, it does tend to be a little better, particularly with Exchange Backups. Except it STILL doesn't properly manage media, so the IMG foldes it creates don't always get deleted (although it reckons they will). Still no joy on having the B2D Files reused or deleted. Hell no, that'd make sense.

So yeah, I'm still here, managing Backup Exec, 7 days a week, doing what it should do for it, and going mental every time I come in to find it's just collapsed without warning. Quality it is not.

Sadly I've still yet to find a better solution at a price point that is sane.

Why are error messages not unique

2008-10-31T15:48:00.002Z

That's what I want to know.

Why is it that you get an error message in Backup Exec, and it spits out an error for you, and so you click on it, which takes you to a web page with a description of that error, right?

Wrong. With Symantec Backup Exec, it just takes you to a list of issues which may or may not be remotely close to what you have issues with, and rarely has any useful answer.

Here I am today again trying to find out why certain jobs keep failing without any sort of sane error reason. Another day fighting Backup Exec.

Server Paused. In a not-actually-paused-at-all kinda way

2008-09-17T23:19:00.002+01:00

For the past 2-3 weeks our CASO Backup Exec Server has been INSISTING that the "Job Status" for every single job is "Server Paused". While it isn't actually uncommon for that to be the case, and certainly I've seen a server have this status from time to time, it isn't the case for days and not on all of our media servers.

It seems the answer is simple (but annoying)... here's what a Symantec Forum post says:

----------------
I have had this problem many times. Like a plague. The fix most times is simple. Go to the devices tab. Highlght your media server. Right click and pause it. Then unpause it. This is caused by an interruption which corrupts a file. Backup Exec shows nothing paused. Just fixed this today after 10 days following a problem with a tape drive. Symantec support knows about it but hasn't published the fix.
----------------

A few months later...

2008-09-12T10:10:00.002+01:00

Backup Exec 10d continues to offer variable results. For straightforward jobs it tends to do a reasonable job, most of the time.

However, each time we add a set of Synthetic jobs to the mix, things start going horribly wrong, and the backup exec engine regularly dies.

While I currently refuse to pay any money for upgrades since they promised these features were part of 10d, and they simply don't work, I have had more luck with an eval of 12d. Of course until we've been running it as long as 10d and it has the same sort of load we won't know for sure, but I don't hold out much hope!

Event numbers

2008-02-26T15:44:00.001Z

So, now that we've managed *touch wood* to iron out the big issues that have been plaguing us for ages, perhaps it's time to look at some of the more annoying niggles we have to contend with.

I've always thought that many event numbers produced by most applications were simply too short and simple, after all, how can you hope to cover all the eventualities with only 4/5 characters. With the change in Backup Exec in recent versions to much longer event numbers you would think they could tie things down much more specifically, but apparently not.

Looking at an error we've been seeing recently, we're getting this when running a backup :

V-79-57344-33938 - An error occurred on a query to database .

So you might think with a possible 1,000,000,000,000 event numbers available that this error would be specific to the problem I'm seeing, and allow me to find more information directly relating to it... of course you'd be wrong.

Clicking on the link provided by the Job History shows me 9 different articles all apparently relating to this one error. 6 of them refer to restore jobs rather than backup jobs, 2 relate to SQL 2005 not 2000 as is the case in this instance, 2 relate to running SQL 7 alongside 2000 which we're not doing and of the 2 which do relate to backups in their titles, the contents either refer to SQL 7 or to you having an incomplete restore operation preventing the backup from running.

Now I realise that producing written content for up to a trillion error numbers would be a mammoth task, but personally if I'm searching for a specific event number and no content exists for it yet I'd prefer to know that, rather than trawl through mountains of knowledge base articles in the hope that one might actually be relevant to my situation! I can always then search using words from the error message to find articles that are similar, and try things from there!

Reliability? Can we say we're onto Stage 2?

2008-01-14T08:44:00.000Z

I'm rather scared, and pleased, to say that we appear to have genuinely made it work - in my last few posts I was still sceptical that Backup Exec was just being nice to us, but it does appear it is genuinely now acting like a competent backup product.

One server has now been up for 27 days, and is still running backups quite happily, with 2,000 odd jobs run in that time (66 a day or so), and the CASO server just getting on with it's jobs and delegation. No more tears.

Our CPS system is still working - that's the most flawless part of Backup Exec I've seen. It's absolutely awesome - it says continuous protection, and it offers just that. We installed it, got over one hurdle of making it listen on a specific IP, and that was that. Our main file servers have just-below-real-time backups 24/7.

Our next challenge (Stage 2, which took 18 months to get to!) is to build a comprehensive reporting and restore testing infrastructure around the software. We want to know everything possible about what it does, so we can provide internal quality reports, ensure we meet SLAs and finally, but not least, ensure customers can be assured of regular, reliable backups.

Restore testing is often overlooked, but certainly not here! It is an essential, and core part of our plan to operate regular (hopefully scheduled) restore tests so we can be sure those backups actually work - as someone else we know found out to great cost - backing up isn't enough!

Forgotten about this blog? Not at all

2007-12-29T21:12:00.001Z

But the problem is Backup Exec does appear to largely work reliably now - at the moment we're on 12 days 5 hours. It's hardly something you can call "reliable to 5 9's (e.g. 99.999% reliable), but it's certainly better than every few hours!

Just to recap the top tips...

1) Don't set more than a handful of jobs to run at once. They won't.

2) Although policies and templates are good ideas, expect to create lots of them because of tip 1.

3) Have plenty of Disk to Disk Folders, and set no more than 1 or 2 jobs per set as the concurrent limit.

4) Make good use of the Disk Reserve settings to ensure it has no excuse to run out of space and fail your jobs for weeks.

5) Set sane retention policies (e.g. don't just set it to weeks for the sake of it)

6) Split jobs by location, server, and drive.

7) Backup Exchange or SQL (and other "Agent" options) as distinct jobs

8) Keep your eye on Backup Exec at all times.

And now, for CPS...

2007-11-23T09:17:00.000Z

Something truely amazing has started happening, as Backup Exec is *still* running, 6 days in, and it looks like we may have turned a corner with it in terms of making it think, act and work, like a competent backup solution.

As a result we've been able to look at other things - since my colleague tried the "CPS" or Continuous Protection Server part of Backup Exec some time ago, we found it was pretty good stuff - it worked straight out of the box, so we figured we'd give it a try - after all it would be more than a little useful if we could just replicate data from one site to another as part of a backup system, as it would boost our protection against failure of one of our key servers.

Installation of the CPS Server was easy, and worked first time. However, getting it to talk to our remote server was a little more difficult, as it's on a different subnet, and, as a bonus, the CPS Server also happens to have multiple NICs on different networks. To annoy us, CPS decided to pick the wrong IP to bind to (and there's no indication it will do this, nor any GUI to choose it).

It's OK though, easily fixed - with a famous registry edit to set a "PreferredAddress" on the CPS Server, and, on the CPS Machine being copied, changing it's "Gateway" address for the Veritas Software to be the IP of the CPS machine and not it's host name (even though it resolves to the same).

Right now we have a working CPS job doing the initial copy of our data, after which it should continuously protect... rock on.

I'm still in a dream

2007-11-20T10:06:00.000Z

I must be. Because we've still got 3 working Backup Exec Media Servers, CASO is working, backups are working, and, with a couple exceptions they all run.

We've got 2 troublesome jobs which fail as they just have issues spitting the files out to the Media Servers and we'll work on those, and one that just fails on System State every day despite Windows itself being able to back it up, Backup Exec insists that "a failure occurred reading an object" and later claims that "shadow?copy?components" is a corrupt file.

So right now, we've had 3 days and 14 hours of running backups - you know we may even get to run test restores because it's not crashed.

Did we hit April 1st? What has been done with the Real Backup Exec?

2007-11-18T22:46:00.000Z

In a brave move, given the near 3 day successful run with Backup Exec over last week, and given that I was more interested in seeing Bill Bailey live over the weekend than staring into the "Alerts" list on Backup Exec, I figured we'd just see if it could cope again.

Amazingly, it has. So far. It's now late Sunday evening but everything is running, and, to top it I've got 3 media servers online, running, and doing what they're designed for. We've even found time to expand the storage on the smallest of the servers from around 1Tb to 3Tb ready for when it starts to get a real workload.

It was too good to be true.

2007-11-16T19:45:00.001Z

Backups have run for 2 days, 23 hours, and just crashed I actually thought we'd make 3 days then.

It appears that "BECatSrv.dll" decided to give up and die.

Now to see if it recovers...

"d" is for "tape"

2007-11-15T16:55:00.000Z

Something is wrong. Very wrong. Backup Exec is still working. That's over a day. It's not crashed or had a funny 10 minutes where it doesn't work. None of this "server paused" rubbish. Still meanwhile in the world of the admin's maintaining it, we've had a good old debate about how the whole thing is supposed to work, how symantec interpret "tape" backups when they're actually disk backups.

The only thing we did resolve is that the "d" in Backup Exec 10d means "tape". Yes. it means Tape. Which is probably how Symantec would have preferred we kept things.

Eye Tests

2007-11-15T08:49:00.000Z

I'm starting to think I need an eye test. In our office I'm the only one with good eyesight that doesn't already wear glasses. Amazingly though, I think that may have to change.

My rationale is that this morning I've opened the old Backup Exec status window, yet there isn't a Server Paused, Loading Media, Queued or other failure at all, and over the past 24 hours we have just 13 out of 250 odd jobs failed, and the 13 are largely "missed" backups which just mean I need to refine the windows so there is enough time for them all to run (I never get that far normally as it crashes so everything's "missed".

So yeah, either my eyes are playing tricks on me, or Backup Exec has performed a minor miracle.

Making it work: Tip #2

2007-11-14T22:53:00.000Z

One of my pet hates of Backup Exec is it's rather poor disk space management for Backup To Disk Folders. There are 2 approaches to Disk Space Management, the Sane Way, and the Stupid Way.

Guess which Backup Exec chooses...

As you can't set a limit on the amount of space a Backup To Disk volume can occupy (almost like a LTO-1 tape has a 100Gb storage capacity, and that's that, if you run out, you run out), you'd expect to be able to do similar on Backup Exec. But while you can tell it how big an individual Backup To Disk FILE is, you can't set a limit on the total space a "Disk" can occupy. The limit of course is the physical space on the drive itself.

The "workaround" is that if you had a drive, let's say it's 500Gb, and you have 6 Backup to Disk "Folders" on that drive. Set each one to have a Disk "Reserve" of something (the default is "nothing", naturally), let's say 10Gb. When you first use Backup Exec, it'll just keep making "B2Dnnnnn" files, but once you get to <10Gb left on the PHYSICAL drive, the Backup Exec Folders will then theoretically start to be overwritten, preventing the problem of running out of space completely.

Follow that? No. OK, try this:

PHYSICAL DRIVE (we'll call it Drive E:) has 500Gb Space, and it's only being used by Backup Exec.

You have 6 folders, each set with 10Gb "Disk Reserve":

Folder A, Folder B etc

Roll forward 2 weeks...

Folder A has 40Gb
Folder B has 2Gb
Folder C has 19Gb
Folder D has 125Gb
Folder E has 1Gb
Folder F has 28Gb
Folder G has 6Gb
...and the physical drive has 279Gb Free.

Roll forward 4 weeks....

Folder A has 100Gb
Folder B has 10Gb
Folder C has 90Gb
Folder D has 160Gb
Folder E has 2Gb
Folder F has 118Gb
Folder G has 10Gb
...and the physical drive has 10Gb Free.

At some point between weeks 3 and 4, the physical space hit 10Gb, so old "media" was overwritten. That's the theory of how this works.

Here's the reality:

* Make sure the PHYSICAL storage is MORE than the expected DATA need - that's full backups, storage for daily changes, incrementals, synthetics and so on, plus any other data you have on the same volume (perhaps Catalogs). It pays to have perhaps 50% more storage than you truely need, and more if you can.

* Make sure your storage calculations are enough for the retention. So if you've got a 4 week retention, do a weekly full, you need x 4 plus incremental storage. Realistically, maybe 6 times the storage.

* If your overall data storage decreases, don't expect more space to appear on the phyiscal drive. Because it treats "media" like a tape, it handles it the same way. If you had 30 tapes, and you only needed 20 of them now, the other 10 don't get erased from the earth, they just stay kicking about in your cupboard. Backup exec keeps your virtual "tapes" in it's cupboard. (we'll explain how to manage this better some other day).

* Regularly monitor your Backup to Disk Folders, particularly once you're getting to the space limits, because if your reserve is too big, and the retention period longer than you can cope with in physical space terms, you'll start getting failed jobs if there is simply no overwritable media left to be used.

* Having "overwriteable" media available is handy in some ways, as it means data beyond the retention period can still be available if the media hasn't yet been overwritten, almost building in a "last chance saloon" retention period, but it is also likely to consume all available space.

* If not all data is equally important, consider having different backup to disk folders, so that more critical data can be given longer actual retention times, but don't forget overwriteable media in one folder can't be claimed by another to add space, so good planning of physical space, folder space and necessary space is required.

If you follow this guide you can reduce the misery of the lacklustre Backup Volume Management. It won't allow you to have absolute limits for a media set (which you'd no doubt want), but it does stop you just running out of space.

Uptime - 1 day!

2007-11-14T21:15:00.000Z

A minor miracle has occured - the Backup Exec services have decided to play nicely together, and as a result the CASO server has now been running for 1 day. It sounds like it's nothing, the sort of uptime that would have perhaps made a Windows 98 use proud, but in fact, with Backup Exec, a whole day without something crashing is certainly a miracle.

Right now we have 3 servers, all up, all running jobs, all receiving jobs delegated from the CASO box, and barely any failures (a couple of those are genuinely not the fault of Backup Exec). It will never last, surely?

d means "dumb"

2007-11-14T13:56:00.000Z

I think that's what it means. Dumb as in "poorly thought-out", "stupidly implemented" and so on. At least that's what I believe. It certainly doesn't point to any amazing disk capabilities as they don't work.

What's the point of this 'd' nonsense?

2007-11-14T09:27:00.000Z

So the 'd' in 10d, is supposed to represent 'disk', as in you can now backup to disk, rather than tape, but hang on... we've had disk-based backups since v8.6, so what's the beef?

Well aparantly the emphasis is more on the 'staging' side of things, ie you can backup disk-to-disk-to-tape. Brilliant I thought, that's just what we need. We can do the first backup to disk, which is nice and fast, then we can put that media set onto tape and store it offsite, presumably using the new 'duplicate' feature. That way, we've got the data onsite in case we need a quick restore and a duplicate copy offsite for disaster recover - what more could we want? Even better, it'll all be really easy to do now, because we have 'Policies', 'Selection Lists' and 'Templates' to save time.

NO, NO, NO! What was I thinking?! Did I actually expect the new features to work, as PROMISED? Silly me. That was HOPING for too much. Skip forward to REALITY, which actually means 'disks' are treated exactly the same as tapes. Why? How much more effort would it have been to expect that a 'd' product would have at least some slight comprehension of disk-based media? I'm sorry, but I'd expect a little more intelligence, such as being able to quote the maximum disk size, so that I could actually allocate a quota for a particular device I was backing up. Too much to ask obviously.

And another thing... Am I the only person that thought that this new 'duplicate' media set feature would work between different Managed Media Servers? After all, they're connected via a LAN and they have the ability to share Files and Folders via normal, everyday UNC shares. You can even create a 'duplicate' job and set the source and target devices from different Media Servers - they're all there in the list, but when you try to submit it, it says no. Now that's something that would've been bloody useful. Imagine, you have a Managed Media Server on another site, via a VPN link and being able to automatically duplicate your Media Set to it, so you can relax in the comfort of knowing that you always have an offsite backup? How much more effort would that have been?

Obviously Symantec are very good at painting pictures, but not so good when it comes to implementation. Maybe they should get out of this software game and get into bull**** marketing instead.

A new morning, a new failure...

2007-11-14T08:49:00.000Z

It's just another average morning, a little nippy outside, terrible morning TV still exists (when will it die?) and I've got a raft of Backup Exec failures on my hands.

It seems that one of our sites has a condition I'll call "fussity", whereby the CASO server will submit it the jobs it's supposed to do, and, for about 90% of them, it'll just spit the dummy. Throwing "loading media" at you for a random period (could be 2 mins, could be 20 mins) eventually it gives up and fails the job. Only for the next one to work. And then next 9 to fail, and so on.... the last time we had a case of "fussity" the only way to stop it being so picky about jobs was to completely uninstall, rename the server, reinstall, reservice pack, and then re-create your devices, media, jobs and so on. I'm hoping to avoid it this time.

Meanwhile the main site figured it wasn't in the mood initially last night until around 8:30pm when I rebooted it. It just kept "recovering" it's own jobs. Good stuff.

All in a day's admin of Backup Exec.

Backup Exec Upgrade Cycle Explained:

2007-11-13T22:46:00.000Z

Here we attempt to explain the Symantec Backup Exec upgrade life cycle. Start at 'Promise' and work your way round clockwise.

A New Strategy

2007-11-13T17:09:00.001Z

I'm going to try a new strategy. We're going to create new jobs, one by one, for the servers. Each job will backup using old style Full/Incremental jobs, to the local server with basic 4 week rotation. Not even synthetics now, we can kiss goodbye to even more space. Thanks Backup Exec!

Let's see in a few days if that works either (see the lifecycle post below) - I'm at the "hope" stage again here...)

Alternatives

2007-11-13T16:07:00.000Z

Let's see now...

CommVault has good reports from a few people, but has a high price tag, and so far quite annoying sales processes (I hate any product you can't just try without asking nicely, agreeing to countless phone calls etc). Just give me the product, and I'll let you know if it's worth continuing discussions.

ArcServe - if hell is frozen over I guess, given it's supposedly "really good" at backups but just not good at bringing them back again when it matters (you know, the point of backing up...). Mind you that's not exactly Backup Exec's strength (mainly because you never get a backup to try restoring from anyhow)

Yosemite - never used it, but it seems to meet the "what it can do" on paper (like Backup Exec then).

Perhaps tin can, string and a 56k modem would help. We could TFTP the files around.

Legal Department at Symantec Please...

2007-11-13T11:34:00.000Z

Right now I'm calling Symantec "Customer Care" to find out where I send a legal notice. I'm inth e UK so the website handily giving us U.S. Addresses isn't ideal - but hell, if it has to go stateside, that's fine.

Why do I want it? Well the software doesn't work, it isn't fit for purpose. I spend more time per day trying to make this stupid system work than makes sense. I'd be better off hiring cheap labour and having them manually run backups.

Anyhow, the "agent" wasn't able to give me any useful help, so I ended up having to just hope that the UK Registered Address will do (it should do):

SYMANTEC (UK) LIMITED
350 BROOK DRIVE
GREEN PARK
READING
BERKSHIRE
RG2 6UH

Queued. You mean "lost the plot"

2007-11-13T10:55:00.000Z

Can we have a drum roll please....? The reason for our lovely "queued" status today is that at some point where the CASO box rolled onto it's belly and refused to play nicely with all the other devices, it managed to leave a "piece" of "media" in use in most of the Backup to Disk folders. End result - the "maximum concurrent jobs" limit has been reached.

The only resolve that seems to work is completely restarting the services (if you just restart Device and Media then the box tends to give up and never run a backup again). However, the trouble with this theory is that there are a couple of jobs running I want to finish first.

We could of course up the concurrent jobs on each device but it gets a bit tedious moving the limits up and down every day.

Really of course, it should JUST BLOODY WORK.

I'm queuing in the rain...

2007-11-13T09:11:00.000Z

I woke up this morning half expecting Backup Exec not to be working, and, as usual, it isn't. There are 2 possible causes for this:

a) My colleagues theory that Backup Exec works as long as you are physically staring at the Job Monitor, is in fact, completely true, and therefore because last night I figured I had other things to do, didn't watch it, and thus must now suffer it's failures.

or

b) (and this is my most likely theory) it's just a crock of craaaaaaap.

I'm now staring at the screen, and, in typical fashion, with no useful or meaningful reason it's just "Queued" for about 15 jobs, all of which have been sitting there for 9 hours. Just queued. No alerts, no reasons. But there is one job running, that's been running for 6 hours, and is slowly very slowly) notching up the byte count.

I am going to completely lose the will to live with this software soon.

Snow stopper (chuckle)

2007-11-12T20:14:00.002Z

It's just gone 8pm, and naturally the backup system has lost the plot. Not that this helps me. It means the last 24 hours of backups have gone south too, (or the 15 odd jobs running have).

Do we know what's wrong - no, of course not, but good old bengine.exe has started chewing 100% CPU now assigned to the pit of hell until further notice. No idea if it's working or not, we're not allowed to know because the CASO box isn't giving status anymore, although I'm sure when I glanced at it just now we had the famous "server paused" nonsense again.

Give it an hour or so and I'll be able to say what died this time...

It's Snowing...

2007-11-12T15:01:00.000Z

Actually it isn't. But it may as well be, as Backup Exec hasn't snuffed it yet. And we're past lunch...

More on "Loading Media"...

2007-11-12T11:58:00.000Z

The earlier problem with "Loading Media" appears to be a false alarm - while the CASO server (the one which is supposed to mean we can manage things centrally without needing to constantly login to all the other boxes) swears blind the job status is "Loading Media", logging onto the actual media server reveals it's working away backing up quite happily.

Yet again the CASO option proves it's worthless.

"Loading Media"...

2007-11-12T10:42:00.000Z

Fantastic news, it's not even 11am yet, but we've got more trusty errors to keep us busy.

Right now we've got the famous "Loading Media" nonsense. 3 jobs being backed up to 3 different "Backup Folder's" (you know, tapeless, medialess stuff) now sitting at Loading Media. No alerts naturally, no reason for them to do this, but they've all stopped and will undoubtedly now sit there forever not backing up.

Making it work: Tip #1

2007-11-12T09:20:00.001Z

As this Backup Exec system has given us so much hell, we've learned a few things you won't find in the manual, so as we think of them, we'll make a note of them for you in case you stumble across this page determined to find fixes to your Backup Exec issues.

My first top tip is this - if you find your server regularly ends up in a 100% CPU used status, not long after jobs should have been submitted to the queue via your policy, the chances are you've come across "incapability bug #1" - it seems Backup Exec can't handle having LOTS of jobs submitted at once - despite this being impossible to control if you create a policy for a group of servers which is recommended. So split your jobs into lots of smaller policies starting them a few minutes apart from each other (we did it 15 mins apart but you can go closer from experiements tried here) (no more than 20 jobs/selection lists per policy). Result, "bengine" stops sitting there at 100% CPU and causing the dreaded "server paused" fault.

A new day and new problems...

2007-11-12T09:12:00.000Z

Ah it's a good feeling today, why? Because only 6 jobs have failed (sort of) at the first site over the last 24 hours. Of course, it isn't quite that simple, as most of those were re-run 2-3 times to make them run, and we haven't got any real backups from the other 2 sites to talk about, but it's progress symantec-style.

Today's first problem (and it's only 10 past 9...) is that despite rebuilding Site 2, it's now developed a new problem - the first backup job, (to one distinct backup to disk volume with it's own media set) is running, but subequent jobs joining the queue cause the job to move to "loading media" and there it just sits... good stuff. So don't schedule jobs either maybe? It's practically a manual setup anyway I guess, so sitting there like a hawk looking for an opportunity to backup won't hurt.

It's just past 7...

2007-11-11T19:09:00.000Z

...and the first batch of Backups have appeared for the evening - since I made some changes last week to split the jobs into even more policies to reduce the chance that "Backup Exec Job Engine" or "Ben-Gin" as we call it will stall, it does seem to cope a little better, although so far this hasn't translated into the system running correctly, it has stopped it failing quite so often around job submission time.

Of course the whole point of policies is defeated by having to create lots of identical policies with slightly different start times (since now any change means changing them all...)... but we'll let that slide. Anything to get Backups to run even once a week would be nice!

A Little History

2007-11-11T18:33:00.000Z

Since this blog is new, I thought I'd share a bit of history about how we came to be in this position of undesireable misery.

Between the 3 of us, we've been using Backup Exec for many years, back when it was "Veritas Backup Exec". It was a reliable product with a few irritating quirks, but on the whole it got the job of Backups done, and we managed a few basic single server setups for various customers, some to single DAT drives, others to large LTO Libraries, but always with reliability of Backup Exec, if not the tape drives.

Move forward a few years...

We're faced with an ever increasing number of servers, multiple sites and the need to backup everything regularly, and in many cases the old "one drive per server" setup just wasn't working anymore. So we built some multi-terabyte Backup Exec Media Servers. Armed with plenty of storage, RAID Arrays and lots of shiny new Backup Exec 10d (d for disk don't you know!) licenses, we set out...

Having looked at all the various promised features, synthetic backups, great support for "Disk Based Backups" (something we really wanted), we figured it would be the wise choice, and, having had no issues with our old v8.6 and v9 installations didn't expect any problem. Yeah sure Symantec had bought Veritas but the veritas name was still there and it was the same product right?

Forward a few months...

Missed and Failed Backups are par for the course, random errors are the norm, and most of the promised features just don't work, or don't work as you'd expect, and some of the most useful features are hindered by completely stupid limitations that render the feature worthless. Oh, and just before you say "It's OK, we'll just do simple backups", don't expect it to be any easier - they don't work either.

Between us we'll post over the next few weeks about some of the biggest problems this product has, keep you up to date with the ongoing hell. If you're considering Backup Exec, don't. Try something else. Backup to 20,000 floppy disks manually copying files. Anything. It will work better.

And so it begins...

2007-11-11T18:16:00.000Z

So here we are with a completely failing Backup Exec setup. As usual, it's having random fits and failures. This weekend was going quite well, or rather, "well" in backup exec terms.

We had 1 out of 3 media servers running, but the one that was running was happily dealing with jobs it was assigned, until 16:30, and then, for no reason at all as far as we can see, it did the usual "Server Paused" thing. You know, the one where it dumps jobs you've had running for hours for no real reason, and just sits like a lame duck until you reboot it (because nothing else works).

Meanwhile box number 2 is still in "paused" status via the CASO tool, because hell, asking is to be enabled causes jobs to just be sent into an infinite loop of "Queued" and "Ready, On Hold" mixed with a load of failures. Good stuff Symantec.

Box number 3 however is really screwed. We've ended up having to reinstall and then when that failed, uninstall, and reinstall from scratch. Of course, that's really upset things, because despite following the instructions, the orphaned "managed media server" is now stuck on the CASO box, as is the new reinstalled one of the same, and all the non-existant "Backup to Disk" drives are too, you know, for good measure which it steadfastly refuses to remove.

No point calling Symantec for support, they never actually have an answer unless it's the most basic of problems.

So what shall we do?

I'm going to uninstall box 2 completely, rename the box on the network so it can't be confused anymore, reinstall, service pack 4 Backup Exec 10d and then configure it - once that's done I'll try a backup job, and if that runs (yeah, because it's so likely to work first time...) I'll try putting it onto managed media server mode so we can actually use it properly.

This is the first post here, but, inspired by a guy ( see http://y33dave.wordpress.com/%20) who eventually ditched Backup Exec for CommVault who maintained a blog of the miserable journey he went on we'll be keeping this Backup Exec Hell blog going...