But the problem is Backup Exec does appear to largely work reliably now - at the moment we're on 12 days 5 hours. It's hardly something you can call "reliable to 5 9's (e.g. 99.999% reliable), but it's certainly better than every few hours!
Just to recap the top tips...
1) Don't set more than a handful of jobs to run at once. They won't.
2) Although policies and templates are good ideas, expect to create lots of them because of tip 1.
3) Have plenty of Disk to Disk Folders, and set no more than 1 or 2 jobs per set as the concurrent limit.
4) Make good use of the Disk Reserve settings to ensure it has no excuse to run out of space and fail your jobs for weeks.
5) Set sane retention policies (e.g. don't just set it to weeks for the sake of it)
6) Split jobs by location, server, and drive.
7) Backup Exchange or SQL (and other "Agent" options) as distinct jobs
8) Keep your eye on Backup Exec at all times.
Showing posts with label disk based. Show all posts
Showing posts with label disk based. Show all posts
Saturday, 29 December 2007
Thursday, 15 November 2007
"d" is for "tape"
Something is wrong. Very wrong. Backup Exec is still working. That's over a day. It's not crashed or had a funny 10 minutes where it doesn't work. None of this "server paused" rubbish. Still meanwhile in the world of the admin's maintaining it, we've had a good old debate about how the whole thing is supposed to work, how symantec interpret "tape" backups when they're actually disk backups.
The only thing we did resolve is that the "d" in Backup Exec 10d means "tape". Yes. it means Tape. Which is probably how Symantec would have preferred we kept things.
The only thing we did resolve is that the "d" in Backup Exec 10d means "tape". Yes. it means Tape. Which is probably how Symantec would have preferred we kept things.
Wednesday, 14 November 2007
Making it work: Tip #2
One of my pet hates of Backup Exec is it's rather poor disk space management for Backup To Disk Folders. There are 2 approaches to Disk Space Management, the Sane Way, and the Stupid Way.
Guess which Backup Exec chooses...
As you can't set a limit on the amount of space a Backup To Disk volume can occupy (almost like a LTO-1 tape has a 100Gb storage capacity, and that's that, if you run out, you run out), you'd expect to be able to do similar on Backup Exec. But while you can tell it how big an individual Backup To Disk FILE is, you can't set a limit on the total space a "Disk" can occupy. The limit of course is the physical space on the drive itself.
The "workaround" is that if you had a drive, let's say it's 500Gb, and you have 6 Backup to Disk "Folders" on that drive. Set each one to have a Disk "Reserve" of something (the default is "nothing", naturally), let's say 10Gb. When you first use Backup Exec, it'll just keep making "B2Dnnnnn" files, but once you get to <10Gb left on the PHYSICAL drive, the Backup Exec Folders will then theoretically start to be overwritten, preventing the problem of running out of space completely.
Follow that? No. OK, try this:
PHYSICAL DRIVE (we'll call it Drive E:) has 500Gb Space, and it's only being used by Backup Exec.
You have 6 folders, each set with 10Gb "Disk Reserve":
Folder A, Folder B etc
Roll forward 2 weeks...
Folder A has 40Gb
Folder B has 2Gb
Folder C has 19Gb
Folder D has 125Gb
Folder E has 1Gb
Folder F has 28Gb
Folder G has 6Gb
...and the physical drive has 279Gb Free.
Roll forward 4 weeks....
Folder A has 100Gb
Folder B has 10Gb
Folder C has 90Gb
Folder D has 160Gb
Folder E has 2Gb
Folder F has 118Gb
Folder G has 10Gb
...and the physical drive has 10Gb Free.
At some point between weeks 3 and 4, the physical space hit 10Gb, so old "media" was overwritten. That's the theory of how this works.
Here's the reality:
* Make sure the PHYSICAL storage is MORE than the expected DATA need - that's full backups, storage for daily changes, incrementals, synthetics and so on, plus any other data you have on the same volume (perhaps Catalogs). It pays to have perhaps 50% more storage than you truely need, and more if you can.
* Make sure your storage calculations are enough for the retention. So if you've got a 4 week retention, do a weekly full, you need x 4 plus incremental storage. Realistically, maybe 6 times the storage.
* If your overall data storage decreases, don't expect more space to appear on the phyiscal drive. Because it treats "media" like a tape, it handles it the same way. If you had 30 tapes, and you only needed 20 of them now, the other 10 don't get erased from the earth, they just stay kicking about in your cupboard. Backup exec keeps your virtual "tapes" in it's cupboard. (we'll explain how to manage this better some other day).
* Regularly monitor your Backup to Disk Folders, particularly once you're getting to the space limits, because if your reserve is too big, and the retention period longer than you can cope with in physical space terms, you'll start getting failed jobs if there is simply no overwritable media left to be used.
* Having "overwriteable" media available is handy in some ways, as it means data beyond the retention period can still be available if the media hasn't yet been overwritten, almost building in a "last chance saloon" retention period, but it is also likely to consume all available space.
* If not all data is equally important, consider having different backup to disk folders, so that more critical data can be given longer actual retention times, but don't forget overwriteable media in one folder can't be claimed by another to add space, so good planning of physical space, folder space and necessary space is required.
If you follow this guide you can reduce the misery of the lacklustre Backup Volume Management. It won't allow you to have absolute limits for a media set (which you'd no doubt want), but it does stop you just running out of space.
Guess which Backup Exec chooses...
As you can't set a limit on the amount of space a Backup To Disk volume can occupy (almost like a LTO-1 tape has a 100Gb storage capacity, and that's that, if you run out, you run out), you'd expect to be able to do similar on Backup Exec. But while you can tell it how big an individual Backup To Disk FILE is, you can't set a limit on the total space a "Disk" can occupy. The limit of course is the physical space on the drive itself.
The "workaround" is that if you had a drive, let's say it's 500Gb, and you have 6 Backup to Disk "Folders" on that drive. Set each one to have a Disk "Reserve" of something (the default is "nothing", naturally), let's say 10Gb. When you first use Backup Exec, it'll just keep making "B2Dnnnnn" files, but once you get to <10Gb left on the PHYSICAL drive, the Backup Exec Folders will then theoretically start to be overwritten, preventing the problem of running out of space completely.
Follow that? No. OK, try this:
PHYSICAL DRIVE (we'll call it Drive E:) has 500Gb Space, and it's only being used by Backup Exec.
You have 6 folders, each set with 10Gb "Disk Reserve":
Folder A, Folder B etc
Roll forward 2 weeks...
Folder A has 40Gb
Folder B has 2Gb
Folder C has 19Gb
Folder D has 125Gb
Folder E has 1Gb
Folder F has 28Gb
Folder G has 6Gb
...and the physical drive has 279Gb Free.
Roll forward 4 weeks....
Folder A has 100Gb
Folder B has 10Gb
Folder C has 90Gb
Folder D has 160Gb
Folder E has 2Gb
Folder F has 118Gb
Folder G has 10Gb
...and the physical drive has 10Gb Free.
At some point between weeks 3 and 4, the physical space hit 10Gb, so old "media" was overwritten. That's the theory of how this works.
Here's the reality:
* Make sure the PHYSICAL storage is MORE than the expected DATA need - that's full backups, storage for daily changes, incrementals, synthetics and so on, plus any other data you have on the same volume (perhaps Catalogs). It pays to have perhaps 50% more storage than you truely need, and more if you can.
* Make sure your storage calculations are enough for the retention. So if you've got a 4 week retention, do a weekly full, you need
* If your overall data storage decreases, don't expect more space to appear on the phyiscal drive. Because it treats "media" like a tape, it handles it the same way. If you had 30 tapes, and you only needed 20 of them now, the other 10 don't get erased from the earth, they just stay kicking about in your cupboard. Backup exec keeps your virtual "tapes" in it's cupboard. (we'll explain how to manage this better some other day).
* Regularly monitor your Backup to Disk Folders, particularly once you're getting to the space limits, because if your reserve is too big, and the retention period longer than you can cope with in physical space terms, you'll start getting failed jobs if there is simply no overwritable media left to be used.
* Having "overwriteable" media available is handy in some ways, as it means data beyond the retention period can still be available if the media hasn't yet been overwritten, almost building in a "last chance saloon" retention period, but it is also likely to consume all available space.
* If not all data is equally important, consider having different backup to disk folders, so that more critical data can be given longer actual retention times, but don't forget overwriteable media in one folder can't be claimed by another to add space, so good planning of physical space, folder space and necessary space is required.
If you follow this guide you can reduce the misery of the lacklustre Backup Volume Management. It won't allow you to have absolute limits for a media set (which you'd no doubt want), but it does stop you just running out of space.
Tuesday, 13 November 2007
Queued. You mean "lost the plot"
Can we have a drum roll please....? The reason for our lovely "queued" status today is that at some point where the CASO box rolled onto it's belly and refused to play nicely with all the other devices, it managed to leave a "piece" of "media" in use in most of the Backup to Disk folders. End result - the "maximum concurrent jobs" limit has been reached.
The only resolve that seems to work is completely restarting the services (if you just restart Device and Media then the box tends to give up and never run a backup again). However, the trouble with this theory is that there are a couple of jobs running I want to finish first.
We could of course up the concurrent jobs on each device but it gets a bit tedious moving the limits up and down every day.
Really of course, it should JUST BLOODY WORK.
The only resolve that seems to work is completely restarting the services (if you just restart Device and Media then the box tends to give up and never run a backup again). However, the trouble with this theory is that there are a couple of jobs running I want to finish first.
We could of course up the concurrent jobs on each device but it gets a bit tedious moving the limits up and down every day.
Really of course, it should JUST BLOODY WORK.
Monday, 12 November 2007
"Loading Media"...
Fantastic news, it's not even 11am yet, but we've got more trusty errors to keep us busy.
Right now we've got the famous "Loading Media" nonsense. 3 jobs being backed up to 3 different "Backup Folder's" (you know, tapeless, medialess stuff) now sitting at Loading Media. No alerts naturally, no reason for them to do this, but they've all stopped and will undoubtedly now sit there forever not backing up.
Right now we've got the famous "Loading Media" nonsense. 3 jobs being backed up to 3 different "Backup Folder's" (you know, tapeless, medialess stuff) now sitting at Loading Media. No alerts naturally, no reason for them to do this, but they've all stopped and will undoubtedly now sit there forever not backing up.
Sunday, 11 November 2007
A Little History
Since this blog is new, I thought I'd share a bit of history about how we came to be in this position of undesireable misery.
Between the 3 of us, we've been using Backup Exec for many years, back when it was "Veritas Backup Exec". It was a reliable product with a few irritating quirks, but on the whole it got the job of Backups done, and we managed a few basic single server setups for various customers, some to single DAT drives, others to large LTO Libraries, but always with reliability of Backup Exec, if not the tape drives.
Move forward a few years...
We're faced with an ever increasing number of servers, multiple sites and the need to backup everything regularly, and in many cases the old "one drive per server" setup just wasn't working anymore. So we built some multi-terabyte Backup Exec Media Servers. Armed with plenty of storage, RAID Arrays and lots of shiny new Backup Exec 10d (d for disk don't you know!) licenses, we set out...
Having looked at all the various promised features, synthetic backups, great support for "Disk Based Backups" (something we really wanted), we figured it would be the wise choice, and, having had no issues with our old v8.6 and v9 installations didn't expect any problem. Yeah sure Symantec had bought Veritas but the veritas name was still there and it was the same product right?
Forward a few months...
Missed and Failed Backups are par for the course, random errors are the norm, and most of the promised features just don't work, or don't work as you'd expect, and some of the most useful features are hindered by completely stupid limitations that render the feature worthless. Oh, and just before you say "It's OK, we'll just do simple backups", don't expect it to be any easier - they don't work either.
Between us we'll post over the next few weeks about some of the biggest problems this product has, keep you up to date with the ongoing hell. If you're considering Backup Exec, don't. Try something else. Backup to 20,000 floppy disks manually copying files. Anything. It will work better.
Between the 3 of us, we've been using Backup Exec for many years, back when it was "Veritas Backup Exec". It was a reliable product with a few irritating quirks, but on the whole it got the job of Backups done, and we managed a few basic single server setups for various customers, some to single DAT drives, others to large LTO Libraries, but always with reliability of Backup Exec, if not the tape drives.
Move forward a few years...
We're faced with an ever increasing number of servers, multiple sites and the need to backup everything regularly, and in many cases the old "one drive per server" setup just wasn't working anymore. So we built some multi-terabyte Backup Exec Media Servers. Armed with plenty of storage, RAID Arrays and lots of shiny new Backup Exec 10d (d for disk don't you know!) licenses, we set out...
Having looked at all the various promised features, synthetic backups, great support for "Disk Based Backups" (something we really wanted), we figured it would be the wise choice, and, having had no issues with our old v8.6 and v9 installations didn't expect any problem. Yeah sure Symantec had bought Veritas but the veritas name was still there and it was the same product right?
Forward a few months...
Missed and Failed Backups are par for the course, random errors are the norm, and most of the promised features just don't work, or don't work as you'd expect, and some of the most useful features are hindered by completely stupid limitations that render the feature worthless. Oh, and just before you say "It's OK, we'll just do simple backups", don't expect it to be any easier - they don't work either.
Between us we'll post over the next few weeks about some of the biggest problems this product has, keep you up to date with the ongoing hell. If you're considering Backup Exec, don't. Try something else. Backup to 20,000 floppy disks manually copying files. Anything. It will work better.
Subscribe to:
Posts (Atom)