It's been a while, but we've recently come across a ridiculous error situation with DPM that caused a random subset of Protected Servers to just stop backing up.
Each Volume or resource would be in "Replica is inconsistent" state. You'd play the usual game of running consistency checks, or consistency & synchronisation but the job goes to "OK" for a while - but the ability to make a new recovery point is missing (eg the "Create a recovery point after synchronizing") option is unavailable, and a short while later a protected volume would return to "replica is inconsistent" state.
After much head scratching and monitoring, we realised it was very simple... the installation path for DPM includes a temp folder, for example:
"C:\Program Files\Microsoft System Center 2012\DPM\DPM\Temp"
The folder "MTA" had been removed as part of a clear up of old temporary resources, and despite this folder not being actively used during a backup it seems not having it breaks DPM.
Simply recreate a folder called "MTA" and you'll then find everything is working just fine again - re-run those consistency checks and then make a recovery point with synchronisation and all will be well.
Hopefully this will help someone else with a similar issue!
Showing posts with label tips. Show all posts
Showing posts with label tips. Show all posts
Saturday, 24 December 2016
Thursday, 12 May 2011
Another Top Tip for DPM - "replica is inconsistent" on System State and Bare Metal Recovery...
I thought I'd share this Top Tip. It is reasonably well documented, but a classic thing you'd often miss.
If you're finding that "System State" and "Bare Metal Recovery" items are frequently sitting in a "replica is inconsistent" state ...which happens a lot on Windows 2008 system... then the chances are you've not got "Windows Server Backup" as an installed feature on the server you're backing up (the protected server).
It's dead simple to sort, run Server Manager, click Add Feature, check "Windows Server Backup" and wait for it to install - job done - run a consistency check and they'll be sorted and work thereafter.
Of course, why the DPM installer doesn't just install this (or at least prompt) as part of the roll out since it is basically a dependency is anyones guess...
If you're finding that "System State" and "Bare Metal Recovery" items are frequently sitting in a "replica is inconsistent" state ...which happens a lot on Windows 2008 system... then the chances are you've not got "Windows Server Backup" as an installed feature on the server you're backing up (the protected server).
It's dead simple to sort, run Server Manager, click Add Feature, check "Windows Server Backup" and wait for it to install - job done - run a consistency check and they'll be sorted and work thereafter.
Of course, why the DPM installer doesn't just install this (or at least prompt) as part of the roll out since it is basically a dependency is anyones guess...
Friday, 6 May 2011
Common DPM Errors...
Since we've now got most of the Data Protection Manager 2010 installations done, I thought I'd share a few common issues we've come across, and the fixes. Maybe this'll save you a LOT of hassle...
"Access Denied (0x80070005)"
Common causes are listed all over the place, suggesting Firewalls as the issues and DCOM Permissions. All entirely possible. One other thing to consider, especially if you've setup Forest Trusts etc, just make sure you've made sure the AD Network holding your DPM Server(s) is fully accessible - and that this traffic isn't restricted either! In our case, we had a Cluster with 2 servers, one in a Subnet (we'll call this Subnet A), another in a different subnet (Subnet B) and our DPM Servers (and the DPM AD Network) in another (Subnet C).
While Subnet A and B could talk without restriction, and A could quite happily talk to C, for historical reasons, B and C weren't completely open for communication. So my tip - make sure you've considered Active Directory Authentication and not just "DPM to Protected Server" issues!
Agents are "unavailable" and "VssError: Invalid value for registry"
This ia bit of an odd one and just "happened" on a previously perfectly happy server. We resolved this by simply removing the account used to push out the agents in the DCOM Config (run "dcomcnfg.exe"), find the "DPM RA" in the list and remove/readd the user. No idea what caused that mind!
Replica is inconsistent with System State and repeatedly so...
Especially if you're on a Windows 2003 SP-2 32-bit system? Yep, thought so. You've probably just not got enough space on the system drive (normally C:\). You should move the normally hidden "DPM_SYSTEM_STATE" folder to another drive, ideally with +10GB free, and then update the data source...
\Microsoft Data Protection Manager\DPM\datasources\PSdataSourceConfig.xml
change:
%SystemDrive%\DPM_SYSTEM_STATE\*
so it points to wherever you put it... easily sorted.
Hopefully they'll help you for now, more tips later!
"Access Denied (0x80070005)"
Common causes are listed all over the place, suggesting Firewalls as the issues and DCOM Permissions. All entirely possible. One other thing to consider, especially if you've setup Forest Trusts etc, just make sure you've made sure the AD Network holding your DPM Server(s) is fully accessible - and that this traffic isn't restricted either! In our case, we had a Cluster with 2 servers, one in a Subnet (we'll call this Subnet A), another in a different subnet (Subnet B) and our DPM Servers (and the DPM AD Network) in another (Subnet C).
While Subnet A and B could talk without restriction, and A could quite happily talk to C, for historical reasons, B and C weren't completely open for communication. So my tip - make sure you've considered Active Directory Authentication and not just "DPM to Protected Server" issues!
Agents are "unavailable" and "VssError: Invalid value for registry"
This ia bit of an odd one and just "happened" on a previously perfectly happy server. We resolved this by simply removing the account used to push out the agents in the DCOM Config (run "dcomcnfg.exe"), find the "DPM RA" in the list and remove/readd the user. No idea what caused that mind!
Replica is inconsistent with System State and repeatedly so...
Especially if you're on a Windows 2003 SP-2 32-bit system? Yep, thought so. You've probably just not got enough space on the system drive (normally C:\). You should move the normally hidden "DPM_SYSTEM_STATE" folder to another drive, ideally with +10GB free, and then update the data source...
\Microsoft Data Protection Manager\DPM\datasources\PSdataSourceConfig.xml
change:
so it points to wherever you put it... easily sorted.
Hopefully they'll help you for now, more tips later!
Wednesday, 14 November 2007
Making it work: Tip #2
One of my pet hates of Backup Exec is it's rather poor disk space management for Backup To Disk Folders. There are 2 approaches to Disk Space Management, the Sane Way, and the Stupid Way.
Guess which Backup Exec chooses...
As you can't set a limit on the amount of space a Backup To Disk volume can occupy (almost like a LTO-1 tape has a 100Gb storage capacity, and that's that, if you run out, you run out), you'd expect to be able to do similar on Backup Exec. But while you can tell it how big an individual Backup To Disk FILE is, you can't set a limit on the total space a "Disk" can occupy. The limit of course is the physical space on the drive itself.
The "workaround" is that if you had a drive, let's say it's 500Gb, and you have 6 Backup to Disk "Folders" on that drive. Set each one to have a Disk "Reserve" of something (the default is "nothing", naturally), let's say 10Gb. When you first use Backup Exec, it'll just keep making "B2Dnnnnn" files, but once you get to <10Gb left on the PHYSICAL drive, the Backup Exec Folders will then theoretically start to be overwritten, preventing the problem of running out of space completely.
Follow that? No. OK, try this:
PHYSICAL DRIVE (we'll call it Drive E:) has 500Gb Space, and it's only being used by Backup Exec.
You have 6 folders, each set with 10Gb "Disk Reserve":
Folder A, Folder B etc
Roll forward 2 weeks...
Folder A has 40Gb
Folder B has 2Gb
Folder C has 19Gb
Folder D has 125Gb
Folder E has 1Gb
Folder F has 28Gb
Folder G has 6Gb
...and the physical drive has 279Gb Free.
Roll forward 4 weeks....
Folder A has 100Gb
Folder B has 10Gb
Folder C has 90Gb
Folder D has 160Gb
Folder E has 2Gb
Folder F has 118Gb
Folder G has 10Gb
...and the physical drive has 10Gb Free.
At some point between weeks 3 and 4, the physical space hit 10Gb, so old "media" was overwritten. That's the theory of how this works.
Here's the reality:
* Make sure the PHYSICAL storage is MORE than the expected DATA need - that's full backups, storage for daily changes, incrementals, synthetics and so on, plus any other data you have on the same volume (perhaps Catalogs). It pays to have perhaps 50% more storage than you truely need, and more if you can.
* Make sure your storage calculations are enough for the retention. So if you've got a 4 week retention, do a weekly full, you need x 4 plus incremental storage. Realistically, maybe 6 times the storage.
* If your overall data storage decreases, don't expect more space to appear on the phyiscal drive. Because it treats "media" like a tape, it handles it the same way. If you had 30 tapes, and you only needed 20 of them now, the other 10 don't get erased from the earth, they just stay kicking about in your cupboard. Backup exec keeps your virtual "tapes" in it's cupboard. (we'll explain how to manage this better some other day).
* Regularly monitor your Backup to Disk Folders, particularly once you're getting to the space limits, because if your reserve is too big, and the retention period longer than you can cope with in physical space terms, you'll start getting failed jobs if there is simply no overwritable media left to be used.
* Having "overwriteable" media available is handy in some ways, as it means data beyond the retention period can still be available if the media hasn't yet been overwritten, almost building in a "last chance saloon" retention period, but it is also likely to consume all available space.
* If not all data is equally important, consider having different backup to disk folders, so that more critical data can be given longer actual retention times, but don't forget overwriteable media in one folder can't be claimed by another to add space, so good planning of physical space, folder space and necessary space is required.
If you follow this guide you can reduce the misery of the lacklustre Backup Volume Management. It won't allow you to have absolute limits for a media set (which you'd no doubt want), but it does stop you just running out of space.
Guess which Backup Exec chooses...
As you can't set a limit on the amount of space a Backup To Disk volume can occupy (almost like a LTO-1 tape has a 100Gb storage capacity, and that's that, if you run out, you run out), you'd expect to be able to do similar on Backup Exec. But while you can tell it how big an individual Backup To Disk FILE is, you can't set a limit on the total space a "Disk" can occupy. The limit of course is the physical space on the drive itself.
The "workaround" is that if you had a drive, let's say it's 500Gb, and you have 6 Backup to Disk "Folders" on that drive. Set each one to have a Disk "Reserve" of something (the default is "nothing", naturally), let's say 10Gb. When you first use Backup Exec, it'll just keep making "B2Dnnnnn" files, but once you get to <10Gb left on the PHYSICAL drive, the Backup Exec Folders will then theoretically start to be overwritten, preventing the problem of running out of space completely.
Follow that? No. OK, try this:
PHYSICAL DRIVE (we'll call it Drive E:) has 500Gb Space, and it's only being used by Backup Exec.
You have 6 folders, each set with 10Gb "Disk Reserve":
Folder A, Folder B etc
Roll forward 2 weeks...
Folder A has 40Gb
Folder B has 2Gb
Folder C has 19Gb
Folder D has 125Gb
Folder E has 1Gb
Folder F has 28Gb
Folder G has 6Gb
...and the physical drive has 279Gb Free.
Roll forward 4 weeks....
Folder A has 100Gb
Folder B has 10Gb
Folder C has 90Gb
Folder D has 160Gb
Folder E has 2Gb
Folder F has 118Gb
Folder G has 10Gb
...and the physical drive has 10Gb Free.
At some point between weeks 3 and 4, the physical space hit 10Gb, so old "media" was overwritten. That's the theory of how this works.
Here's the reality:
* Make sure the PHYSICAL storage is MORE than the expected DATA need - that's full backups, storage for daily changes, incrementals, synthetics and so on, plus any other data you have on the same volume (perhaps Catalogs). It pays to have perhaps 50% more storage than you truely need, and more if you can.
* Make sure your storage calculations are enough for the retention. So if you've got a 4 week retention, do a weekly full, you need
* If your overall data storage decreases, don't expect more space to appear on the phyiscal drive. Because it treats "media" like a tape, it handles it the same way. If you had 30 tapes, and you only needed 20 of them now, the other 10 don't get erased from the earth, they just stay kicking about in your cupboard. Backup exec keeps your virtual "tapes" in it's cupboard. (we'll explain how to manage this better some other day).
* Regularly monitor your Backup to Disk Folders, particularly once you're getting to the space limits, because if your reserve is too big, and the retention period longer than you can cope with in physical space terms, you'll start getting failed jobs if there is simply no overwritable media left to be used.
* Having "overwriteable" media available is handy in some ways, as it means data beyond the retention period can still be available if the media hasn't yet been overwritten, almost building in a "last chance saloon" retention period, but it is also likely to consume all available space.
* If not all data is equally important, consider having different backup to disk folders, so that more critical data can be given longer actual retention times, but don't forget overwriteable media in one folder can't be claimed by another to add space, so good planning of physical space, folder space and necessary space is required.
If you follow this guide you can reduce the misery of the lacklustre Backup Volume Management. It won't allow you to have absolute limits for a media set (which you'd no doubt want), but it does stop you just running out of space.
Monday, 12 November 2007
Making it work: Tip #1
As this Backup Exec system has given us so much hell, we've learned a few things you won't find in the manual, so as we think of them, we'll make a note of them for you in case you stumble across this page determined to find fixes to your Backup Exec issues.
My first top tip is this - if you find your server regularly ends up in a 100% CPU used status, not long after jobs should have been submitted to the queue via your policy, the chances are you've come across "incapability bug #1" - it seems Backup Exec can't handle having LOTS of jobs submitted at once - despite this being impossible to control if you create a policy for a group of servers which is recommended. So split your jobs into lots of smaller policies starting them a few minutes apart from each other (we did it 15 mins apart but you can go closer from experiements tried here) (no more than 20 jobs/selection lists per policy). Result, "bengine" stops sitting there at 100% CPU and causing the dreaded "server paused" fault.
My first top tip is this - if you find your server regularly ends up in a 100% CPU used status, not long after jobs should have been submitted to the queue via your policy, the chances are you've come across "incapability bug #1" - it seems Backup Exec can't handle having LOTS of jobs submitted at once - despite this being impossible to control if you create a policy for a group of servers which is recommended. So split your jobs into lots of smaller policies starting them a few minutes apart from each other (we did it 15 mins apart but you can go closer from experiements tried here) (no more than 20 jobs/selection lists per policy). Result, "bengine" stops sitting there at 100% CPU and causing the dreaded "server paused" fault.
Subscribe to:
Posts (Atom)