I’ve always thought it strange that no one likes to run backups. It just surprises me. Only today I had to deal with an e-mail outage that I was unsure I would be able to recover from, and what kept everything in perspective was my knowledge that we had good backups. I was free to attempt any recovery path I chose, secure in the knowledge that I had a failsafe, and that the worst case was the loss of 12 hours of mail – as opposed to all of it.
I’ve worked in quite a few IT departments, and with the exception of Wright Medical (who quite happily sent 240 LTO4 tapes to permanent off site storage every week), none of them had a good backup solution in place when I arrived. Some of these were organizations that depended entirely on their data to survive as a company, but backups were something that somehow no one had ever had time to get around to.
It isn’t too difficult to find out why this is true. Backups are a chore. They’re expensive, they’re complex, they constantly need to be watched. There is always something that comes ahead of them on the priority list, and until you need them, you don’t get any operational benefit from the resources invested in your backup strategy.
That being said, I’ve put together a backup process where I work now that has the advantages of being relatively inexpensive and relatively functional.
Hardware first. Our organization has about 30 servers in 3 locations. I needed a hardware solution that would be large enough to support my backup need, but also be portable enough that I could store each site’s media at another site. I had no need for permanent archiving of data, this was purely for recovery and DR. I was looking for speed, so as not to have a huge backup window every night, so I wanted a disk based solution. I’m also just not a tape fan. Not only because tape has a high failure rate, but because my options would either to be constantly changing tapes, or needing to buy a hardware / software carousel solution that would have cost more than I wanted to spend. One disadvantage is that most disk based solutions didn’t have the portability I was looking for, and it is inconvenient to carry a SAN around from location to location. The good news was that I was able to find a disk based solution that fit my needs.
I ended up using the RAIDFrame system by Highly Reliable. (Their web site is http://www.tapesucks.com, which I think is a nice touch). The media is self-contained RAIDPac cartridges (containing 3 drives in a RAID 0 or RAID 5 array) with each cartridge supporting up to 9 TB of usable storage. Not bad for removable media, if I do say so myself. Not too many tapes out there that support 9 TB per tape. With either a 5 bay RAIDFrame or 1 bay RAIDFrame at each location I could put a week’s worth of backup into one device, and then only have to rotate media once per week
Once I had hardware, I needed to decide on software. The servers I manage are mostly running Windows Server 2008, with a few 2003 servers thrown in just to keep the environment from being too homogenous. Over the years I’ve used Backup Exec, Networker, and ARCserve on the proprietary side, and NTbackup on for free side to backup Windows servers, so those were my starting points. I also looked at “cloud” backup options.
My first decision was to throw out cloud. First, the initial backup takes too long, including the shipping of media. Second, in the event of a disaster, you’re guaranteed at least 2 days of downtime. Third, I would be unable to easily perform regular recovery testing. Finally, I just don’t have faith in “cloud” backups as a mature technology at this level. It works great for me for personal backups, but I’m not ready to commit my employer’s data to an environment outside my control. I need to be able to decide which risks to mitigate and which to accept, and not have that done fore me based on someone else’s bottom line.
That being said, I really wanted to see if I could do this with NTbackup. No licensing, no clients, and it’s my oft-mentioned favorite price point – free.
The first thing I did was to break my servers up my priority and size to keep my backup window uniform and under 8 hours. Production servers are backed up 3 nights a week – either MWF or TThS. Dev and support systems get backed up once a week, during the day on Saturdays and Sundays. The second thing was to attempt to overcome a couple of the limits of NTbackup. (It’s free, so it isn’t exactly feature rich.) NTbackup cannot back up to removable media – my servers see the RAIDPacs as network shares, so we overcome that limitation. NTbackup has no scheduling functionality – overcome by using scripts and scheduling those scripts with task manager. NTbackup only lets you define one backup in the console – again overcome by scheduled scripts.
On the 2008 servers, everything was very straightforward. I started by creating a backup user that was a member of the backup operators group. Using that account (which also had log on as s batch job right), I schedule something like this:
cd \ cd windows cd system32 wbadmin start backup -backupTarget:\\STORAGEDEVICE\RAIDPac\SERVERNAME\Monday -include:c: -allCritical -vssFull > \\STORAGEDEVICE\RAIDPac\SERVERNAME\Monday_SERVERNAME_BackupLog.txt
STORAGEDEVICE is the server I am backing up to
STORAGEDEVICE is sharing the RAIDPac as \RAIDPac
SERVERNAME is the server I am backing up. It only has one drive – “C:”
I create the folders “servername” and “Monday” in advance. I also create appropriate scripts and folders for Wednesday and Friday.
Finally, I have the scheduled task send an e-mail to me with the logfile attached for my review. This is done with the built in functionality of the 2008 task scheduler, and I already defined where the logfile will live.
For 2003 servers, it is a little more difficult. Not for the backup so much, as for getting that e-mail with the logfile. I really like getting those e-mails every morning to compare against my daily checklist, but the 2003 task scheduler does not include e-mail capability.
For this I use Andrew Taylor’s “Email NT Backup log file” script. Then my script has the additional line of:
That way the logfile will again be e-mailed to me after the backup is complete. His file is very easy to customize, and he gives explicit instructions on his website which I won’t duplicate here.
We backup our SQL databases and logfiles to the RAIDPacs as well – but only keep the last few when the media is swapped out. Since the databases back up a few times per day and the logs every 15 minutes, the media would fill up quickly without being purged regularly, which I do with this:
Forfiles /p D:\SQL1Backups /s /m *.* -d -2 -c "cmd /c del /q @path" Forfiles /p D:\SQL1Logs /s /m *.* -d -1 -c "cmd /c del /q @path" Forfiles /p D:\SQL2Backups /s /m *.* -d -2 -c "cmd /c del /q @path" Forfiles /p D:\SQL2Logs /s /m *.* -d -1 -c "cmd /c del /q @path" Forfiles /p D:\SQL3Backups /s /m *.* -d -2 -c "cmd /c del /q @path" Forfiles /p D:\SQL3Logs /s /m *.* -d -1 -c "cmd /c del /q @path"
I also backup our MySQL databases with a nice little script by Matthew Moeller at Red Olive. This one beats others I’d written myself because it automatically backs up all databases on the server without having to define them specifically. His site also does an excellent job at explaining how it works, which I also won’t repeat.
And finally my last free backup trick is backing up Hyper-V VMs in Powershell.
Between all of these tools either homegrown, or following the experience of others, I have been able to put together an inexpensive backup solution that meets all of our current needs, that doesn’t require proprietary software and licensing, nor does it leave us unable to recover from a hardware failure in a reasonable amount of time.
See? Aren’t backups fun?