Backups at Futurice
Greetings from the IT team! I thought I'd quickly describe how we handle backups at Futurice. In short: we try to backup everything and keep a decent amount of history.
For servers and virtual machines, we have two backup servers running BackupPC, located in Helsinki and Espoo. All virtual servers - even "I'll just test this, it's not important at all" testing servers - are automatically backed up daily without doing anything manually. BackupPC deduplicates backups, meaning that identical files take up space only once. BackupPC keeps backups almost indefinitely, but the interval between backups grows. In our configuration, daily backups are kept from the previous week, bi-daily ones after that for a few weeks, then every four days for a few months, once a week for year, and finally once a month forever. Of course, we have special configurations for some hosts that require more or less backup history.
In addition to the BackupPC hosts, we make USB disk offline backups four times a year of all irreplaceable data: version control, network disk, wiki, issue trackers, configuration files. Those disks are transported to a bank safe. In case of a huge catastrophe (for example, simultaneous fires in Espoo and Lauttasaari), we would lose at most 3 months worth of data. This is, of course, a very bad situation, but it won't bring the whole company down.
For laptop backups we have a separate server with a large disk rack. Every employee can create new virtual backup disk with a simple web interface. It supports AFP and Samba for Mac OS X Time Machine or Windows Backup. The hardware is made up of a cheap disk rack with 16x 750GB SATA disks in RAID5 with one hot spare drive. The disk controller automatically tests disks and the hot spare periodically, so we shouldn't have an unpleasant surprise with both a broken RAID system and a broken spare disk.
For the BackupPC servers, we use 15TB of RAID5 storage running on slow SATA disks. Currently, we have 214 hosts backed up with almost 1500 full backups and 1000 incremental backups, totaling almost 67 000 000 files and 21TB of non-deduplicated backups. After deduplication and compression, this is reduced to less than 2TB. The rest of 15TB disk space is not wasted, since our surveillance camera images and other miscellaneous files go to the same servers.
Ages ago, our backup server was an old desktop computer with a large disk running in the corner of our office. Back then, there were only a few hundred gigabytes of backups. When the time came to find a larger solution, we evaluated different backup solutions. We decided that, for our environment, tapes or commercial backup software didn't offer us enough benefits to outweigh the costs. Using tapes requires more manual work - changing tapes, keeping track of used tapes and ensuring that restores are working. With BackupPC, restoring data for testing purposes is a straightforward task, and does not require any interruptions to running backups. A large enough tape library with a changing robot was prohibitively expensive. As for commercial backup software solutions, having secure, incremental, deduplicated backups on a Linux server was simply not available for a reasonable price. Also, the possibility to backup new virtual hosts automatically was not possible with a reasonable amount of effort. Our solution of having a server with large disks was much more flexible, easier to setup and more cost effective.
