Disaster Recovery

While everyone else in Co-operation Ireland is getting ready to help with Maracycle, I've been stress-testing our disaster recovery mechanisms. The good news is no data was lost. The bad news is that recovery takes longer than expected, but I still recovered two computers in one working day.

Scenario One:
Laptop has close encounter with a beverage container. Beverage emerges as clear victor.

Experience: Got a quote for the repair from Apple since the computer (a white iBook from 2 years ago) is no-longer under warranty. Its cheaper to replace whole computer than to repair the damaged one. So, I purchased replacement iBook. I then put the hard drive from the damaged one into an external caddy and used the OS X "transfer settings" command, which worked like a charm. Kudos to Apple!

Scenario Two:
We use FreeBSD on lots of our servers, including our firewalls. The hard drive in our Belfast firewall failed catastrophically on Thursday morning. Luckily I was already on my way to Belfast (due to scenario one) and had a spare hard availabvle.

The thing is, installing the operating system (FreeBSD 7.3) and the few applications needed (sudo, ipsec-tools) took under an hour. Re-compiling the kernel to include the latest security patches and IP Sec support took ages on the old hardware.

In the end, after a false start or two, I got the firewall re-installed in time for everyone to check their email before they went home for the day. I then stayed quite late getting the VPN re-configured. I think the lesson learned from this is that I need to keep an up-to-date spare hard drive or two fit to be plugged into any of our four firewalls, with all four sets of configuration files backed up onto it. Having a hot-plug backup of the firewall configuration would have changed yesterday from being an 18 hour ordeal into a 30 minute "here's one I prepared earlier" magic trick.