Well, today was “fun”.
So, first, some backstory.
Last month, my boss worked out a deal where we would take over management of a server for another group on campus, which hosts 2 websites, and a file share (shared over AFP). We took possession of that machine, a Power Mac G5, on the 30th. I hooked it into the KVM, network, and secured the firewall and services on it. And just yesterday, got backups working fully.
Okay, so, if you don’t see where this is going….
First thing in my inbox that isn’t junk is a message about one of the websites on this box being down. I take a look, and indeed, the system is hung. Reboot results in the system being responsive for about 30 seconds before hanging again. Booting to Single User Mode (SUM) and fdisk’ing results in errors along the lines of:
AppleATADiskQueueManager: setPowerState (blah blah) timed out after 10000 ms.
Great. So, I fiddle some more, and then pull the drive. Lo and Behold, it’s a Seagate ST31000340AS, with Firmware SD15. If you don’t know about this problem, see the interwebs. Basically, Seagate shipped a pretty large batch of drives with bad firmwares that resulted in drive failure. In some cases, the failure did not impact the data, and simply updating to the new firmware restored the drive. I was not so lucky. After updating the firmware on a PC I keep handy, the drive did the same thing.
So, I finally got the machine up and running somewhat without a network connection. Just long enough to do a mysql dump on the system. Once that was done, I pulled the drive, hooked it to an external firewire -> sata controller (WiebeTech UltraDock, and tried to grab data off the drive. This mainly required powering on the drive, copying some files until things started failing, then cycling off, back on, and doing some more. Yes, the backup had not happened yet, since there’s a weekly full, then nightly incrementals. *sigh* But, luckily the former owners had a backup (clone) from the machine on the 29th of last month. Right before we took possession.
So, between those two sources, and the mysql dump that did work, I managed to move the websites over to our main web server, which is much more capable, has a RAID1 drive (duh?), and the websites mainly exist on the SAN. Oh, and it’s backed up quite regularly.
So, about 5 hours of work and the sites were back up. And I moved their file share (which thankfully had not had any changes happen since the 29th) to another server (I don’t like running file services on the webserver, for obvious reasons).
That’s really about it. I have some basic (as in, same functionality as the badge had) for the Gas mileage. I’m hoping to add a lot more functionality in the next week or so.
And as a parting statement… I hate the new title of the SciFi channel (SyFy). It’s ridiculous. I guess they spell things “Syence Fyction?”
