You may have noticed that my site has been offline for a few days. Or you may have noticed and thought I was simply being quiet. I’m known to some to be that way now and again. Not the case this time, though. I was offline good and proper. Now, after lots and lots of LogoRhythms, I’m back on line…
What happened was this: the utter corruption of my RAID1 setup, resulting in the loss of nearly all the data in the mirrored HD partition in which my web server data was stored. My /home directory was mounted on the same drive, but it wasn’t affected for some reason. My /www directory, home to my web server and all the data it served, was nearly completely lost, with plenty of data unrecoverable. At least I have my initial configurations, and somehow, the daily MySQL backups of all my databases.
No, I didn’t have the data backed up elsewhere. I relied on RAID, figuring that HD failure was my biggest concern, not driver/controller/software failure and the loss of superblocks and disk labels.
Yes, I’m an idiot. I should have known. Lesson learned.
So, over the weekend, with the blessings of my lovely and incredibly understanding wife, I exercised my hardware research muscles, and put together the specs for two identical systems to serve as production (Britannia) and development (Sosaria). Then I went out and bought it all (see lovely and incredibly understanding statement above). Here’s the thing… I spent about a third on both of those systems as I spent on my gaming rig 6 years ago, and they’re both arguably nearly three times as fast (by certain measures and points of view… they screamed through the initial SSH host key generation routines so fast I didn’t even realize it was done until I was sitting at a logon prompt). The hardware is anything but crazy fancy, but it is working.
As an aside, it was interesting to watch the two machines run package installs and updates in tandem through two simultaneous PuTTY instances. Nanoscopic differences in the hardware always had one machine or the other running ahead.
It’s nice to know I still got what it takes to piece together a working computer from separate components, and have it boot the first time I give it the juice. Granted, they were all new parts, so there wasn’t any real hacking involved, but it still feels nice.
I kept costs down by sticking with on-board graphics (who needs a distinct graphics card to push console text?), and already having monitors sitting around. I’m saving space by using a KVM switch, instead of multiple monitors & keyboards. It’s a mechanical switch, so until I get a battery backup in place (not cheap), I may run into issues if a brownout causes unplanned reboots. I’m keeping it set to my prod box in that case, to at least simulate some consistent uptime. I haven’t had a brownout in a few years, either, so I’m due…
The two servers are identical in every way, from the hardware up through the OS and (at least at this time) the software packages installed. That way, I can be sure what I develop on dev will work without issue on prod. In fact, once it’s all set up the way I want, I won’t even need to log in to prod to publish new work. I’ll just check in the changes via Subversion and sip my beverage of choice as it all “just works.”
To help avoid future data loss issues (rare is the perfect solution), my dev box will serve as backup to my prod box by default, and I’ll have some large external drives hanging off my dev slow box for yet more backups (via relatively slow USB 2.0 connections – but backups just have to work, not be fast – I’m still looking into a NAS solution, b/c that would just be cool). I’m thinking about looking into off-site solutions too (cloud?), but haven’t started that yet. I’ve no doubt some of you will have opinions on this set up, which I’m more than willing to entertain, assuming they don’t involve too much more money. :)
I’m excited to finally have a semi-proper development environment again. And develop I will… not only to recreate what I lost over the last 6 months (which wasn’t that much to be honest), but also to create whatever it is I’ll create from here out.
But first, I’ll fix all the broken images, and then I’ll get to creating a new theme for this site…