Well, the primary server (host to all the virtual servers within) had a glitch in it's drive array.
Some Background First...
To explain, there's a drive array module which controls the computer's access to the hard drives. With this card, I can pair a number of drives together to logically so they appear to the operating system (i.e. Microsoft Windows, Apple Mac OS, VMware ESX/vSphere, Novell Netware, etc.) as one big drive or a couple of big drives.
When a drive array spans 3 or more drives, the drive array starts to use parity technology for redundancy. This is, in short, a mathematical calculation where a block of data is stored on any of the drives and it's mathematical checksum is stored on the other drives. What this does is provide a high level of redundancy in the event a hard drive within the array of drives goes bad. Very cool so long as fewer than half of the drives in the array go bad at one time.
Back to the Issue...
None of my drives are reporting any issues.
In my case the array glitched (similar to losing power or power spike) and lost the drive configuration. In higher end array modules (and even in this one) I'm supposed to be able to pull the array configuration from the drive back to the card. That failed due to something with the glitch.
What About Backups?
I very likely could have restored from backup if I had backups of each virtual machine - remember though, each virtual machine is essentially a complete computer in and of itself (and I have - correction, had - quite a few). So, unfortunately, I do not have the resources (technically or financially) to do backups of these systems on that level. And I learned long ago with email and database systems, if you don't go all the way with backups (and backup everything) it's best to just do what you can to minimize your losses and downtime.
You did maintenance on the servers the other day. Did that have a play or possibly cause this issue?
Short answer, No. I was done with maintenance on Saturday and the server crashed while I was out buying presents on Tuesday around noon. Plus, what I did had no relation to the drive array system.
My Analogy of the Issue...
Probably the best way to explain this is, if you drive a full stretch Cadillac and go to the grocery store and fill up that huge trunk with groceries, and then 30 miles down the road simultaneously all four tires blow and the transmission breaks, do you think it's because the trunk is full of groceries or is it possible something else which may have happened (like running over glass in the roadway)? :-)
As for backing it up, we could probably have four spare tires in the trunk as well as a spare transmission - but it's unlikely anyone would go to that length of protection on a personal basis.
For anyone using my email server, I've transferred it out to a service (provided by No-IP.com) so it will no longer go down (at least - not because of something I've done, or my Internet connection, or my server glitching). I've gone ahead and paid the first year service fees to get the transfer going and email is already flowing in. This service was already providing basic anti-spam services (I didn't have them set too high) so it will continue to do that - and we can also increase the spam protection services as well as the settings for individuals. I will also be re-setting up the reflection addresses which existed in my own server (things like group addresses which forward/reflect email to all members of the group).
What are the differences?
Here are the differences which you'll notice:
- No server based calendar. Personal calendar within your email client (i.e. Outlook, Apple Calendar, etc.) will be useable.
- No server based address book. I think everyone had their own address book on their computer anyways. This will still work the same.
- Web Email will have a different look and feel.
If you wish to use another email service (such as Comcast or Verizon) and want your "@flaming.ws" or "@feely.ws" email forwarded to it, please let me know.
Also, the reverse is true - if you wish to use the "@flaming.ws" or "@feely.ws" email service and currently have another service, please let me know. Either way can be accommodated and this email service is guaranteed to be up 99.99% of the time.
For those of you who are using my Corporate Antivirus license, that server is also gone. Your Antivirus client will be fine for a couple of weeks if you wish to buy another (I would highly recommend it). I would recommend Trend Micro or AVG. I've not used AVG but have heard it is good from others. For those who have multiple computers (I think everyone does) and want to purchase Trend Micro, I've provided a link to the online store which allows you to purchase Trend Micro for multiple computers.
A couple little tricks on purchasing Antivirus software - specifically Trend Micro
- Purchase it ONLINE from Trend Micro. Don't go to the store! You'll pay more at the store!!
- Buy the best one. It's worth it!
- I really like Trend Micro, but when purchasing Trend Micro or any other software, be aware of the "Download Protection Service". It's a waste of money! You'll always be able to download a copy of Trend Micro's software - no matter the situation!
- Don't skimp on virus protection! Remember everything you have on your computer - and what a malicious person could do with it. Trend Micro is having a serious sale right now so now's the time to purchase for your Windows pc.
Hit the "Trash Can" icon to remove the "Download Protection Service" and save yourselves about $8.
Is that everything?
Surprisingly, that's about it. Amazing how a complete system wipe can bring you back to a clean slate.
The only other thing is an apology for any inconvenience or issues this may have caused to anyone. Know that your email is now in better hands now and will not go down.