Down but not out and lessons learned

Some of our regular readers may have noticed that Online Marketing Blog was not available for about 26 hours this week. We were informed after calling our hosting company, that the site was under attack by hundreds locations sending unresolvable requests or a denial of service attack. The host shut the site down without telling us as it was on a shared server and the increased traffic was affecting other web sites.

After numerous attempts to get access to our data with ambiguous responses from our host, we finally decided to change hosts. I planned to do this on Friday night so DNS propagation would occur over the weekend. But the hosting company was not able to do anything to resolve the situation in the short term so we made the call to change hosts with more support resources asap.

The database was restored and everything finally back by 4pm CST yesterday.

Losing front and back end access to such an important part of our public communications was certainly an alarm to our hosting situation, which we had taken for granted over the past 2 years. However, situations like this are both challenges and opportunities for people to show their problem solving and technical skills.

A few lessons learned:

1. Don’t just backup your blog database, backup a ghost or mirror image of the whole thing.
2. Instill with the appropriate team members the importance of uptime and the fact that there is no time clock for solving such issues. It’s a 24/7 situation.
3. There is always more than one way to solve a problem, especially if it’s a new and difficult problem.
4. Diplomacy wins over threats with vendors, but only to the degree that you are able to effectively communicate a sense of urgency.
5. Make sure you host your blog or web site with a provider that offers 24/7 support.
6. Times of need are very telling in regard to who your “friends” are.
7. The absence of blogging provides an excellent glimpse at whether you’re spending time productively or not.

Now we’re back online and everything is ship shape. Thanks to Thomas for pulling though in the final hours to get everything back online.

Update: Apparently, Online Marketing Blog wasn’t the only one hit by a DOS attack yesterday.