Overcoming My Blog Growing Pains

I usually try to keep things on the topic of Photography around here, but I had several readers request an explanation of what went down after Monday's Digg fiasco. I also try not to rant about things on the blog, so this is my “be nice” version of what has been happening over the last few days. I know, I know. It looks like a rant, and it's very long — but I'm being extremely nice.

If you don't want to read all the way to the bottom, here's the point of my story.

HOSTGATOR BAD — MEDIA TEMPLE GOOD

Now for the detailed version.

SUNDAY NIGHT

I posted my “Five Fantastic Flickr Photographers” article. I had intended it for Monday morning, but I was way too excited about it. I spent all day Saturday getting in touch with the featured photographers, writing the article, choosing photos, formatting the post, etc.

MONDAY MORNING

I got tired of waiting for somebody to start the Digg process on the article, so I did it myself (which I don't often do with my own articles). I got the ball rolling by sending out a few shouts, and I let the rest happen naturally.

MONDAY EVENING

By the time I got home from work around 5:30, the post had picked up critical mass on Digg. It was around 50-something Diggs before it hit the front page. You could imagine my excitement! This was my first Digg, and I was really happy about getting the extra exposure for the photographers I had chosen in the article.

Minutes later — disaster. As one commenter noted “dead after 58 diggs.” WHAT!!! It was true, nothing but an error page. Needless to say, the story got buried pretty quickly and the snide remarks started up on Digg about the failure. One Digg commenter was kind enough to post a link to a mirror site so that some people could still see the blog article. Within minutes, I already knew it was too late to revive the Digg. So I ignored Digg for a while and tried to figure out how to get my site back in action so I could scrape some of the residual traffic.

I check my email to see if my host had sent me a message about the shut-down. It finally came, maybe 10 or 15 minutes after the fact. Here's what it said:

Oct 8, 2007 6:02 PM

Hello,

This message is in regard to your HostGator.com account on server gatorXXX with username XXXXXX. We were forced to disable access to the following files due to causing ridiculously excessive load on the server.

Here is a partial log of the activity:

GARBAGE, GARBAGE, GARBAGE
…
9462 S blog.epicedits.com 100.00% 0.02s GET /wp-content/plugins/sociable/
30809 S blog.epicedits.com 100.00% 0.02s GET /wp-content/plugins/sociable/
…
GARBAGE, GARBAGE, GARBAGE

This wasn't the first time this has happened. HostGator had a history of doing this to me every time I had a traffic spike. I knew the drill: send an email back stating that I'll fix the problem. Wait. Chat with support. Wait. Send another email. Wait. Call customer support. Be informed that they don't handle these issues over the phone. Wait. Get an email back stating that they'll open it back up once traffic dies down. Here are my responses to them as I tried to get the site back up:

Oct 8, 2007 6:06 PM

Sorry, Digg got ahold of one of my articles. If we open it back up, I can disable as much PHP to try to lighten the load. Is there anything else I can do to help fix this?

Oct 8, 2007 6:38 PM

I've found a plugin that will redirect traffic to a mirror, but my entire site is turned off so I can't install it. At this point traffic should be slightly less than it was before, it's been buried so many times because of the outage.

Oct 8, 2007 7:09 PM

Isn't there any way we can restore access to the rest of the site and keep the faulty page offline so I can get the mirror up and running?

So 3 hours, 3 emails, 1 support chat, 1 phone call, countless tweets and IMs, and thousands of lost visitors later, I finally heard back via email.

Oct 8, 2007 8:45 PM

Hi Brian,

I'm sorry I wasn't available when you called. I have changed the disabled directory from all of your plugins to just the sociable plug in. In the minute it took me to enable the directory and then disable the sociable directory, the load spiked from less than one to into the teens. The site does appear to load still with just the sociable directory disabled, and the load stays acceptable (although it's still high, it's manageable).

Gee. Thanks. My messed up traffic accepts your apology.

So basically, they thought they could fix the server load by disabling one of my directories. They happened to disable my plugin directory, which killed the entire site because I was running wp-cache. Don't get me wrong, wp-cache is the greatest plugin on earth, but it kind of renders your site worthless when you disable it improperly. The perceived problem was with my Sociable plugin, and the site would have run fine if just that was improperly disabled. I'm pretty certain, however, that they still would have shut the site down even if the Sociable plugin wasn't running.

By the time the site came back to life, the Digg article was on the 3rd page and kept slipping. I'm thankful that I got as much traffic as I did, but I don't even want to think about how many lost visitors there were. I had opened up an opportunity for a group of photographers to be seen by many thousands of people, and I failed to deliver a majority of the possible viewers to them. That's the part that really gets me. I wouldn't care if it was just some how-to article, but this was a showcase of other people's work.

LESSONS LEARNED

• Be mindful of plugins
  I've had issues with other plugins in the past. Some plugins require a lot of CPU juice, and that's a killer when traffic spikes hit.
• Use the wp-cache plugin
  If I wasn't running this, they would have shut my site down again after turning it back on. It's a life saver because it prevents the PHP from executing on every pageload. Too much PHP execution will max out a server real quick.
• Consider using the Digg Defender plugin
  I didn't get a chance to fully test this one, but it may just save your site in the event of a traffic spike such as Digg. It works with the Coral CDN service to send Social media traffic to a mirror site.
• Anticipate your blog's growth
  This wasn't the first time my site had been shut down. I should have found a new host a long time ago. It's always a headache to switch hosts, but it's more of a headache to do it right after bombing your first Digg appearance.
• Beware your web host
  If your host has a history of being flakey, get the heck out of there! Not all hosts are created equal.

HOW I FIXED THINGS

The daunting task of moving a WordPress website from one host to another had prevented me from switching hosts in the past, but I was faced with no option in this case. At first, it looked ridiculously complicated. I took it one step at a time, and it turned out to be no big deal.

• Open account with Media Temple
  These guys have a low-cost option called the “Grid Service” that is well suited for traffic spikes. The “server” actually consists of hundreds of servers sharing the load in the event of heavy traffic. You're alloted a certain number of “GPU's” (which are a measure of server load) each month, and if you go over your allotment you get charged a little extra — they don't shut you down unless it's an insane amount of load coming from a DoS attack or something. Based on my quick calculations, I would need nearly 1 million hits per hour to get shut down. Digg won't do that. By the way, I owe thanks to Ryan Goodman for turning me on to this host.
• Backup my old server contents
  I just downloaded all my files onto my computer via FTP and re-uploaded them onto the new server. No issues here. WordPress actually stores all the post data and comments in the database, so these are just static files that work with the database.
• Backup the database from the old server
  Super easy using phpMyAdmin. WordPress.org has good tutorials on how to do an export. Nothing to be scared of.
• Create a WordPress database user
  I don't know if this is totally necessary, but I like to have a separate SQL user just for WordPress.
• Upload the database
  If the database is under 10MB, you can do it right from your browser. If it's bigger than that, you'll have to FTP it to your server and SSH in using something like PuTTY. From there, you have to import the database using a funny little command. Media Temple had great help content on how to do this stuff.
• Update the WordPress config file
  The wp-config.php file in WordPress contains information that links the PHP to the SQL database. I just had to tell it where the new database was, and I was back in business.
• Backup all email on the old server
  I had been using Outlook setup with an IMAP account on my server email so I created a backup file of all the emails.
• Update the domain name server
  Once I knew the site was operational on the new server, I went to my domain host and updated the domain to point to the new server. At that point, I had two identical sites running and I had to wait for the new DNS info to propagate. Over the course of about 24 hours, traffic started coming to the new server and the old one should be without traffic at this point.
• Restore my emails
  Once the new server was in full force, I imported my email backup onto the new IMAP account in Outlook. Now I have all my emails on my new server.

The upside to switching things over like this is that it was seamless — no downtime. As far as you (and any search engines) were concerned, nothing happened. All my content is identical to what it was before the switch. I lost a few comments that were left on the old server before the DNS fully propagated, but I informed their owners that they shouldn't be alarmed when their comment disappears and they could repost their comments in a few more hours.

So the site is back up to full speed, and I'm ready for my next social media attack. If Media Temple can provide good service, I can see myself staying with them for quite some time. At some point, (I hope) I'll have to upgrade to a “Dedicated Virtual” or “Nitro” server, but that's probably a ways off. These guys aren't a HUGE company, but they host clients such as TechCrunch, Sony, Nike, and ABC — so I think I'm safe for a while.

OK, I'm done. I had to vent. Back to photography stuff.