What I learned from DailyIllini.com
Waaaaay back in 2009, DailyIllini.com decided to get out and build and maintain the web incarnation of the University of Illinois’ student newspaper all on their own. The site’s former host, College Publisher, was no longer meeting the paper’s needs (and wants), and on top of that, it’s future was uncertain (it had recently been acquired by Viacom, it looked like prices were going to go up, etc.).
After much discussion and research, though, in retrospect, not nearly enough, we ended up deciding on Drupal as the content management system for the new site. As the company’s IT director/web developer had recently left for a new position, it was the first major project I (and my meager two years of professional experience) was in charge of. But I was confident, and I had someone on my team who was already familiar with the software.
What I learned was… Drupal is a beast of a CMS. Very powerful, very complex, and perhaps most importantly, very, VERY resource hungry.
It was already a beast, but we somehow managed to turn it into a monster. Our plans for the site were grand, to say the least. Multimedia, user interaction, social media integration, widgets that pulled in content from other company sites… all sorts of bells and whistles to make it stand out in a world of cookie-cutter news sites.
We very quickly found out that, while it ran great when it had no real traffic hitting it, the second it was out in the wild for the general public to run rampant on, our poor server ground to a halt. The development timeline for the site was much too short… and on top of that, following the departure of our IT director, we were lacking expertise on little things like load testing, and server configuration.
Long story short, we had to rethink our plans a bit. Caching was added (to the point that I’m not sure there was anything else we could possibly have cached), user interaction was scaled back, queries were optimized, and the bells and whistles were trimmed down to more realistic levels. We got our hosting provider involved and set to work tweaking server settings as well, and, after a very rough and rocky initial launch, finally got things running pretty smoothly.
Speed was still an issue, however… and even though our server’s load wasn’t constantly skyrocketing anymore, it still managed to lock itself up on a fairly regular basis. Ultimately, it was decided that we simply needed more hardware.
With that in mind, we upgraded to dual servers, with one dedicated exclusively to hosting the site’s database while the other served up the Apache/PHP end of things. And it did help… for awhile. The problem with sites like DailyIllini.com, though, is that they just keep getting bigger. Our database eventually cleared the 5GB mark, and we started seeing some very familiar problems cropping up. Load would spike, the server would lock up, and the site would go offline.
And while the downtime wasn’t horrendously prolonged (we were well within 99.9% uptime per month), it tended to occur at very inconvenient times, on days when our traffic was at it’s highest.
Eventually, DailyIllini.com had to be migrated to a new, less complex, less heavily-customized CMS. It is now hosted by Detroit Softworks, a company very similar to College Publisher, minus the Viacom buy-out.
So, what, you may ask, did I learn from this experience? Well:
1) Timelines and deadlines are good things, but don’t let executive pressure overrule the IT department’s common sense. A longer development period and more thorough research of our options would have saved us a lot of headaches.
2) Have a plan for caching in mind from the start. Start with moderate caching and enable more if and when you need to. Drupal’s Boost caching module is what saved us when things when wrong at launch… but had we installed it from the beginning, we could have avoided a lot of stress.
3) Load testing is important and should not be an afterthought. If you don’t know how to do it, either learn, or find someone who does.
4) Be aware of your limitations and plan accordingly. Even cached, Drupal is resource hungry out of the box, and we did things with it that probably doubled or even tripled it’s consumption. In the end, that was our biggest downfall. Drupal, while powerful, was probably not the best choice for what we had envisioned.
5) Most importantly… Trust your instincts. I was unsure about Drupal even after the decision had been made to use it, but didn’t speak up because of my lack of experience managing a project.
So… failed experiment, learning experience, nightmarish mess, or all of the above? At least I can say that I came out of it more knowledgable than I was when I went in.
