2008-11-12 Russell pouring ketchup at Elmo's Diner

SuperFast Performance Monitoring Systems was an ordinary, average production monitoring company, promising to keep an eye on web traffic and alert customers if they needed to scale up their cloud hardware to match incoming demand. Their core product was simple, straightforward, and solid, doing what it claimed to do without incident ... but it wasn't sexy. Enter Wile E. Coyote, Supergenius Programmer, hereafter called Will for short.

Will didn't seem to be a bad programmer, at first. He was a little slower than he promised, but his task was a complex one: he was to generate multi-variable graphs of the performance of the apps, something your ordinary front-end programmer wasn't necessarily versed in. With a little help, he got the visualizations running. They were sleek, sexy, and downright spectacular, and they wowed the pants off the Marketing folks.

The feature shipped, Will was given a certificate and a hat to reward him for going above and beyond, and sales went through the roof. Everyone was happy.

Except ...

It didn't start out terrible, not really. Of course, they dogfooded their own software, so they kept a close eye on the performance of the graph feature, but it was above and beyond the specs, so people gradually began ignoring that corner of the monitor.

A month or so later, however, it wasn't spectacular anymore. And another month after that, it was looking sub-optimal. Sluggish, even. Nothing too bad, nothing that would piss off users—quite—but something to be concerned about.

"Not to worry, not to worry. It's probably just the disk I/O. I'll just optimize the caching algorithm a bit, we'll be back in tip-top shape." Will's excuse sounded plausble enough, and he dove right into it. And sure enough, the speed bounced back ... a little.

Two weeks later, the performance gains were gone. There was obviously a problem, and it was only getting worse.

Will didn't look quite as confident this time, but he dove into the project with only a little less gusto than before. He drove out to the datacenter to test network connections, he switched to a more highly optimized image generation library, he added more RAM to the database. Each change added just a bit more performance ... which was lost again the next week, or the week after that.

Will wasn't getting any shiny certificates now. He was getting phone calls from department heads and emergency strategy meetings. He took to hiding in the breakroom with his laptop so people couldn't find him to ask for status updates. He lost weight. His hair went frequently unbrushed. All he did now was seek out this performance issue as the meter crept further and further into the "red zone". There was no getting around it: the feature was slow.

Marketing started drafting up new sexy features to sell product, and gave them to someone other than Will. He was a pariah, accursed, unable to escape the burden of his past.

Still, he perservered. It was SQL, he argued. The overhead from relational tables was eating up their performance, they needed to move to Mongo yesterday.

Nobody was having it. Their other report visualization tools were working fine on SQL Server. What made his fancy graphic so different?

Will became a shadow of his former self, always sulking in the corner during status meetings while mumbling about Cassandra. Finally, he cracked, and left the company for greener pastures. Nobody threw him a going-away party.

A couple weeks after his departure, a coworker named Brad decided to take a look into the code himself. He was working on Marketing's newest idea, and wanted to learn from the "supergenius" who'd built the previous toy.

There was a ton of filtering logic in the code. Concerned, Brad took a look at the DAO's query to get the user's information, the very first query in the series of operations that led from historic data to dynamic image:

SELECT * FROM Users
[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!