The People against Scalability
In my career in technology, I’ve heard one specific term so many times that it’s lost all meaning.
The stack needs to be scalable. Our team needs to be scalable. Why aren’t we using a scalable database? That thing in particular, that thing over there, it’s not scalable enough!
And it’s not just that it’s meaningless, it’s that I find it entirely missing perspective about 90% of the time. Usually it’s the rather benign need to pad resumes, and have “experience” to brag about to recruiters and bar-buddies. Regardless of the intent, it always has the same effect:
Over-engineering, over-building, and over-thinking everything.
I think this is worse in two camps, the “extreme-enterprise” world, and the aspirational “hyper-growth startup”, both with the same goals and “hardcore motivation”. In either camp, there’s often a misplaced idea that their thingy is so important it will entirely change the world, and do so very quickly. The line of thinking is that the purchasing lead times for typical hardware and software are so slow and cumbersome that we have to be ready well in advance.
And yes, there are times that things pick up much faster than expected, and it’s very prudent to be prepared for that. Setting up a load-balancing group so that your stack won’t fall over when 10,000 people click your sign-up link is a very good idea. But it’s also very easy to go way too far solving problems you might not even have.
Ok, let’s back up a bit. I like to identify the fundamental “problems” as three types:
- Good Problems
- Bad Problems
- Unknown Problems
Of these, Unknown is the worst because it’s unpredictable. You don’t know about the showstopper bug in your code that hasn’t brought down Prod on a friday night, and you don’t know a sinkhole is about to eat your house. These can be mitigated through “emergency planning” but never avoided altogether.
Bad Problems are everywhere. They’re the known bugs, the TODOs, the crack in your basement that just keeps growing. They’re that rash you can’t quite get rid of, and that wart that keeps coming back. They can be solved, often with great effort and little recognition. They’re not sexy and promotion-worthy, but they’re the bedrock of our society.
Good Problems are the ones you want. My steak is too juicy and my lobster is too buttery. My record collection is too big. I have too many new users signing up. These are the problems everybody wants, because it’s some indication things are going well.
People that focus on scalability at all costs are laser-focused on solving Good Problems, but often to the detriment of the Bad Problems. Rather than address the aging sewers, they’d rather just build a new subdivision on the outskirts of the city.
I’ve seen this play out before in software development. A pie-in-the-sky goal turns into a codified rule, and a best case scenarios becomes a “realistic” estimate. Bugfixes get deprioritized in the name of chasing scalability with another rewrite in another trendy language.
Users and customers can sense this, but can’t put it into words. Over-optimizing one area means taking away attention from other places, and letting the Bad Problems stack up. One service can sustain peak loads of 100,000 rps, about 97,250 rps higher than observed, but the “Submit” button doesn’t render properly on the page anymore.
The lesson I’ve learned is that when you’re small, act small. Actually talk to your users. Don’t pretend to have a whole support department, and don’t try to pretend to be a billion dollar unicorn. Your first customers are too valuable to ignore. Don’t assume they’ll come back if you botch it.
And more importantly, focus on the actual problems you have, and deal with the Bad ones first. Yes, someday you might have thousands of sign-ups in a day, but that day isn’t today. And when that happens, that’s a Good Problem to have.