Choose Boring Technology (2015)
336 points - yesterday at 5:48 PM
SourceComments
This is one of my favorite blog posts, and it can basically be encapsulated in the idea of "innovation tokens." It is one of the most useful concepts I have had as a PM / eng leader in my career. It helps actually make the the right tradeoffs, and helps even more in explaining those tradeoffs to colleague of all levels. Highly recommend.
The tricky bit I think though, is deciding what actually is boring, since really we should factor complexity in too.
I work a lot with data in python, for in memory data, the most popular choices are:
- pandas - pyspark - polars
Pandas is the most established, although has a lot of technical limitations, as well as complexities.
Pyspark is definitely an "industry standard" choice, but now you're dealing with distributed computing when you probably didn't need too.
Polars is the simplest in terms of API and not being distributed, but then is the newest.
I can easily imagine a discussion where three engineers all agree that they should choose the "most boring" tech, but all choose a different option.
Engineers should understand requirements, risks, tradeoffs, and potential gains. New technology may be right for that. Novel approaches may be right for that. "Novel" or "New" are only proxies and they're weak.
For example, I may think "New" means untested, but is that true? What if a new project has Jepsen testing, a fuzzing suite, massive compute running tons of oracle tests, etc? I should just say "Choose well tested" instead of "Choose old" - lots of old software is very poorly tested.
Maybe I think that "Old" implies better documentation, but does it? Lots of older projects have insane cruft and weird edge cases that are undocumented and accumulated over years.
Why do we need a metaphor? Why is "innovation token" helpful?
If you're incapable of evaluating a technology in terms of these properties, you aren't a serious developer and "boring" will not save you.
Sit down, write our your requirements, determine candidate solutions, and choose them based on their fit. "Boring" means nothing, it's a vague proxy term. "Well tsted", "performant for our use case", "developers know it", etc mean something.
> MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
Literally every one of these has caused hilarious and disastrous failures for me in my career. But yep, boring.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens.
What if you know NodeJS really well? Or MongoDb? What if you have empirical, verifiable reasons for why they fit better?
I'm a bit tired of "simple" and "boring" and other nonsense words in this field taking up the air in the room that should be spent evaluating solutions on their actual merits.
On the other hand, IBM was late getting into integrated circuits. They had Solid Logic Technology, automated machinery for putting transistors into ceramic substrates to make tiny but discrete circuits. That's what powered the IBM System/360. Worked, but kept mainframe prices high and made IBM late to minicomputers.
Innovation when the problem is hard is more interesting. Look at the history of US long range bombers. The B-29 was effective, but underpowered, and had a lot of trouble getting off the ground fully loaded. So, after WWII, the next development was the B-36, which answered the question "what if we scaled up the B-29?"[1] Six propellers, and four jet engines (added late in the design cycle). Was a sky barge, but it worked, as long as no one was trying hard to shoot it down. No flyable aircraft remain. That was the boring technology approach.
After that came the B-47, which answered the question "what if we scaled up a jet fighter to bomber size?"[2] The B-47 was all jets, no props. It needed solid-fuel rocket boosters (!) to help it get off the ground, and a drag chute to slow it down on landing. It was terrible to fly; its operating speed and altitude were too near the "coffin corner" where stall and Mach buffet meet.[3] No flyable aircraft remain.
Then came the B-52. New engines. New airframe. New design. [4] That's the high-risk approach. It worked. Hundreds are still in active service. Outlasted most of its successors, the B-58 Hustler (a supersonic intercontinental bomber), the B-70, the F-111, etc.
On the commercial side, we have Boeing, whose most successful airliner is a variant of the B-737, which first flew in 1967. But that's another story.
[1] https://www.youtube.com/watch?v=vKQYG_fA2uM
[2] https://www.youtube.com/watch?v=l1-urTRxeEM
[3] https://en.wikipedia.org/wiki/Coffin_corner_%28aerodynamics%...
Using the language of the article, I’d say “push all your innovation tokens into agents” is probably a good move. This means the tech your agents work with should all be boring tech.
Another way of saying this is “use in-distribution technology”. If agents are substantially better at Rust than Zig, probably you should use Rust, even if Zig is “better”. The amount that Zig is better is going to get swamped by the amount that in-distribution agents are better.
(This is not a claim that Rust actually is better, just a hypothetical fact pattern for discussion.)
Instead I like “only work on impossible problems”
Most of them turn out to be impossible, but some of them turn out to be possible.
I’ve never met anyone who could pick 3 and be confident in getting even one right. Tokens are a terrible analogy for innovation or research.
In hindsight I’ve had to sift through hundreds or more to fine one that worked.
I thought this post was helpful when I first started thinking about startups.
After more time, I think boring tech isn’t worth thinking about.
So many jobs are sold as “we’re pragmatist's” and when you show up there’s 5 devs, 50 repos and most of the work is discussing if x requirement should be a new micro service. The product is usually an web app with 10 entities and and an API.
Suppose it’s keeping people in jobs
1. I once worked for a company that had a large Cassandra cluster that was primarily serving the role of a distributed append-only log. This role is as perfect a fit for Cassandra as I can imagine. When we needed a distributed database for authentication, we decided to use Cassandra for it since we had in-house expertise (it was "boring technology") and a cluster we could piggy-back on for a while. When we needed a distributed database for a key-value service, we used the same reasoning and chose Cassandra again. There were many problems with Cassandra that I won't go into, but I will say that straying too far from Cassandra's comfort zone stretched it to a breaking point. And we had some downtime and rough nights because of it. Sometimes boring technology isn't enough, you also need boring workloads for it.
2. I was an early adopter for Rust. Before 1.0, it wasn't clear to most people whether the language would mount to anything. To me, I saw a formal proof assistant being put into programmer's hands and could tell right away it had a bright future. Rust was not a boring technology back then. You might say it is now (for some use cases). Adopting shiny new technologies that measurably improve confidence is not a risk. What's foolish is getting comfortable and complacent with familiar, aka boring, technologies that are difficult to use correctly.
We're "scalable" over here. It's a sexy problem to have.
The example they gave was something like they wanted to use rabbitmq for a new side project (I might be misremembering) but they were forced to make it work with redis instead. The author said that years later he found out that that side project had exploded in popularity and it had coped with it fine because the infra team were already handling the stack and it wasn't some snowflake deployment.
Does anyone remember the post I'm talking about?
Sadly, that is real out-of-the-box thinking.
And generalizations in software development are always wrong.
This should not be new.
Make your shit work. Make it work well. Don't fuck with a thing that's working as desired.
In this way, you'll make everyone's lives at least a little easier.
This should be a mandatory teaching to anyone who wants to use the moniker of "engineer".
Don't let one affect the other.
It's a case by case basis. I wonder why people try to encapsulate these things into "philosophies" that are simple minded and stereotypical.
Choose what's better. That's not always necessarily boring technology. Sometimes it is... sometimes it isn't.