A couple million lines of Haskell: Production engineering at Mercury
283 points - today at 12:01 AM
SourceComments
There was very little need to fix bugs, and usually functionality could be added very rapidly (e.g., we were the first to achieve a certain certification for hosting sensitive data on AWS).
Though occasionally functionality had to be added more slowly, because we had to write from scratch what would be an off-the-shelf component in a more popular platform. But once we did it, it worked, and we were back to our old velocity, and not slowed by the bloat and complexity of dozens of off-the-shelf frameworks. We could also adapt rapidly because we controlled a manageable platform, which is how we were able to move fast to AWS when there was a need.
The system also had some technical bits of architectural secret sauce from the start (for complex data, and Web interaction), which enabled a lot of rapid development of functionality, and also set the tone for later empowering smartness.
One difference with our system, from the Haskell fintech, was that our team size was very small (only 2-3 software engineers at a time, and someone who managed all the ops). So we didn't have the challenges of hundreds of people trying to coordinate and have a coherent system while getting their things done. Instead, there was usually one person doing more technical and architectural changes to the code, and a prolific other person doing huge amounts business logic functionality for complex processes.
With careful use of current/near-term LLM-ish AI tools, software development might find some related efficiencies of very small and incredibly effective teams. But the model that comes to mind is having a small number of very sharp thinkers keeping things on an empowering and manageable path -- not churning massive bloat to knock off story points and letting sustainability be someone else's problem.
Haskell is admittedly, probably the most powerful widely (or even somewhat widely) used language for doing this, but this general pattern works really well in Rust and TypeScript too and is one of my very favorite tools for writing better code.
I also really like doing things like User -> LoggedInUser -> AccessControlledLoggedInUser to prevent the kind of really obvious AuthZ bugs people make in web applications time and time again.
I've found this pattern to be massively underutilized in industry.
As a customer of Mercury, it's truly one of the critical companies my toolkit, and I just can't help but feel that their choosing of Haskell made their progress, development and overall journey that much better. I realize that you can make this argument with most languages, and it's not to say that a FP lang like Haskell is a recipe for success, but this intentional decision particularly pre "vibe coding" and the LLM era seems particularly prescient, of course combined with their engineering culture that was detailed in the post.
> This is not a complaint about volunteer maintainers. It is simply one of the ambient risks of building serious systems on a smaller ecosystem.
And so instead of paying the lib authors who already have domain expertise and know their codebase, they chose to rewrite it from scratch/fork without contributing back. So classic.
If only cross-compilation became easy so that I can develop on my chip Macs and deploy on x64/AMD Linux servers.
>statically linking Haskell binaries is quite a challenge
>build requirements really slow down the process. I have to use dockers to help cache dependencies and avoid recompiling things that have not changed, but it is still slow and puts out large binaries.
Also, the Docker-based deployment takes a lot of time as it needs to recompile each module. While you can cache some part of it, it's still slow.
Meanwhile with Go it's painless. And i am not the only one having this issue:
https://news.ycombinator.com/item?id=47957624#47972671
Such a shame Haskell is beautiful and performant language still build is slow.
The Mercury site also looks way better than most other banks I have ever used (load speed is also very good.) On the danger of seeming like a shill (I'm not), I'm tempted to try them out.
what does that mean?
In languages with option types, if you want to weaken the type requirement for a function parameter, or strengthen the guarantee for a return type, you have to change the code at every call site. E.g, if you have a function which you can improve by changing
- a parameter Foo to Option<Foo> or
- a return value Option<Bar> to Bar
you would have to change the code at all call sites. Which could be anything between annoying and practically impossible.
In languages that solve null pointer errors instead with untagged union types (like TypeScript or Scala 3), this problem doesn't occur. So you can change
- a parameter Foo to Foo | Null or
- a return value Bar | Null to Bar
and all call sites of the function can remain unchanged, since the type system knows that weakening the type requirement for a parameter, or strengthening the promise for a return type, is a safe change than can't cause a type error.
So yes, option types do avoid null pointer exceptions, but they solve the issue in a very suboptimal way.
>A couple million lines of Haskell, maintained by people who learned the language on the job, at a company that moves huge amounts of money? The conventional wisdom says this should be a disaster, but surprisingly, it isn't. The system we've built has worked well for years, through hypergrowth, through the SVB crisis that sent $2 billion in new deposits our way in five days,1 through regulatory examinations, through all the ordinary and extraordinary things that happen to a financial system at scale.
This one is quite telling. Do people have counter examples?
Try a better programming language next time, dagnabbit!!!
(There will be downvotes I suppose. More lines of code the better?)
I’ve been using Mercury for 5 years. In that time, I’ve been able to wire transfer money without having to worry it might disappear (functionally impossible at certain other banks), created hundreds of virtual debit cards each with their own limit and pulling from different accounts, created dozens of accounts (a “place to put money”) named by function (each of my household utilities gets its own account, with an automatic rule to pull in money whenever it gets paid out), and… well, I think that covers everything.
This has given me unprecedented insight into my financial life. I know exactly how much I spend on groceries, on each utility, and on entertainment. I can project ahead and get a burn rate for my household. And my ex wife uses it too, on the same login, which is as easy as “make an account named with her first name” and a corresponding virtual debit card.
I’m convinced the only reason people don’t use Mercury is that they don’t know what they’re missing.
You have to pay for personal banking (a couple hundred a year iirc), but the business banking is free. If you want to try them out, you can start an LLC for a few dollars (at least in Missouri) and get overnight access to Mercury. All that’s required is your EIN.
They’ve been one of the single best products I’ve ever used. The sole wrinkle was when they canceled all their existing virtual cards due to reasons, which threw my recurring billing into chaos. But every great company is allowed at least one mega annoyance, and that one was a blip.
If you’re wondering whether to try them out, the answer is yes, and I’m excited for you to discover how cool it is. https://www.mercury.com