I am big on reproducibility (nix aficionado) and determinism (flagging test failures are a red-alert, all-hands-on-deck situation in my world) and correctness.

I am also big on testing (the correct things). And nine-nines (big on Elixir).

And... I'm also big on agent-assisted dev. Which requires pretty much every check in the book to stay productive in. And that's fine to me. I've seen bugs that I wouldn't have made myself. And I've also seen my own bugs fixed. They've all gotten fixed in short order. I don't see why this is a problem.

Raise your personal standards.

Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.

Excellent post. People always defend agentic/LLM-driven development by saying, "Well it's good enough", or "It works most of the time."

That may be tolerable for some user-facing app. But what if we start normalizing failures in the libraries, the infrastructure, and the compilers? Everything descends into a mess of unreliability, and that slows EVERYTHING and EVERYONE down.

> This leads to a normalization of inexplicability.

It’s also tightly connected to a normalization of lack of accountability.

> This isn't "getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" -- you still have to do the hard part.

This is probably losing the younger portion of the audience by now. ;)

> When a button breaks on a website, I have a model about what should have happened. Somewhere a contract got broken. [...] I might not have access to debug just an HTTP status 500, but I expect there to be somebody whose job is to understand why the endpoint is 500ing. The ownership is well-defined albeit opaque³.

> For many users, however, the actual experience is roughly just "stupid thing sucks." Software already feels capricious; more failures just change the rate of frustration.

I am betting author does not use cloud services much. It is not just "users", it's developers as well. Github is returning 5xx? AWS service does not work? Your email did not get delivered? Nothing we (developers) can do, "stupid thing sucks".

"Confidence scores" have always implied an anthopocentric meaning that doesn't exist. An algorithm doesn't have "confidence" in the way that a person has confidence, but as soon you put something with that name in front of a business person they assume the number is always a meaningful "letter grade curve" or "universal percentage". I still believe so much that the old quote to "there's lies, damned lies, and then statistics" remains a key to understanding so much why ML is leading to dumb outcomes versus hype. People don't understand statistics, so machines that produce nothing but statistics especially confuse people. (I feel this applies to LLMs as well.)

The "normalization of inexplicability" is indeed infuriating. It has always been bad when it comes to computer software, and it's increasingly creeping into other consumer products that depend on embedded software.

I bought a new electric car recently. For the most part I've been quite happy with it. Shortly after I bought it, it started popping up a warning message saying "check EV system" every time I started it. By the time I brought it into the dealership, the warning had gone away, and the technician just told me something to the effect of "eh, I guess it just does that sometimes, let us know if it happens again." Hardware fault? Software bug? Who can say?

Like most modern cars, it has connectivity and Google Maps built into the infotainment system. The vast majority of the time, it works fine. Sometimes it says it has no connectivity (meaning no traffic data and suboptimal routes) for the duration of a drive, even in areas with a strong cell signal where it normally works fine. Sometimes the car says it has connectivity, but Google Maps still thinks it's offline. Sometimes Maps will actually load and display a route, but the "start navigation" button just spins forever as though it's still waiting for something. Are these related issues? Is there a common cause that might be fixable? Who can say?

(Conveniently enough, the warranty specifically does not cover any failures of software or firmware to operate correctly.)

I’ve validated Jev’s confidence score. Accuracy scales linearly with confidence for the 3 use cases I tested. >.9 it matched a human labeler. I immediately discovered a user behavior I didn’t expect for ~$3. I can now mitigate in real-time due to low cost and latency. This may have a major positive financial impact for all our customers.

Not sure why anyone would feel the need to dunk on this thing without showing a real failure example.

I build ai systems for ops automation and don’t understand the author’s pessimism.

I agree with that Evals are a scarce commodity rn. A business needs to define what good looks like. This is a laborious, and sometimes politically controversial, process.

Given good Evals, frontier llms are a magical tool that can automate tasks and do them more accurately than human ops teams.

Exactly which llm to use depends on the mixture of speed, cost and quality of the output.

Jev makes a claim to expand some regions of the Pareto frontier. I look forward to testing if this is true.

There are many areas of work we can’t automate rn. We cannot create good Evals either because time horizons are too long, or it’s too difficult to create good Evals.

That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.

If you spend more time with a product, you’re more likely to choose to do it again, even if it’s because of failure or annoyance. You’d justify it somehow (I have a leg up now or something). This actually applies to looking at things as well; a brightly colored box on the supermarket shelf is simply more likely to be chosen because you look at it first and longer.

Wasting your time and resources is a power signifier, but getting you to waste your own time and your own resources is hegemony.

"Appeal to adult animated show" is lame enough on Reddit but it's particularly odious blogspam here.

See also "normal accidents" [1], which are an even bigger worst case scenario, which we already struggle with, and will only get worse.

"Normal accidents, or system accidents, are... inevitable in extremely complex systems. Given the characteristic of the system involved, multiple failures that interact with each other will occur, despite efforts to avoid them... while operator error is a very common problem, many failures relate to organizations rather than technology, and major accidents almost always have very small beginnings. Such events appear trivial to begin with before unpredictably cascading through the system to create a large event with severe consequences." [1]

[1] https://en.wikipedia.org/wiki/Normal_Accidents

That’s not even an attempt to be charitable to jev. It can make decisions from unstructured data. Nobody uses anything just because it is cheap…

Another of the ways that software "engineering" is drifting even further away from traditional engineering. Failure mode analysis? Root cause analysis? I have no idea, I just talk to this magic box.

That was awfully specific. Apple has managed to normalize inexplicable failures of things that had been working before for over a decade, across all of their apps and the OS. And they didn't even need AI for that. But AI will definitely speed up the rate of blunder everywhere.

This is the problem building "it does everything" machines.

If you are pitching that your service can do potentially "whatever the client wants" you have such a thin basis on which to provide contracts and guarantees as a provider. The narrower the function, the clearer you can be about what's supposed to happen and why things might have gone wrong.

When you're using probabilities as the fundamental approach to computation, all of that goes out the window. Nondeterminism is powerful because it's insanely flexible, but the cost of that flexibility is predictability and expectation. Determinism was humanity's primary choice for formalisms and technology precisely because it reduces complex problems and situations to repeatable mechanics that are easy to understand. Deterministic tools can't do a lot in the grand scheme of things, but it is precisely these limitations that make them work well in concert and keep them comprehensible.

I've been saying this for years – it's a huge societal problem that most people, including powerful people, view technology in general and software in particular as something slightly magical, something that cannot be truly understood. We're seemingly forever doomed to accept that technical things are a bit shitty and that we'll have to do a rain dance every now and then to appease them. Instead of, you know, throwing out the shitty products and making or buying good ones.

Lovely post, and lovely phrase (normalizing of inexplicable failures).

Inexplicable has a range from “I can’t explain that” to “that is not in the realm of possibility”. Windows failures especially have been steadily moving from A (already pretty sad) to B in the past years and I’ve been thinking about how utterly unempowering that is. Tools stop being tools if they have a secret life of their own.

Excellent post!

It frustrates me quite a bit that the best solution to many issues is "Just try again tomorrow", trusting that the problem occurring on Azure is caused by some obscure backend service being down six timezones from our office, and the thing™ will start working again in a few days when Microsoft fixes said backend service.

Why is the author pretending that it is only now that inexplicable failures are "being normalized"? They've been normal(ized) ever since this industry has existed.

Literally go no further than the age old advice of "have you tried turning it off and then back on again?", and then that actually working.

Did this person never experience the effects of rocking the boat just a little too much? Daring to do a little too good of a job? How?

[flagged]

[flagged]

That can't be gold, must be movie props. Gold is stored in reverse pyramid, not upright pyramid, as to be movable at all. And it is not usually stored that high without shelves, since the floor below should not be indented by half a metre and picking up the last gold at floor level would be more cumbersome.

(Yes, storing gold on a what the room door claims is a toilet is a best practice, as thieves won't look for it there. But now the security through obscurity got leaked /s )

skipped