Brutal!

> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion

> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested

I’ve come to learn an interesting dichotomy in software:

Some software is correct by construction, in the sense that the overall design makes sense and the cases that don’t work are generally local bugs.

Some software is incorrect by construction, and it tends to work by some combination of luck and by carefully working around the incorrect design in most or all code paths that could expose the incorrectness. (AI can be quite effective at finding the relevant places to insert workarounds.)

ParticleDB sounds like the latter category. Plenty of large projects have portions that are in one category and portions that are in another.

As an aside, for the last day or two, I’ve been contemplating that the Linux kernel’s handling of folio pinning seems to be in the incorrect-by-construction category. Good times. No, I haven’t found a hard-to-fix bug that I would credibly characterize as a security or likely accidental corruption issue [0], but this seems mostly like luck.

[0] I do appear to have re-discovered a bug that some massive Anthropic dump from a couple months ago also found, and it’s getting fixed. If your application does “fixed” io_uring I/O or certain types of iommu-based or maybe RDMA on a memfd, you might plausibly encounter it on rare occasions. I didn’t bother making a reproducer.

> Given the severity and extent of the safety errors we found, and the process which produced them, Jepsen recommends users halt further deployment of ParticleDB and migrate to other databases as soon as possible. Consider checking and/or rebuilding database state from more reliable sources, if available.

I guess that's what happens when you vibecode a database...

> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available. It is unclear what properties might be proven, or whether they described ParticleDB as built.

> We present 96 issues in ParticleDB, ranging from the pedestrian (crashes, deadlocks, data loss, etc.) to the piquant (partially-invisible rows which could not be deleted, WHERE 4 = 5, dropped tables re-emerging in distorted forms, confused columns, strings turning into integers, and so on).

Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release

It’s wild how bad it is. Not surprising for a vibecoded database, but why did they hire Kyle for a test, instead of first doing ANY testing themselves?

https://github.com/particledbai/jepsen/issues/18 is utterly humiliating. I can’t believe they kept going after getting that report. I would die of shame.

(Edit for clarity: this is quoted from the linked article)

  You could write predicates over non-existent columns:

  root=> SELECT * FROM t
         WHERE definitely_not_a_column = 4;
   a | b
  ---+----
   4 | 5
  (1 row)
  Or ask if 4 = 5

  root=> SELECT * FROM t WHERE 4 = 5;
   a | b
  ---+----
   5 | 6
  (1 row)

Good on them for being good sports I guess, but after reading this I would never touch a piece of software built by anyone involved with this company. The sheer hubris of their claims combined with the specific reality of their broken garbage software (with the contradicted marketing claims and hallucinated documentation) is literally breathtaking.

It must have been darkly amusing to see all these defect tickets coming in, only for them to frantically shovel each one to increasingly overworked LLM agents to apply pig-lipstick while the humans in charge oh-so-slowly start to realize their software quality is not quite what their glazing agents had been telling them and that what they have on their hands is basically sub-mongo-quality slop. What a world we live in, huh?

> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof.

Perhaps they meant formerly, probably just a speech to text problem

> Strings sometimes turned into eight-bit integers (#37)

FINALLY we know what happened with eMudhra's keys!

(Sorry. Really couldn't resist.)

Shipping a bug like "WHERE 4 = 5" returning rows says it all to me

I do not know in what order Kyle et al. wrote the report, but as I progressed through different parts I think I could detect instances of an increasingly fatigued/disheartened/unhinged tone--without compromising the professionalism we expect from Jepsen analyses, of course. These moments of humanity brought me joy.

More cynically, I suspect that many low-quality database projects succeed or fail based more on marketing and hype than robustness. A dishonest product could simply take the line "engineers at Orkes were able to produce a highly sophisticated database system" from the report out of context, quote it on their homepage attributed to aphyr/Kyle/Jepsen, and reap short-term rewards and popularity. I hope I'm wrong.

Imagine asking for a review before checking with the brutally powerful agents we have today all the low hanging fruits. And, at the point we are, you can likely go at incredible levels of automated bugs seeking and do the whole test in house.

A long time ago I worked with Aphyr to evaluate a database with Jepsen. He was extremely professional, and he found a few real issues which we resolved extremely urgently. We took things seriously and wrote a significant amount of testing ourselves, and by the time our engagement began, we were pretty confident. Overall it was a great experience and I highly recommend his services.

By comparison, this report paints an absolute clown show. The amount of anomalies is perhaps understandable (a single architectural flaw can have a lot of impact) but the fact that they found so many disparate issues is telling.

The fact that they then said they addressed the issues, but didn't, is about the biggest red flag imaginable. I would never let any client use this software under any circumstances.

I am wondering how this was allowed to be tested from a corporate strategy point of view of the company Orkes.

This is such a huge reputational hit, seems like a terrible outcome for a Series B company funded by big name VCs

What does this say about their main product?

Since the library claims to be postgres compatible, I wonder if its developers ran the existing postgres jepsen suite against it (possibly with some adjustments). Or why not.

Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.

Aphyr does great work, Jepsen always kicks it out of the park :)

Slop DB written in Rust. They probably changed the landing page after the bad results, you can still see the old one: https://web.archive.org/web/20260518203012/https://particled...

"Written in Rust · Formally Verified · PostgreSQL Compatible"

Great to implement a DB just like human memory, unstable, and untrustworthy.

Aphyr (and/or his team) is the best technical writer in the biz. These Jepsen reports are always so clearly explained. This one is fun because the tone occasionally verges on disgust.

Every database vendor should pay to get their software Jepsen-tested.

Fatality

Is/was ParticleDB open source? There is a Github link on their website, however the only public repo in their org seems to be the Golang client library.

tigerbeetle much different result

https://jepsen.io/analyses/tigerbeetle-0.16.11

https://news.ycombinator.com/item?id=44199592

I've never heard of this system. Came out of nowhere and closed source it seems. Could not find much analysis on it at all which is not the best sign.

No one’s ever using it after this. Sounds like an unnecessary pile of shite.

Is there any concrete evidence they vibe coded it? It seems 100% self-evident, but have we actually ruled out human hubris and gross incompetence?

It’s really great to see more of these @Aphyr, I really enjoy them and learn something from each. Thank you!

Dude please set up an RSS feed I've been reading your jepsen posts since like 2014 and haven't realized how much i missed.

Damn aphyr if there were ever a time to write one of your old school meme filled reports it would have been this one lol

Well done. Nice D&G riffing in the endnotes too

glorious!

Now do ArcadeDB, please. I know it's full of bugs. They fucked up DFS vs BFS and it was in the codebase since they copy-pasted the code from OrientDB.

I wonder what the internal testing pipeline looked like before this. A codebase with that many fundamental issues suggests either no integration tests were written or they were all mocked out to pass green. It seems strange that a team would market a system as formally proven without actually running the workload through even basic consistency checks themselves first.

That part is the weird bit to me. If the existing harness breaks right away, that's the signal to stop and fix things, not to book an outside audit. Running the suite in CI on every build would have caught a lot of this.

-https://github.com/jepsen-io/particledb is currently 404 (probably still private?)-

EDIT: It's up now

[dead]

[flagged]

Congrats to the particledb team for getting this far and thanks to @aphyr for the analysis.

It takes gumption to build a new storage engine. I remember when etcd[0] first made it's way into @aphyr's cross-hairs. We took all of the input very seriously because we knew that a data store which is untrustworthy is useless.

Resolving these issues likely won't be easy, but if you can get to the point where you walk away from Jepsen unscathed the trust in your software will be greatly amplified.

0: https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul