Deterministic Simulation Testing Is for Everyone
56 points - yesterday at 1:19 PM
SourceComments
The other one is https://alperenkeles.com/posts/vocab-for-testing/
I truly understand our (only) job is to become the oracles of the machine: https://www.osequi.com/manifesto
Iโve been building a framework of my own and itโs already found dozens of important and critical bugs on my team at work.
The next problem to solve after this, though, is formalized intent.
A bug is often a bug if the app is not behaving as intended. What we intend is not always so clear, and it will remain an area of the engineer's and product designer's responsibility, because software is complex, and what we want cannot be computed so simply. Defining and understanding intent will remain a "human bottleneck" for many companies.
But I'm totally convinced we're moving to deterministic execution: n various implementations, in different languages, running on different stacks, and a minimal (maybe even hardware) voter taking care of the n-modular redundancy to pick the correct "answer".
0-day Postgres CVE? No problem: one of the implementation gives bogus answers and is detected as broken.
0-day Java? Same thing.
This shall make 0-day in anything but the voters a thing of the past (and the voters shall be minimal and have a tiny attack surface).
The answer can be an API query but we can go much lower: network packets, deterministically created, by n systems and all going to the voter(s). Whatever the "answer" is, it's going to be deterministic and verified from the lowest possible level.
The implementations shall be written from specs, formally verified, etc. But the one thing that'll guarantee any fuck up is caught is going to be n independant implementations of the same specs.
And as soon as one of the implementation/stack return even just one network packet that doesn't match what the other systems do produce: something is fucked up and has to be fixed.
This shall require throwing a huge lot of the non-deterministic bullshit away but we'll see systems doing just that: n-modular redundancy and voters, at the lowest level possible.
I'm 100% sure we're going there (and AI is going to write most of that for us).