The leader boards are from the pre Fabrice Bellard days, btw. Neural network modeling helped finding better patterns in text.

Also, you could say the same for the related data search problem. How to prepare data, so that it can most efficiently searched. Smallest encoding vs fastest search. Databases are mostly very, very stupid compared to more data-specific tuned algorithms. Like factor 1000 slower and bigger.

Related:

Data Compression Explained (2011) - https://news.ycombinator.com/item?id=40631931 - June 2024 (1 comment)

Data Compression Explained - https://news.ycombinator.com/item?id=5931493 - June 2013 (14 comments)

Data Compression Explained by Matt Mahoney - https://news.ycombinator.com/item?id=1179242 - March 2010 (1 comment)

Isn’t the idea of AI precisely to find universal compression from arbitrary input data, at least with LLMs?

I say transformers are the best compression systems

This is the guy who created Zpaq btw. Super interesting but niche backup/archive software.

does anyone have any sources to read about ai-based compression?

I remember hearing a lot about "compression is a lot about prediction", but I don't remember reading any practical result

Matt is a great guy to explain this kind of stuff. He's very helpful.