Well either AA is biased bribed riged whatever or chinese models are benchmaxxed or maybe its both

Interesting that changes in the index always happen when „non us big tech“ models seem to be a little to good for comfort.

This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.

The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.

Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.

I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.

The drop for K3.1 Pro & MiMo V2.6 Pro is quite surprising. Are they benchmaxxed?

(EDIT: A child comment has refuted the AA score comments, AA runs using their own harness.)

One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.

On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.

This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.

[flagged]