Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.

I stopped trusting benchmarks ever since LLMs started speaking alien like English, if I can't understand what they are saying how can I trust them.

Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.

Of course having a holdout set is better than not having one. It’s just not a silver bullet.

Not trying to defend anyone, but in my experience the latest models have been performing significantly better than 8 months ago. So, in my book the extent of over-fitting on benchmarks seems to be covering my use cases.

> it's become easier than ever to make serious performance gains

Is that true and if yes why? I was under the impression that it would become more difficult over time to make serious performance gains, which would also fit with reaching for benchmark hacking rather than relying on natural gains.

Cheating and overfitting, as discussed in the article, are the most obvious problems with benchmarking LLMs. But there is also the aspect that, at least for closed models, the tokens still have to be sent to the provider's servers for inference. This makes the holdout set not as held out as it may appear. OpenAI and Antropic probably don't care about your private set of regex benchmarks, but for the headline "closed" benchmarks, I'd be surprised if they haven't collected a nice representative set of "holdout" problems to be examined at leisure.

Nice article. As perf optimisation has become more like a machine learning problem, maybe there is opportunity to use other techniques for checking generalisation. Like e.g. cross validation

Maybe use fixed bugs from closed source to benchmark these "frontier" models.

I think an interesting direction for benchmarking is to take inspiration from metamorphic testing. Metamorphic testing is a way of extending property-based testing (in which you ask the test framework itself to automatically generate many random (input, expected output) pairs to test for you, instead of manually writing individual tests yourself) to handle situations where (a) it's hard to independently come up with the right answer for a specific given input, but (b) relationships between inputs imply checkable relationships between outputs. For example, if you're trying to test your own implementation of sin(), it's hard to automatically generate random (input, expected output) test pairs without using a separate, trusted implementation of the sine function, which may not be available; but one thing you can easily do is check, for many different random x, that sin(x) == -sin(x+180).

How to apply this idea to benchmarks? Basically, look for simple transformations of the input instances that should yield simple transformations of the outputs -- in particular, outputs that, in a non-overfitted implementation, should take the same length of time to compute. For regexes, you could rotate a subset of non-magic characters in both the string and the regex (e.g., A -> B, B -> C, ..., Z -> A).

Another example would be to reverse both the string and the regex (taking care to handle parenthesised regex subexpressions correctly) -- unlike the previous one, it's not expected that the transformed instance will take the exact same length of time, but there should not be too much blow up.

[flagged]

[flagged]

[dead]

[flagged]

[dead]

[dead]