What is an AI benchmark? why leaderboards keep lying to you

TL;DR: Benchmarks are standardised exams for models. They are useful for tracking the field and nearly useless for choosing a tool for your work. Build a five-prompt test of your own tasks — it will tell you more than every leaderboard combined.

What they measure

A benchmark is a fixed set of questions with known answers, scored automatically. Common families:

• Knowledge and reasoning exams — graduate-level multiple choice across dozens of subjects.
• Coding — can the model produce code that passes hidden unit tests, or resolve a real GitHub issue.
• Maths — competition problems with a single verifiable answer.
• Human preference arenas — anonymous head-to-head votes on which of two answers people prefer.

Why the numbers mislead

1. Contamination. Benchmarks are public. Public text ends up in training data. A model may have effectively seen the exam. Labs try to filter this; nobody can guarantee it.
2. Teaching to the test. When a benchmark becomes the scoreboard the industry reports, optimising for it becomes rational — and the score decouples from general capability. (This is Goodhart’s law, and AI is a textbook case.)
3. Ceiling effects. Once every frontier model scores 88-92%, a two-point gap is noise plus prompt formatting, not a meaningful difference.
4. Selective reporting. Launch charts show the benchmarks where that model wins. That is marketing, not fraud, but read it as marketing.
5. Preference is not accuracy. Arena rankings reward answers that look good — confident, well-formatted, appropriately long. A fluent wrong answer often beats a hedged right one.

What benchmarks are genuinely good for

Direction, not decisions. They show that the field moved, that a class of problem became tractable, that a small model now matches last year’s large one. That is real information. It just does not answer “which should I pay for,” because your work is not the exam.

Build your own eval in twenty minutes

1. Collect five real tasks you actually do — a client email, a summary of a document you know well, a tricky formatting job, a factual question in your niche, a rewrite.
2. Write down what a good answer looks like before testing. This prevents grading on vibes.
3. Run all five through each candidate with identical prompts.
4. Score against your criteria, not against which felt smarter.
5. Keep the file. Rerun it whenever a new model ships. Ten minutes, and you will know within a day whether an upgrade is real for you.

The factual question is the important one: use something you can verify, because fluency masks error. That is the whole problem with hallucinations — they read exactly like correct answers.

If you want a starting point rather than building from scratch, my four-way assistant comparison is structured this way: real tasks, stated criteria, no leaderboard screenshots.