A model with a better spot on a public leaderboard can lose to a cheaper one on your task. Leaderboards are built that way: they answer a different question.
Why a public leaderboard does not answer your question
Public tests measure general ability: math, code on generic problems, factual recall. Your task is almost always narrower and more specific. Parsing one document format, sorting into your own categories, writing in your product's tone.
There is a second reason to distrust the table as it stands. Popular benchmark datasets have been public for a long time, and parts of them may have leaked into the training data of newer models. Then the model "remembers" the answer instead of solving the problem from scratch.
The third reason is the simplest. A leaderboard averages over thousands of unrelated queries, while in production you have one type of request repeated thousands of times. An average score on someone else's mix tells you nothing about that one case.
What an eval on real examples looks like
The idea: take twenty or thirty real examples from your product, fix what counts as a correct or acceptable answer for each, run them through a few models and compare on one metric.
You do not need a lab. It is a quick check that answers a single question: which model and which prompt give the best result on my data for my money.
A first set in twenty minutes
Step one, five minutes. Collect twenty real queries your product has already received, or write close copies. Skip synthetic ones: you need the phrasing users actually type.
Step two, five minutes. For each query, write what a good answer looks like. One line is enough: "states the amount and the date", "does not invent a product that does not exist". Formal metrics can come later, a short criterion is enough to start.
Step three, five minutes. Write a small script that sends all twenty queries through the API with one fixed prompt and saves the answers to a file. Pseudocode:
for item in dataset:
response = call_model(model_id, item.prompt)
save(item.id, model_id, response, tokens_used, latency)
Step four, five minutes. Run the same set on a second, cheaper model, and on a third candidate if you have one. Put the answers side by side and mark by hand which ones pass the criterion.
Score it instead of eyeballing it
A rough metric beats an impression. The simplest one is the share of examples where the answer passed your criterion. Twenty examples, fifteen passed: 75%.
The second number is cost. Divide the cost of the whole run by the number of examples to get a price per answer. Now the question changes from "which model is smarter" to "which one gives the quality I need for less". Often the winner is not the flagship but the tier below it, given a slightly longer prompt.
If mistakes are expensive, add a third metric: the share of cases where the model did not just miss by a little but confidently made something up. Those answers cost more than an ordinary inaccuracy.
Mistakes that make a homemade eval lie
Three to five examples give a random result: a model can get lucky or unlucky on specific wording. Twenty is a sensible floor, thirty to fifty is safer.
Invented examples in place of real user queries are another trap. Synthetic prompts tend to be easier than real phrasing, so the score comes out inflated.
The third trap is a different prompt for each model. If every model got its own polished wording, you cannot tell what you compared. The models, or your skill at writing prompts?
When to revisit the set
The set does not stay fixed. Update it when the input format changes, when a new class of query appears and when a new version of a model you already use ships. An old eval carried over to a fresh version can be wrong in either direction.
Keep the examples and the response logs next to your project code, not in personal notes. In six months it is the only way to explain why this model was chosen.
In short
A public leaderboard describes an average task in an average world. Your eval on twenty real examples, with a price per answer, describes your task and your budget. Start small: twenty queries, one script, two models. The first number shows up within the hour.