The Five Scores Predate the Card by Two Months
Through OrcaRouter both the tuned build and its base are one key away, which makes the dates on the abliterated Qwen3.8 Flash Next page checkable rather than decorative. All five benchmark rows were evaluated on 2026-08-25; the card was released 2026-10-26 [VENDOR] [OURS]. Fifty days. The scores are not wrong — they are simply about a different moment than the one you are buying into.
For a buyer, that gap changes what the rows can be used for.
What an evaluation date actually tells you
A vendor benchmark run answers one question: at the time of measurement, on this harness, the checkpoint scored this. It does not answer what the served endpoint does, because serving is a separate system.
Three things can change between the evaluation and the release:
The checkpoint itself. A tune is not always frozen at first evaluation; a later revision would make the published rows stale.
The serving stack. Quantisation, batching, and the inference engine all affect real behaviour, and none of them is part of a benchmark run.
The guardrails uploader-side. An API can add filters, truncation or refusal handling in front of a model whose weights are unchanged. Whether it does is exactly the sort of thing a benchmark cannot see.
None of this means the numbers are misleading. It means they describe the tuning, and everything else has to be verified separately.
The second data point is missing
On a mature card you would check the evaluation date against a second source: weeks of production telemetry showing latency, throughput, error rate and volume. That is how you tell whether a 73.5 on tool use survives contact with real traffic.
This card does not have that yet. Its measured volume is 1,345 tokens over seven days [OURS] — a probe, not a population. The latency block on the page reports a p50 of 500 ms and an error rate of 75% on that same handful of requests, and neither number should be read as performance.
So the page gives you one kind of evidence — a vendor’s five rows dated August — and no second kind. That is not a red flag about the model; it is a red flag about concluding anything from the page alone.
What to do with a freshly released card
Three habits, in order.
Treat the benchmark rows as a hypothesis. “Strong at code and science reasoning, competent at tool use, ordinary at engineering delivery” is a testable claim. Test it.
Check the dates on everything. A row dated before the release and a traffic figure of a thousand tokens are both facts; a reader who skips them will over-trust the table.
Run your own ten cases. The same afternoon test described elsewhere in this series applies here: ten real requests, four columns, one verdict. At this card’s rates the whole exercise costs cents.
The result of that test is worth more than five vendor rows, because it is measured on the artifact you actually produce.
Why vendors publish early evaluations
The timeline is not unusual, and the reason is straightforward: benchmarks take time to schedule and run, and a launch date does not move for them. Measuring in August and shipping in October is normal engineering.
The failure mode is on the reader’s side, not the vendor’s. A table with no visible date invites the assumption that the numbers describe the product as shipped. The date is present here, in the benchmark metadata, and a reader who checks it gets the right picture.
That is worth saying because most cards do not surface the date at all. Where it is visible, use it.
The alternative evidence, and what it is
For the base model you have a different picture: 757 million tokens of weekly traffic, a measured p50 of 4,945 ms, 163.3 tokens per second, and an error rate of 3.36% [OURS]. That is a populated dataset and it describes what the base actually does under load.
It says nothing about the abliterated build, which is a different checkpoint on a different deployment. The temptation to borrow the base’s numbers is strong and wrong.
If you need serving evidence for this card, the only honest source is your own traffic over the next few weeks. A ten percent slice gives you that in a week, and the log you keep during it is worth more than any table published before launch.
The question a date cannot answer
Even with a fresh evaluation, one thing stays out of reach: whether the tuning matches *your* definition of acceptable behaviour. A benchmark tests correctness on fixed tasks; a security team cares about a boundary that moves with context, jurisdiction and client.
That is the argument for keeping a small adversarial set of your own — twenty prompts that used to be refused, twenty that were not — and running it against every candidate card. It takes an hour to build and it outlives any model release.
Date-stamp your own evaluation
If the vendor’s numbers are dated, yours should be too. Write the date, the checkpoint string and the harness version into the results file.
Two reasons. A security team will re-run the evaluation after the next release, and knowing exactly what the previous run measured is what makes the comparison valid. And if the card is ever re-pointed to a different checkpoint, the dated record is the only way to notice that the numbers moved without you changing anything.
A results file with a date is a measurement. One without is an opinion that happens to contain numbers.
Keep it in the repository next to the code it evaluates. The date belongs beside the result, not in someone’s notes.
The takeaway
Five scores dated 2026-08-25, a card dated 2026-10-26, and 1,345 tokens of measured traffic between them. The numbers are real, the gap is real, and the only way to close it is to measure the card yourself.
Read the rows as the vendor’s claim about the tuning. Do not read them as a description of the service you are about to call.
The final discipline is simpler than any of this: write down what you measured, when, and on which model string. Most organisations cannot answer those three questions about a model they deployed six months ago, and that is why every release feels like a fresh evaluation.
