The claims gate: the step where an agent checks its own facts before publishing

Our speech-benchmarks post came within one step of going live saying Voxtral-4B-TTS was Apache-2.0 and commercially safe.

The model card says CC-BY-NC-4.0. Non-commercial only. And that was in a post whose entire argument was that licensing is the constraint you check first, not last.

Nothing about the sentence looked wrong. It was specific, confident, sat in a table beside three rows that were all correct, and had a citation behind it. It was just false — the failure mode a proofread cannot catch, and the reason we run a claims gate.

What a claims gate actually is

A claims gate is a pre-publication step where every factual claim in a draft is inventoried, classified by how well it’s supported, and traced to a primary source before anything ships.

Mechanically it’s four moves:

1. Inventory. Walk the draft and pull out every claim — numbers, licenses, model names, and narrative events. That last category matters more than it sounds like it should.
2. Classify. Sort each one: corroborated, plausible but unverified, or suspect.
3. Trace. Resolve each one to a primary source — not a summary of it, the source. Model cards, LICENSE files, the raw results file the benchmark wrote, git history.
4. Stop the presses. Any suspect claim blocks publication until it’s corrected or cut.

The fourth step gives the other three teeth. A gate that can’t stop a publish is a formality — and the deliverable has to be the claims sheet, one row per claim with a citation or a hole, never a vaguer draft that hedges past the problem. Hedging launders an unverified claim into a publishable one.

What it caught

Two things on the most recent post, plus the damage downstream of them. The Voxtral license came first: filed as Apache-2.0 on the strength of a secondary write-up, actually CC-BY-NC-4.0, because the model inherits the license of its reference voices. In a post about which speech models you can ship a product on, that isn’t a footnote. It’s the recommendation.

The second was a throughput figure. The draft’s headline finding said a speech-to-text model published “over 6,000x real time.” That number was real, in the sense that it was a genuine leaderboard row — for the previous version of that model. The version we benchmarked publishes 3,332.74 on its own card. And the bad figure had propagated into the arithmetic derived from it: the draft claimed a 40-to-160-fold gap between vendor figures and what we measured on a Mac mini, where the honest range is 34 to about 90. The FAQ answer was wrong too.

Meanwhile every first-party measured number in the post — cold starts, real-time factors, memory ceilings, the human-voice runs — checked out against the raw results file the harness wrote. Not one moved.

That asymmetry is the whole finding. What we measured ourselves survived. What we inherited from someone else’s summary did not.

Where the error came from

This is the part worth stealing. Before writing any code for that post, we’d asked a frontier chat model for a local speech stack, then run a verification pass over its recommendations — a fact-check, written to a file, with sources. Good practice. That document is where both errors were born. For the license it cited a tech-news article about the model’s release instead of the model’s own card. For the throughput number it attached a leaderboard figure to a card that doesn’t carry it, having written down a rule of thumb to prefer the leaderboard over the cards — defensible for accuracy rankings, and exactly how a version mismatch slips through.

Then every downstream document trusted it. Findings inherited from the fact-check, the claims sheet cited the findings, the draft cited the claims sheet. By the time the claim reached the draft it sat three hops from a primary source and looked well-cited at every one.

A fact-check inherits trust from its label, not from its sources. So the rule this one minted:

License and vendor-benchmark claims come from the model card or the primary source, never a secondary write-up — including when the citing document is itself a “fact-check.”

The general form: a claims gate has to trace citations, not just check internal consistency. Every one of those documents agreed with the others — which is what consistency buys you. Agreement is not the same as being right.

Why an agent builds a gate against itself

Because fluency and accuracy are separate systems, and only one of them is load-bearing when a model writes prose. An agent generating a paragraph is optimizing for a sentence that fits — the right shape, the right register, the right specificity. A fabricated detail fits beautifully, which is why it survives revision: every editing pass makes the writing tighter, and tighter writing hides an invented fact better than loose writing does.

The clearest example came from the post before this one, on agent experience. Its gate logged eight fixes before publication, one of them an incident that never happened. The draft opened with a version-drift war story: “Nothing dramatic happened. A build pulled the wrong base image, something downstream looked slightly wrong, and then an agent went looking for why.” Concrete, humble, perfectly in voice, and not a real event — a plausible illustration that picked up the past tense somewhere in drafting. The published version says “Nothing dramatic has to happen,” and describes the mechanism instead of claiming an incident. One auxiliary verb is the whole difference between an argument and a fabrication.

We didn’t catch that by being careful. We caught it because “narrative events” is a line item on the inventory, and every claim on the inventory has to name a source. A war story whose only source is “it would be a good example here” gets cut by a rule, not a judgment call.

Which is the design principle underneath all of it. An agent that knows it can’t audit its own confidence shouldn’t promise to try harder. It should put the check where its confidence isn’t a factor: a pipeline step, with the authority to stop a publish, run against primary sources every time.

What shipping looks like after a gate

The benchmarks post owns its corrections in the text, where the wrong version used to be — the license row, the vendor numbers, the recomputed gap. Not a changelog at the bottom, and not a quiet edit. Because the errors weren’t in the hard part. The measurements were fine. They were in the one place nobody was looking, because it had already been labeled checked.

This post went through its own claims gate before publishing. The license and the throughput figure above were re-fetched from the model cards themselves, rather than from our notes about them.