How the verdicts are actually produced.

What the system does

The technical problem is not writing comparisons. It is gathering distributed, adversarial, multilingual opinion about a fast-moving object and separating signal from vendor noise — where the noise is well-funded, deliberately shaped for search, and frequently indistinguishable from first-hand experience at the surface level.

Three properties make that tractable. Every voice is reached through a permalink, so it can be re-read. Every source carries two independent scores rather than one, so "credible" and "disinterested" cannot be silently traded against each other. And every published claim is checked against its own receipt by an agent that did not write it, whose verdict is cryptographically bound to the exact bytes it approved.

The pipeline

Six stages. The sweep parallelises; the verification does not — you cannot verify a card that does not exist yet. The two loops are the interesting part: one runs until a card passes, the other runs when the world changes.

Solid arrows are the build path. The amber loop is the adversarial FAIL cycle. The dashed loop is event-triggered — see freshness, where the honest answer is that nothing is on a timer.

Reaching the sources

Channels are reached through agent-reach, an MCP layer over per-platform clients, because plain HTTP fetching cannot see most of what matters. Ordinary pages come through direct fetch or Jina Reader. A logged-in browser is the last resort, used once, operator-authorised, and disclosed on the card that needed it.

Two axes, never collapsed into one

Every source carries reliability and neutrality as separate 0.0–1.0 scores, per (source, topic) pair rather than per domain. Both are ours, not the publisher's, and both are printed in the ledger on the card itself.

Collapsing these into a single "trust" number would make that page either uncitable or verdict-bearing, and both are wrong. Split, it does exactly one job: primary for prices and tiers only, never allowed to carry the verdict. The same split lets a hostile venue raise a source's weight — a Codex-favourable head-to-head posted in a Claude-partisan subreddit is more credible for being posted there, not less.

"ai-coding-agents": [
  { "source": "api.github.com/repos",  "type": "primary",     "reliability": 1.00, "neutrality": 1.00,
    "note": "License, language, created, stars. Objective, re-runnable." },
  { "source": "hn.algolia.com",        "type": "primary",     "reliability": 0.90, "neutrality": 0.85,
    "note": "One entry per thread, real points/comments." },
  { "source": "snorkel.ai/leaderboard", "type": "eval house",  "reliability": 0.90, "neutrality": 0.85,
    "note": "Its own two surfaces disagreed on #1 — treat top-of-board gaps as noise.
             Sells data services = mild stake." },
  { "source": "xiaohongshu.com",       "type": "sentiment",   "reliability": 0.60, "neutrality": 0.70,
    "note": "SIGNED urls only. EXCLUDE reseller/中转-API vendors, who saturate the topic." },
  { "source": "anthropic.com + openai.com", "type": "vendor", "reliability": 0.90, "neutrality": 0.10,
    "note": "Primary for prices/tiers ONLY. Never a verdict." }
]

Stake detection

Stake is a property of the voice, not the domain, so it is detected per source at sweep time and the exclusion is published by name. The classes that get excluded from the voice count:

• Vendor employees and official accounts — including the highest-engagement post in an entire corpus, which is precisely why engagement volume is used as a signal nowhere on any card.
• Sponsored reviewers — one of the highest-reach English-language "tests" of a model discloses vendor sponsorship in its own description. It carries nothing.
• Affiliate and lead-gen properties — of ten first-page results for one comparison, four were commercial or affiliate; for a second phrasing, six of ten.
• Comped beta testers and embargoed access — one detailed production writeup was pulled from the card face after its own second paragraph disclosed pre-release access under embargo.
• Course sellers, ambassadors and resellers — checked at the bio level, not the post level.
• Second-hand relays — a compilation of someone else's benchmark data is a citable source but not a voice, and is excluded from n.

The eval loop, and what it costs

In the vocabulary this is a verifier loop over a retrieval pipeline, run adversarially, with the corrections log as its published artefact. It is not novel. What is unusual is that the loop is allowed to be expensive, allowed to fail repeatedly, and its failures are printed on the product.

A synthesizing agent writes the card. A different agent — with no memory of writing it — audits it against the receipts directory and returns PASS or a numbered defect list naming the exact field. The builder fixes; the verifier re-runs. Only a PASS publishes. Self-review does not count, because an agent that both gathers and judges its own work reliably misses its own mistakes.

A claim that got dropped

A card cited a quotation from a B站 video. On verification it could not be located — not in the video description, not in 184 comments including all eight uploader replies, not in the reply API. The quote was deleted and treated as fabricated, along with the claim resting on it. No attempt was made to find a weaker source that would support the same sentence; the sentence went too.

When the brief disagreed with the source

A verification round was briefed that a thread stood at 187 comments and one commenter at 20. The live source read 183 and 19. The card printed 183 and 19. This is a small example on purpose — the rule is only worth anything if it also runs when the stakes are trivial.

The expensive one

A third independent verifier failed a card that two rounds had already improved, and found four claims that were false on the card's own publication date — a vendor had changed which plans included a model that same day. The announcement was inside two sources the card already cited: one author's very next post, and a note whose title was the announcement, quoted from its last paragraph while its first went unread.

Verifying the verifier

The strongest evidence that the loop is real rather than ceremonial is the second-order catch: the register of cut material was itself certifying a true sentence as fabricated. A quotation had been removed as invented; it was genuine, correctly attributable to a source the card already cited, and the original error was a misattribution rather than a fabrication. A false accusation of fabrication, living inside the artefact whose entire job is to prevent fabrication, is worse than the error it replaced. It was found by a verifier auditing the corrections rather than the card.

# Adversarial verification — round 6
VERDICT: PASS
VERIFIED: 2026-08-10
VERIFIED-CARD-SHA256: 20c797ce…50f2887   ← recomputed at build; a one-byte edit voids it

## Rounds on this card
R1 FAIL — 8 defects
R2 FAIL — 2 (half-fixes; both survivors in verdict_note, the most-read field)
R3 FAIL — 2 (12 card-face citations with no ledger row)
R4 FAIL — 2 (the page itself misreported which rows were quoted)
R5 FAIL — 2 (a stale quantity, spelled "594.0 GB" — the guard searched "594 GB")
R6 PASS
R7 scoped re-sign after fixing R6's non-blocking item

Every FAIL was right.

A bug is not fixed until it cannot silently return

Content verification catches wrong claims. It does not catch a generator that renders the right data wrongly, or a guard that has been passing for a month without testing anything. That is a separate register with a separate discipline.

70 defects are registered, numbered 1 to 70 with no gaps. Every entry names the bug, its root cause, and its guard. The rule for closing one: fix it, register it, add the guard, then prove the guard fails on the old broken state and passes on the fix. A guard that was never seen red is not evidence of anything.

The validator is 67 KB against the generator's 40 KB — roughly 1.7× more code checking the output than producing it. That ratio is the honest summary of where the engineering went.

Three defects worth naming

• A stale approval shipped a rewritten card. The publish gate checked that the string VERDICT: PASS appeared in the verification file. It did — from a round that had audited a completely different version. The gate was checking a proxy, not the thing. Fixed by binding the verdict to a SHA-256 of the card and recomputing it at build.
• A red validator did not block a deploy. The deploy chain piped the validator into tail; in a pipeline the shell reports the last command's exit status, so a failing gate returned zero. Fixed by making one script the only deploy path, with the validator run unpiped under set -euo pipefail.
• The same defective idiom existed in four places, and fixing the guard fixed only the guard. A non-global regex read the first link of each item and never the body prose. The copy inside the generator — the one the reader sees — told readers a card had 136 sources with 36 quoted, when 58 were. A validator that agrees with a page is worth nothing if both are wrong the same way.

Counting voices

• Counts, never percentages. A sentiment bar is a count of independent voices with a roster behind it. No card carries a percentage derived from a sample too small to support one.
• A source is not a voice. A source can be cited for a fact and excluded from n: eval houses, vendor docs, trade press, and relays of someone else's data are all citable and none of them are people with an opinion.
• Posture is classified toward the decision, not the tool. "This is impressive but I am not moving my production loop" is not positive sentiment about a switch. The question a card answers is whether to switch, so the axis has to be the switch.
• Under-claim, never over-claim. sample counts distinct sources and may never exceed the ledger. This is enforced mechanically — the advertised n must reconcile against the sentiment segments and the ledger.

Why most cards say LOW

Confidence is assigned to the verdict, not to the effort. It answers one question: if you follow this card and it turns out badly, how surprised should we be? Sample size, evidence independence, and whether the decisive measurement exists at all are what move it. Volume of research does not.

Of 7 published cards, 6 are LOW. The most heavily researched card on the site — 136 ledger rows, 6 verification artefacts on disk — is one of them, because the number that would settle its central question does not exist anywhere in any language, and no amount of sweeping conjures it.

What triggers a re-sweep

Nothing is on a timer, and this page will not pretend otherwise. There is no scheduler in this repository. Re-sweeps are event-triggered and operator-initiated, and the cost is stated plainly below in limitations.

The triggers that have actually fired:

• A dated claim goes false. A card published seven days before a model's weights shipped had four load-bearing claims expire on that date. It was re-swept on three axes — licence, self-hosting, access — and republished with the corrections on its face.
• A tooling defect invalidates a sweep's coverage. Discovering the mandated channel layer had been skipped forced a full re-sweep of one card, which then found seven further defects in the same pass.
• A verification fails. The loop is itself a re-sweep trigger — a defect list can send the builder back to the sources rather than to the prose.

Staleness is handled by dating everything and forbidding claims that decay silently. Relative time expressions — "recently", "this week", "five days old" — fail the validator, because they are true when written and false later with no edit. Quoted speech is exempt: a source's words are never altered to satisfy a checker.

What this system does not do

Stated because a reader auditing the method needs them, and because a system's real shape is in its gaps. Current as of the build date at the top of this page.

What it is built with

Reader-facing summary: the Method section. Every card's own coverage footer carries the gaps specific to that sweep. All 7 cards →