What you optimise for when the claim you need to make has no ground truth to check it against: not correctness, but the internal consistency of a large set of judgements with each other.
Named here after the practice, not after a literature — ⟨Background: the underlying idea is old (coherentist theories of justification in epistemology; paired-comparison scaling in psychometrics, where a consistent latent scale is solved from inconsistent pairwise judgements). Neither lineage is invoked by the source material.⟩ The page exists because Upright Oy has built a production system on it, which makes the concept concrete enough to argue with — and because it is the same problem shape as the EIC, reached independently.
The worked instance
Source: Leena-2 (Ojala, 2026), with the groundwork in the CoCoCo evaluation (Ojala, 2025).
The question “is steel’s greenhouse burden bigger than steel’s contribution to infrastructure?” has no answer key. It bundles counterfactual reasoning, causal attribution and value systems, and “no one has a general way to define impact, or its size, that holds across everything”. So Upright:
- Runs thousands of pairwise magnitude comparisons.
- Solves for the single set of implied sizes most coherent with all the pairwise answers at once.
- Uses those implied sizes as training targets, post-trains, and repeats.
- At inference, repeats the solve per product over thousands of fresh comparisons.
The metric (CoCoCo) is the share of individual comparisons that agree with the aggregate solved from all of them. It measures agreement, not truth, and the source says so.
Why this is a real category and not just a fudge
- It is checkable. Transitivity is a mathematical property. “These 800 sizes are the coherent solution to these 10,000 pairwise judgements” is a statement that can be verified without agreeing with any single judgement, and without seeing them.
- It transfers. Upright reports the post-trained model beating its own base model on the Allen Institute Fermi estimation benchmark despite never training on Fermi problems — weak evidence that coherence-training develops general estimation skill rather than a house style.
- It is falsifiable in one direction. Incoherence is proof of a defect; coherence is not proof of correctness. That asymmetry is the whole value and the whole limit.
The failure mode: a loop with no exogenous anchor
⟨Inference: this is the criticism to lead with. A system trained on the reconciled version of its own output has closed the loop on itself. It will converge — to a fixed point of its own priors, which is not the same as converging on the world. Upright states that real data and science anchor the process “where they exist” but never quantifies the anchor’s weight or coverage, so from outside the text there is no way to tell how much of the result is measurement and how much is elaborate self-consistency.
Coherence is therefore not a substitute for ground truth. It is a substitute for dense ground truth, and it is only sound to the extent that sparse exogenous measurements pin the scale down at enough points. What such a system needs is not more coherence but more anchors — which is precisely what an attested-data layer produces. See Cryptographic trust infrastructure and Verifiable carbon accounting.⟩
The second-order version of the same problem is documented in the same material: Upright’s 2025 evaluation found frontier LLMs systematically flatter companies and minimise environmental impacts, attributed to training corpora heavy with corporate sustainability reports. A coherence loop run on top of a biased corpus produces coherent bias — internally impeccable and systematically wrong. Coherence has no way to detect this from inside.
The general case, stated from outside ⟨added 2026-11-02⟩
The failure described above — an output-level property standing in for a process-level one nobody measured — is not specific to Upright or to coherence. Capraro, Marcus & Quattrociocchi (2026) state it as a general objection to AI evaluation: “similarity in outputs does not imply similarity in the processes that produce them”, with the worked instance being human and LLM credibility judgements that agree while the criteria driving them differ (Loru et al., PNAS 2025). See The output–process gap.
Two things that page adds to this one:
- It widens the target. If the argument holds, it does not only undercut CoCoCo; it undercuts the field’s headline benchmark — the Turing test — by the same step. Coherence is in no worse position than humanlikeness, which is a defence of Upright as much as a criticism.
- It supplies the replacement criteria this page did not have. The anchors argument above says what coherence is missing; Capraro et al. say what to measure instead — generalization under rephrasing, appropriate suspension of judgement, and adaptive reframing. The second is directly testable against CoCoCo: a coherence solve has no place to put “I don’t know”, because every pairwise comparison is forced.
⟨Inference: that last point is a real defect and not a framing one. The method requires an answer to every pair; abstention is unrepresentable in the solve. So a magnitude the model has no basis for enters the system with the same weight as one it does, and the coherence metric cannot distinguish them — which is the no-exogenous-anchor problem again, now visible in the data structure rather than in the training loop.⟩
Why it matters for SINE
It is the EIC’s problem shape, reached from the other side. An Epistemic Integrity Certificate carries a claim about an agent’s reasoning rather than about the world — the fourth evidence type in the CCTI stack. Upright has built a large commercial system whose entire output is exactly that kind of claim, and has no way to make it credible to a skeptical counterparty: the harness is closed, the technique is patent-pending, the whitepaper is gated. Buyers — including Big Four assurance teams, per Upright’s own “100 % pass rate” claim — currently take it on trust.
⟨Inference: that makes it an unusually clean candidate use case for WP-C. The property to be proved is mathematical rather than semantic; the inputs are genuinely proprietary rather than notionally so; and there is an existing, paying, skeptical audience. Compare the use cases named in the CCTI proposal, where the evaluation case was still unselected as of the 2026-06-17 draft.⟩
And it is a commitment device without an enforcer. Training for coherence binds a model’s future judgements to its past ones — the same structural function as a credible commitment, but internal and statistical, with no third party able to check it on any given inference. Ostrom’s second puzzle applies directly: “Without monitoring, there can be no credible commitment” (Ostrom’s commons design principles). The EIC is the missing monitoring.
Related
- Epistemic Integrity Certificate — the primitive that would make such a claim checkable
- Upright Project — the entity running this in production
- Leena-2 (2026) · Evaluating LLMs (2025)
- Cryptographic trust infrastructure — where the missing anchors would come from
- Ostrom’s commons design principles — the credible-commitment puzzle
- Decision-grade carbon-data criteria — the adjacent question of when inferred data is good enough to act on
- The output–process gap — the general form of this page’s central criticism