Similar outputs do not imply similar processes. An evaluation that scores what a system produces therefore leaves open how it got there, and whether the competence it appears to demonstrate extends anywhere else.
Named here after the failure, not after a literature. The page exists because the wiki now holds the same structure three times — once as a commercial product, once as a security finding, once as a critique of the field’s headline benchmark — and because it is the premise underneath the EIC: if outputs identified processes, there would be nothing for a certificate about reasoning to carry.
The general statement
Source: Capraro, Marcus & Quattrociocchi (2026), arguing against human likeness as a benchmark for AI.
“Similarity in outputs does not imply similarity in the processes that produce them. … Therefore, a humanlike answer on a particular test leaves open how the system arrived at that answer and whether the competence it appears to demonstrate extends to other problems.”
Their worked instance is Loru et al. (PNAS 2025): human and LLM judgements of news-source credibility agree to some degree while the observable criteria driving them differ. The agreement is real and the inference from it is invalid.
Two structural reasons the gap is expected rather than accidental:
- Language is not thought. Severe aphasia can leave arithmetic and causal reasoning intact (Mahowald et al. 2024; Kean et al. 2026) — so fluency is not evidence of the faculty it is being read as evidence of.
- The objective does not reward the process. Autoregressive pretraining rewards next-token prediction. Which reasoning capacities emerge from that “must … be established empirically” — they are not entailed by the training signal.
What closes the gap, according to the source
Not better imitation tests. Three capacities that are properties of the process, observable only by varying the task rather than by scoring one:
- Generalization — does competence survive unfamiliar instances and rephrasings? The counter-evidence is that it often does not, and not monotonically: Zhou et al. (Nature 2024) report that larger and more instructable models become less reliable, with errors on easy items persisting.
- Appropriate suspension of judgement — does the system withhold when the evidence is insufficient, and does it know where its competence ends?
- Adaptive reframing — does it revise what the problem is when evidence, causal structure or social context change, without being told to reframe? ⟨Note: proposed, not demonstrated — the supporting paper is listed as forthcoming.⟩
Three instances in this wiki
| Instance | Output-level property measured | Process-level property assumed | Who says so |
|---|---|---|---|
| Turing-test claims | Judges cannot tell the text from a human’s | Reasoning, generalization, adaptation | Capraro et al. (2026) — outside critics |
| CoCoCo / Leena-2 | Pairwise magnitude judgements are mutually consistent | That the implied magnitudes track real impact | Upright states the limit itself; see Coherence as a truth substitute |
| Prompt-injection hardening | The model refuses when asked to misbehave | That it is robust to being told a false fact | Ojala (2026) — 45 breaks against 25 models |
⟨Inference: the three are the same error in three registers, and the third is the one that shows the stakes. Refusal behaviour is an output; the property anyone cares about is whether the model’s beliefs can be moved by planted evidence. Hardening the output left the process untouched, and the attack surface turned out to be facts, not instructions — which is why that source is the wiki’s best outside argument for WP-A’s provenance layer. The same move applies upward: if an output-level pass cannot certify a process, then a claim about a process has to be carried by something other than the output, and that something is either a trusted third party or a proof.⟩
The honest limit
Processes are not directly observable either. Nothing in the source offers a way to inspect an LLM’s reasoning; the three replacement capacities are still measured behaviourally, just over a wider and more adversarial sample of tasks. The gap is narrowed by varying the input, not by opening the box.
⟨Inference: that leaves the instrument question open in exactly the place SINE works. The source’s recommendation to institutions — procure on “evidence of benefit”, compare assisted with unaided performance — assumes the buyer can run the experiment. A procuring institution evaluating a closed commercial system cannot: it sees outputs, the vendor holds the harness, and the vendor’s own measurements are authored by the party served by the result (the explicit criticism the wiki already makes of Upright’s CoCoCo, see AI Advantage). A checkable claim about the process, issued by the system and verifiable without exposing it, is the missing instrument — which is the EIC restated as a procurement problem rather than an agent-negotiation one.⟩
Related
- Capraro, Marcus & Quattrociocchi (2026) — the source that states the gap in general form
- Coherence as a truth substitute — the commercial special case, and the “coherent bias” failure mode
- Epistemic Integrity Certificate — the primitive that would carry a process claim
- AI Advantage — CCTI’s measurement design, and the self-authored-metric warning
- AI’s shortcomings in understanding consequences (Ojala, 2026) — the security instance
- Cryptographic trust infrastructure — where a process claim would be made checkable