SINE Strategy Wiki

AI Advantage (as a measurable claim)

5 min read

The empirical core of the CCTI project (WP-D, SINE / Cambridge). Source: sources/proposals/2026-06-17-aria-ccti-proposal-draft-2.md.

The comparison

Three arms, on at least one vertically integrated industry use case (aspiration: two contrasting ones):

  • AI-with-CCTI — agents that can produce and verify attested evidence and EICs
  • AI-without-CCTI — agents without the trust infrastructure
  • Human negotiation — the incumbent baseline

Reported with effect sizes and confidence intervals at M4 (30/6/2027), and committed to being publishable either way; a No-Go triggers a “headline-claim pivot” rather than a quiet retreat.

Cooperation-unlocking decision

The qualitative counterpart, due at M3 (31/3/2027): a demonstrator that exhibits a decision which would not have been reachable without the trust infrastructure. This is the existence proof; the effect-size study is the magnitude claim.

Why it is unusual

⟨Inference: three features are worth noticing, because they are the parts another organisation could copy.

  1. The measurement target is coordination outcomes, not model capability. Most AI evaluation asks how well a model performs a task; this asks whether an infrastructure changes which agreements are reachable at all.
  2. The human arm makes it a claim about institutions, not about AI: the interesting result is a comparison against how the transaction is done today.
  3. Pre-committing to publish a null result, with the fallback named in advance, is a credibility device — and one SINE could reuse in its own sustainability-data work, where “did the data infrastructure actually change any decision?” is the question nobody answers.⟩

Outcome 5 of the project claims exactly this: methodological precedent for empirical AI-Advantage measurement, transferable to other creators and to Arena challenges.

Follow-on

WP-E (conditional, from Jul 2027) takes the observed patterns and asks which institutional equivalent — insurance, arbitration, or reputation — emerges among agents in the AI-with-CCTI setting. The final publication list also names a “Coasean qualitative-framing analysis”. ⟨Background: the Coasean reference is to transaction-cost economics — institutions exist because bargaining is costly; if verifiable evidence collapses those costs, the boundary between firm and market moves. Not elaborated in the source.⟩

A methodological precedent for the M4 gate

Added 2026-09-10. M4 (30/6/2027) commits CCTI to publishing the AI-advantage result “either way”. A small external precedent for what that looks like in practice: Upright, needing to measure something no benchmark covered, defined its own metric (CoCoCo — consequence magnitude comparison consistency), stated plainly that it measures internal consistency and not correctness, published a table unflattering to most frontier labs, and reported an unplanned generalisation to an unrelated benchmark. See Evaluating LLMs (Ojala, 2025) and Leena-2 (2026).

⟨Inference: two transferable lessons and one warning. Inventing a domain-specific metric is legitimate when none exists, provided its limits are stated in the same breath — CCTI will be in exactly that position. The warning is the other side of the same coin: a metric authored by the party served by the result is discounted by readers, and Upright’s is self-published, not peer-reviewed, run on a closed harness. If the AI-advantage measure is to survive contact with ARIA, its definition and harness need to be public before the result is.⟩

An external prescription of the same design ⟨added 2026-11-02⟩

Capraro, Marcus & Quattrociocchi (2026), arguing that AI evaluation should abandon human likeness, close with a recommendation to institutions that is WP-D’s structure reached independently and from a different discipline: evaluations should “compare assisted with unaided performance”, and procurement and renewal should be guided by evidence of benefit rather than by capability demonstrations. Three authors with no knowledge of CCTI, writing for a psychology audience, prescribe the arm structure WP-D already committed to.

Useful less as validation than as a source of secondary measures. WP-D currently commits to effect sizes and confidence intervals on a coordination outcome. The paper names three quantities that would sit underneath that at no extra arm:

  1. Confidence calibration — and specifically, uncertainty “communicated in ways validated against actual performance”.
  2. Willingness to suspend judgement. Their own experiments found that access to deliberately incorrect AI advice reduced participants’ willingness to say “I don’t know”, lowered accuracy and raised confidence, with accuracy incentivized (Marcoccia, Quattrociocchi & Capraro 2026).
  3. Users’ ability to detect and reject incorrect advice.

⟨Inference: the third is the one worth lifting. A trust-infrastructure claim predicts it directly — if attested evidence and EICs do anything, the AI-with-CCTI arm should reject bad claims more often, and that is an observable count rather than an inferred effect. It also gives the M4 gate a result that survives a null on the headline measure: “no detectable difference in agreement rate, but the CCTI arm rejected 3× more unsupported claims” is a publishable finding and a Go-adjacent one. Worth raising against the M4 publish-either-way commitment and the named headline-claim pivot.⟩

The funder-level recommendation in the same paper — require funded projects to assess how benefits and costs are distributed, including effects on access to expertise, worker autonomy and the concentration of economic gains — is the register ARIA-facing language already uses, and is a free citation for a proposal that wants to show it is measuring distribution and not just capability.

Graph View

Backlinks