Self-published technical report (UPTR-202501) by Juho Ojala, CTO and
co-founder of Upright Oy,
8 April 2025. Source:
sources/impact-measurement/2026-09-10-upright-evaluating-llms.md.
The first of three linked Upright texts and the groundwork the other two build on — read it before Leena-2 and the alignment post.
The claim
Alignment work today is deontological — rules, duties, principles — and the report argues this “is expected to fall short as AI systems grow more autonomous”. The proposed replacement is consequentialist alignment, which requires a system that can “accurately evaluate and compare the scope of potential outcomes of any action”. Upright frames its own nine years of company-impact quantification as “consequentialist alignment of the private sector” — the same capability pointed at capital rather than at model behaviour.
CoCoCo — the measurement
Consequence Magnitude Comparison Consistency. A model performs thousands of pairwise impact-size comparisons (“is steel’s GHG burden bigger than its contribution to infrastructure?”). From those, an aggregate set of implied impact sizes is solved. The metric is then the share of individual pairwise answers that agree with the aggregate solved from all of them — formally, the share whose log-space ratio falls within 0.5 standard deviations of the implied scale.
CoCoCo measures internal consistency, not correctness. The report is explicit about this and treats it as forced: the questions have no answer key. This is the move that the whole Upright programme rests on — see Coherence as a truth substitute.
Design requirements it says the task exercises: consistent counterfactual reasoning, balanced causal attribution, a coherent value system, and robust handling of uncertainty.
Results (3,200 pairwise comparisons per model, 11 models)
| Model | Type | CoCoCo | Release |
|---|---|---|---|
| Claude 3.7 Sonnet (thinking) | Hybrid | 73.25 % | 2025-02-24 |
| Claude 3.7 Sonnet | Hybrid | 72.47 % | 2025-02-24 |
| OpenAI o1 | Reasoning | 72.28 % | 2024-09-12 |
| DeepSeek R1-1776 | Reasoning | 69.59 % | 2025-01-20 |
| OpenAI o3-mini | Reasoning | 66.88 % | 2025-01-31 |
| OpenAI GPT-4 | Classic | 66.22 % | 2023-03-14 |
| OpenAI GPT-4o | Classic | 66.03 % | 2024-05-13 |
| Gemini 1.5 Pro | Classic | 64.72 % | 2025-02-15 |
| Gemini 2.0 Flash | Classic | 61.03 % | 2025-01-30 |
| Mistral Large 2 | Classic | 59.84 % | 2024-07-24 |
| GPT-3.5 Turbo | Classic | 30.47 % | 2022-03-01 |
Scores rise with model recency; no clear cost-performance winner. On a weaker variant requiring only consistent ordering, the leading model reaches ~90 %.
The finding that matters most here: greenwashing in the training corpus
Three stated findings on factual alignment, as distinct from internal consistency:
- Internal consistency correlates with factual accuracy.
- LLMs “naturally overemphasize impacts at the production stage while underestimating impacts during end-use”.
- LLMs “often favor contrafactuals that cast companies’ impacts in a more positive light and minimize especially environmental impacts”.
Upright’s attributed cause, stated as a suspicion rather than a demonstration: the training data “include large volumes of corporate sustainability reports and similar content”, which carry a positive spin and focus on readily measurable internal operations.
⟨Inference: this describes a closed and pathological loop — corporate self-reporting enters the training corpus, the corpus trains the models, and the models are now sold to assess corporate sustainability, whose output becomes further reporting. Greenwashing entering the epistemic base and coming back out laundered as model knowledge. Upright names the bias but not the loop. It is the sharpest available argument in this material for why attested primary data is not interchangeable with inferred data, and it bears directly on PACT and ESG reporting in the financial sector.⟩
Status of the evidence
Self-published, not peer-reviewed, benchmark authored by the party being served by the result. The comparison against Upright’s own model is hedged in the text itself: “the Upright net impact model should not be regarded as ground truth”. A promised follow-up evaluation against 300M research papers and 30,000 manually annotated papers is announced but, as of 2026-09-10, not found published.
Related
- Upright Project — the entity
- Coherence as a truth substitute — the concept this report establishes
- Leena-2 (2026) — the model built on this finding
- Consequences as an alignment problem (2026)
- AI Advantage — CoCoCo as a precedent for inventing a domain metric
- CCTI — where that precedent is cited against the M4 “publishable either way” commitment ⟨link added by the 2026-09-14 lint; the citation was one-directional⟩