Skip to main content
HardDecisions

How we measured · methodology

How we measured decision model accuracy on ProofWriter

What the models were asked, which problems they saw, how answers were scored, what the numbers cannot tell you.

The dataset: ProofWriter

ProofWriter (Tafjord, Dalvi and Clark, 2021) is a multi-hop deductive reasoning dataset from AI2, published under CC BY 4.0. Each problem is a short theory of facts and rules in English and a statement to judge against it.

Version
V2020.12.3, official test splits: depth-0, depth-1, depth-2, depth-3, depth-5, NatLang
Gold answers
every selected label recomputed by hard_decisions.solver; zero disagreements. Our solver reproduces every published test label, on both tasks, with zero disagreements.
Pool
108,440 open-world test questions, from which the open-world sample is drawn
Synthetic
The theories are generated from templates, some reworded by people (the NatLang set). Models may have seen the dataset in training.

The sample: 1,800 problems per task, 3,600 in all

Running every model on the full test set is not practical, so the harness draws a sample with a fixed seed (0). Items are spread evenly over proof depth and the correct answer first, so deep proofs and the unknown answer are never starved the way a uniform draw would starve them; inside each cell, items are chosen to cover the other properties and spread across theories. A bigger sample with the same seed only adds items, so answers already collected stay valid.

Open world: true, false or unknown: items per depth and answer

DepthtruefalseunknownTotalAvailable in the pool
010010010030055,428
110110110030229,409
210110110130312,534
31011011013036,694
41011011013032,380
5101101872891,995

Closed world: true or false: items per depth and answer

DepthtruefalseTotalAvailable in the pool
015015030055,079
115015030028,872
215015030012,839
31501503006,922
41501503002,523
51501503002,101

ProofWriter publishes no separate depth-4 split; depth-4 items come from the splits built for deeper theories.

The two tasks

The same kinds of theory under two rules for what cannot be proved. Every model got exactly this question and these options, in this order.

Open world: true, false or unknown

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

  • true: The statement follows from the facts and rules.
  • false: The negation of the statement follows from the facts and rules.
  • unknown: Neither the statement nor its negation follows from the facts and rules.

Closed world: true or false

Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.

  • true: The statement holds under the closed-world assumption.
  • false: The statement does not hold under the closed-world assumption.

One request per problem, the same for every model

Each model answered each problem once, with the problem's text, the task's question and its options. Nothing was retried into a valid answer.

An answer that is not exactly one of the options is scored as wrong and kept in the record as given. Across every scored run so far there were 0 such answers.

Scoring

Accuracy
The share of problems answered correctly, with a 95% interval: the range the score would fall in if we drew the problems again (a percentile bootstrap, 1,000 draws, seed 0).
Floors
Chance (one over the number of options) and the best constant guess: always giving the most common correct answer in that slice.
Recall and macro-F1
Recall is the share of items with a given correct answer that the model got right; macro-F1 averages the F1 of each answer, so a model cannot score well by ignoring one.
Paired differences
Two models on exactly the same items: the difference in accuracy, with its own 95% interval. Narrower than comparing two separate intervals.
Breakdowns
Accuracy by 12 properties of the problems: proof depth, gold answer, theory kind, negation in the theory, negated statement, question strategy, paraphrased rules, theory's deepest proof, theory length, number of rules, number of facts, proof size. Only depth is controlled.
Repeatability
Each model answers every item a second time; we report agreement, Gwet's AC1 and Cohen's kappa. Details.

What the numbers cannot tell you

At about 300 items per depth, a depth's accuracy is uncertain by roughly ±5 points; at about 100 items per depth and answer, roughly ±10. Only large differences are resolvable at that level.

Run records

Every run the harness has recorded with its manifest: when, how many requests at once, on what machine and how busy it was. Earlier runs predate the manifests.

ModelTaskPurposeStartedFinishedAnsweredAt onceMachineLoad average at start
GLiDE, Oct 2Closed worldscored run2026-10-02 08:56 UTC2026-10-02 09:05 UTC1,800 of 1,8008Apple M1 Max, 32 GB14.63 / 21.12 / 17.17
GLiDE, Oct 2Open worldscored run2026-10-02 08:37 UTC2026-10-02 08:38 UTC20 of 204Apple M1 Max, 32 GB6.84 / 11.13 / 14.95
GLiDE, Oct 2Open worldscored run2026-10-02 08:39 UTC2026-10-02 08:56 UTC1,778 of 1,7808Apple M1 Max, 32 GB8.06 / 10.72 / 14.46
GLiDE, Oct 2Open worldscored run2026-10-02 09:05 UTC2026-10-02 09:06 UTC2 of 21Apple M1 Max, 32 GB7.04 / 9.34 / 12.49
GLiDE, Oct 3 rerunClosed worldscored run2026-10-03 18:41 UTC2026-10-03 18:45 UTC1,800 of 1,8008Apple M1 Max, 32 GB13.73 / 11.29 / 15.96
GLiDE, Oct 3 rerunOpen worldscored run2026-10-03 18:29 UTC2026-10-03 18:41 UTC1,800 of 1,8008Apple M1 Max, 32 GB6.28 / 9.22 / 22.45
GLiNER2.5-DecideClosed worldscored run2026-10-02 13:52 UTC2026-10-02 13:56 UTC1,800 of 1,8001Apple M1 Max, 32 GB5.01 / 6.68 / 7.71
GLiNER2.5-DecideOpen worldscored run2026-10-02 13:47 UTC2026-10-02 13:52 UTC1,800 of 1,8001Apple M1 Max, 32 GB6.02 / 7.28 / 8.31
gliner-2.5-decide-hostedClosed worldscored run2026-10-03 18:26 UTC2026-10-03 18:29 UTC1,800 of 1,8004Apple M1 Max, 32 GB9.44 / 11.63 / 26.16
gliner-2.5-decide-hostedOpen worldscored run2026-10-03 18:22 UTC2026-10-03 18:26 UTC1,800 of 1,8004Apple M1 Max, 32 GB6.02 / 14.97 / 31.65
gliner-2.5-decide-hosted-labels-onlyClosed worldscored run2026-10-03 18:33 UTC2026-10-03 18:36 UTC1,800 of 1,8004Apple M1 Max, 32 GB13.33 / 11.04 / 19.73
gliner-2.5-decide-hosted-labels-onlyOpen worldscored run2026-10-03 18:29 UTC2026-10-03 18:33 UTC1,800 of 1,8004Apple M1 Max, 32 GB5.77 / 8.72 / 21.65
GLiNER2.5-Decide, without instructionsClosed worldscored run2026-10-02 12:47 UTC2026-10-02 12:50 UTC1,800 of 1,8001Apple M1 Max, 32 GB4.89 / 6.81 / 8.62
GLiNER2.5-Decide, without instructionsOpen worldscored run2026-10-02 12:44 UTC2026-10-02 12:47 UTC1,800 of 1,8001Apple M1 Max, 32 GB5.87 / 7.8 / 9.29
Kev-4BClosed worldscored run2026-10-01 12:19 UTC2026-10-01 12:58 UTC1,800 of 1,8001Apple M1 Max, 32 GBnot recorded
Kev-9BClosed worldscored run2026-10-01 15:34 UTC2026-10-01 16:18 UTC1,800 of 1,8001Apple M1 Max, 32 GB12.73 / 13.39 / 11.19
Kev-9BOpen worldscored run2026-10-01 14:43 UTC2026-10-01 14:44 UTC20 of 201Apple M1 Max, 32 GB7.95 / 7.82 / 8.05
Kev-9BOpen worldscored run2026-10-01 14:44 UTC2026-10-01 15:34 UTC1,780 of 1,7801Apple M1 Max, 32 GB7.72 / 7.82 / 8.04
GLiNER2.5-DecideClosed worldrerun (timing, retest)2026-10-02 14:01 UTC2026-10-02 14:06 UTC1,800 of 1,8001Apple M1 Max, 32 GB8.3 / 6.69 / 7.01
GLiNER2.5-DecideOpen worldrerun (timing, retest)2026-10-02 13:57 UTC2026-10-02 14:01 UTC1,800 of 1,8001Apple M1 Max, 32 GB4.82 / 5.74 / 6.97
GLiNER2.5-Decide, without instructionsClosed worldrerun (timing, retest)2026-10-02 12:53 UTC2026-10-02 12:56 UTC1,800 of 1,8001Apple M1 Max, 32 GB7.65 / 6.92 / 7.97
GLiNER2.5-Decide, without instructionsOpen worldrerun (timing, retest)2026-10-02 12:50 UTC2026-10-02 12:53 UTC1,800 of 1,8001Apple M1 Max, 32 GB5.5 / 6.39 / 8.11
JevClosed worldrerun (timing, retest)2026-10-01 12:04 UTC2026-10-01 12:09 UTC1,800 of 1,8001Apple M1 Max, 32 GBnot recorded
JevOpen worldrerun (timing, retest)2026-10-01 11:58 UTC2026-10-01 12:04 UTC1,800 of 1,8001Apple M1 Max, 32 GBnot recorded
Kev-0.8BClosed worldrerun (timing, retest)2026-10-01 16:32 UTC2026-10-01 16:36 UTC1,800 of 1,8001Apple M1 Max, 32 GB5.2 / 5.56 / 6.33
Kev-0.8BOpen worldrerun (timing, retest)2026-10-01 16:28 UTC2026-10-01 16:32 UTC1,800 of 1,8001Apple M1 Max, 32 GB5.84 / 5.5 / 6.55
Kev-4BClosed worldrerun (timing, retest)2026-10-01 13:56 UTC2026-10-01 14:21 UTC1,800 of 1,8001Apple M1 Max, 32 GB11.14 / 8.15 / 7.02
Kev-4BOpen worldrerun (timing, retest)2026-10-01 13:29 UTC2026-10-01 13:56 UTC1,800 of 1,8001Apple M1 Max, 32 GB7.26 / 6.97 / 7.22
LayaClosed worldrerun (timing, retest)2026-10-01 14:28 UTC2026-10-01 14:34 UTC1,800 of 1,8001Apple M1 Max, 32 GB7.12 / 8.46 / 8.5
LayaOpen worldrerun (timing, retest)2026-10-01 14:21 UTC2026-10-01 14:28 UTC1,800 of 1,8001Apple M1 Max, 32 GB10.03 / 9.27 / 8.55
GPT-6 LunaClosed worldrerun (timing, retest)2026-10-01 15:57 UTC2026-10-01 16:27 UTC1,800 of 1,8001Apple M1 Max, 32 GB7.88 / 8.32 / 8.96
GPT-6 LunaOpen worldrerun (timing, retest)2026-10-01 15:30 UTC2026-10-01 15:57 UTC1,800 of 1,8001Apple M1 Max, 32 GB15.51 / 10.17 / 9.47

Check it yourself

Every score on this site is computed from the saved answers, offline, without calling a model. The harness rebuilds them and this site from the same files:

hd verify          # recompute every ProofWriter gold label with the independent solver
hd replay          # rescore every saved answer: studies/*.jsonl
hd report          # write RESULTS.md
cd site && npm run build    # rebuild this site from studies/, engines.yaml and the run manifests

Every number is also in one data file.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.