How we measured · methodology
How we measured decision model accuracy on ProofWriter
What the models were asked, which problems they saw, how answers were scored, what the numbers cannot tell you.
The dataset: ProofWriter
ProofWriter (Tafjord, Dalvi and Clark, 2021) is a multi-hop deductive reasoning dataset from AI2, published under CC BY 4.0. Each problem is a short theory of facts and rules in English and a statement to judge against it.
- Version
- V2020.12.3, official test splits: depth-0, depth-1, depth-2, depth-3, depth-5, NatLang
- Gold answers
- every selected label recomputed by hard_decisions.solver; zero disagreements. Our solver reproduces every published test label, on both tasks, with zero disagreements.
- Pool
- 108,440 open-world test questions, from which the open-world sample is drawn
- Synthetic
- The theories are generated from templates, some reworded by people (the NatLang set). Models may have seen the dataset in training.
The sample: 1,800 problems per task, 3,600 in all
Running every model on the full test set is not practical, so the harness draws a sample with a fixed seed (0). Items are spread evenly over proof depth and the correct answer first, so deep proofs and the unknown answer are never starved the way a uniform draw would starve them; inside each cell, items are chosen to cover the other properties and spread across theories. A bigger sample with the same seed only adds items, so answers already collected stay valid.
Open world: true, false or unknown: items per depth and answer
| Depth | true | false | unknown | Total | Available in the pool |
|---|---|---|---|---|---|
| 0 | 100 | 100 | 100 | 300 | 55,428 |
| 1 | 101 | 101 | 100 | 302 | 29,409 |
| 2 | 101 | 101 | 101 | 303 | 12,534 |
| 3 | 101 | 101 | 101 | 303 | 6,694 |
| 4 | 101 | 101 | 101 | 303 | 2,380 |
| 5 | 101 | 101 | 87 | 289 | 1,995 |
Closed world: true or false: items per depth and answer
| Depth | true | false | Total | Available in the pool |
|---|---|---|---|---|
| 0 | 150 | 150 | 300 | 55,079 |
| 1 | 150 | 150 | 300 | 28,872 |
| 2 | 150 | 150 | 300 | 12,839 |
| 3 | 150 | 150 | 300 | 6,922 |
| 4 | 150 | 150 | 300 | 2,523 |
| 5 | 150 | 150 | 300 | 2,101 |
ProofWriter publishes no separate depth-4 split; depth-4 items come from the splits built for deeper theories.
The two tasks
The same kinds of theory under two rules for what cannot be proved. Every model got exactly this question and these options, in this order.
Open world: true, false or unknown
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
- true: The statement follows from the facts and rules.
- false: The negation of the statement follows from the facts and rules.
- unknown: Neither the statement nor its negation follows from the facts and rules.
Closed world: true or false
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
- true: The statement holds under the closed-world assumption.
- false: The statement does not hold under the closed-world assumption.
One request per problem, the same for every model
Each model answered each problem once, with the problem's text, the task's question and its options. Nothing was retried into a valid answer.
- Jev. Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.
- GLiDE, Oct 3 rerun. The same GLiDE, run again on October 3 with exactly the same request and settings, because Fastino's CEO asked us to re-evaluate it: he said Fastino had problems during the launch. The October 2 run is kept and shown beside it. Fastino's API reports the model as "glide" both times, so the responses do not show what changed.
- GLiDE, Oct 2. GLiDE is a hosted decision model made by Fastino Labs, which describes it as a "thinking" decision model: it forms a probability over the options and spends more computation when the leading option is uncertain. Fastino serves it on the same System One request format as Jev, so it got exactly Jev's request: the text, the question and the options, one request per item. Its size and architecture are not published.
- GPT-6 Luna. GPT-6 Luna is a hosted large language model made by OpenAI. We used it as a classifier: one request per item through the Chat Completions API, with reasoning effort set to none (its lowest setting) and a strict JSON schema that allows only the task's options. Its size and architecture are not published.
- Kev-9B. Kev-9B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 9-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.
- Kev-4B. Kev-4B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 4-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.
- Kev-0.8B. Kev-0.8B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 0.8-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.
- GLiNER2.5-Decide. GLiNER2.5-Decide is an open decision model made by Fastino Labs: a 340M-parameter schema classifier on a DeBERTa-v3-large encoder, released under Apache 2.0. It answers in one pass; its model card says it "does not reason." We ran it on our own laptop with Fastino's gliner2 package, PyTorch on Apple's GPU, with everything every other model gets: the problem, then the question's instructions and each option with its description, in the message, and the options as its answer labels. On the same inputs, our copy gave Fastino's hosted model's answers.
- GLiNER2.5-Decide, without instructions. The same GLiNER2.5-Decide checkpoint asked in the bare form Fastino's hosted API documents: the problem text, which ends with the statement, and the answer options as labels, without the question's instructions or the option descriptions. It is shown beside the full form to show what those add.
- Laya. Laya is an open decision model made by Convai Innovations: a 421M-parameter ModernBERT-large encoder with decision heads. We ran it on our own laptop with the laya package, PyTorch on Apple's GPU.
An answer that is not exactly one of the options is scored as wrong and kept in the record as given. Across every scored run so far there were 0 such answers.
Scoring
- Accuracy
- The share of problems answered correctly, with a 95% interval: the range the score would fall in if we drew the problems again (a percentile bootstrap, 1,000 draws, seed 0).
- Floors
- Chance (one over the number of options) and the best constant guess: always giving the most common correct answer in that slice.
- Recall and macro-F1
- Recall is the share of items with a given correct answer that the model got right; macro-F1 averages the F1 of each answer, so a model cannot score well by ignoring one.
- Paired differences
- Two models on exactly the same items: the difference in accuracy, with its own 95% interval. Narrower than comparing two separate intervals.
- Breakdowns
- Accuracy by 12 properties of the problems: proof depth, gold answer, theory kind, negation in the theory, negated statement, question strategy, paraphrased rules, theory's deepest proof, theory length, number of rules, number of facts, proof size. Only depth is controlled.
- Repeatability
- Each model answers every item a second time; we report agreement, Gwet's AC1 and Cohen's kappa. Details.
What the numbers cannot tell you
- ProofWriter is synthetic. Its theories are generated from templates (some reworded by people), and models may have seen it in training. It measures one kind of reasoning, not decisions in your domain.
- One protocol for every model. Each model answered each item once, with the same question and the same options. GPT-6 Luna ran with reasoning off, its lowest setting, as a direct classifier; another setting would be another model.
- Latency is not like for like. Hosted models' times include the network and the vendor's servers. Open models ran on one laptop. Compare speed within a setting, not across settings.
- Only depth is controlled. The sample balances proof depth and the answer. Other breakdowns, such as negation, length or paraphrase, move with depth and with each other: they show where errors fall, not why.
- Costs are list prices. A hosted model's cost is its published price times the tokens its own API reported. Discounts, free tiers and invoices are not included; open models have no API cost, only the hardware they run on.
- Every number re-runs offline. Scores are computed from the saved answers; one command rebuilds them without calling a model. Results still being scored are marked as such, never filled in.
At about 300 items per depth, a depth's accuracy is uncertain by roughly ±5 points; at about 100 items per depth and answer, roughly ±10. Only large differences are resolvable at that level.
Run records
Every run the harness has recorded with its manifest: when, how many requests at once, on what machine and how busy it was. Earlier runs predate the manifests.
| Model | Task | Purpose | Started | Finished | Answered | At once | Machine | Load average at start |
|---|---|---|---|---|---|---|---|---|
| GLiDE, Oct 2 | Closed world | scored run | 2026-10-02 08:56 UTC | 2026-10-02 09:05 UTC | 1,800 of 1,800 | 8 | Apple M1 Max, 32 GB | 14.63 / 21.12 / 17.17 |
| GLiDE, Oct 2 | Open world | scored run | 2026-10-02 08:37 UTC | 2026-10-02 08:38 UTC | 20 of 20 | 4 | Apple M1 Max, 32 GB | 6.84 / 11.13 / 14.95 |
| GLiDE, Oct 2 | Open world | scored run | 2026-10-02 08:39 UTC | 2026-10-02 08:56 UTC | 1,778 of 1,780 | 8 | Apple M1 Max, 32 GB | 8.06 / 10.72 / 14.46 |
| GLiDE, Oct 2 | Open world | scored run | 2026-10-02 09:05 UTC | 2026-10-02 09:06 UTC | 2 of 2 | 1 | Apple M1 Max, 32 GB | 7.04 / 9.34 / 12.49 |
| GLiDE, Oct 3 rerun | Closed world | scored run | 2026-10-03 18:41 UTC | 2026-10-03 18:45 UTC | 1,800 of 1,800 | 8 | Apple M1 Max, 32 GB | 13.73 / 11.29 / 15.96 |
| GLiDE, Oct 3 rerun | Open world | scored run | 2026-10-03 18:29 UTC | 2026-10-03 18:41 UTC | 1,800 of 1,800 | 8 | Apple M1 Max, 32 GB | 6.28 / 9.22 / 22.45 |
| GLiNER2.5-Decide | Closed world | scored run | 2026-10-02 13:52 UTC | 2026-10-02 13:56 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 5.01 / 6.68 / 7.71 |
| GLiNER2.5-Decide | Open world | scored run | 2026-10-02 13:47 UTC | 2026-10-02 13:52 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 6.02 / 7.28 / 8.31 |
| gliner-2.5-decide-hosted | Closed world | scored run | 2026-10-03 18:26 UTC | 2026-10-03 18:29 UTC | 1,800 of 1,800 | 4 | Apple M1 Max, 32 GB | 9.44 / 11.63 / 26.16 |
| gliner-2.5-decide-hosted | Open world | scored run | 2026-10-03 18:22 UTC | 2026-10-03 18:26 UTC | 1,800 of 1,800 | 4 | Apple M1 Max, 32 GB | 6.02 / 14.97 / 31.65 |
| gliner-2.5-decide-hosted-labels-only | Closed world | scored run | 2026-10-03 18:33 UTC | 2026-10-03 18:36 UTC | 1,800 of 1,800 | 4 | Apple M1 Max, 32 GB | 13.33 / 11.04 / 19.73 |
| gliner-2.5-decide-hosted-labels-only | Open world | scored run | 2026-10-03 18:29 UTC | 2026-10-03 18:33 UTC | 1,800 of 1,800 | 4 | Apple M1 Max, 32 GB | 5.77 / 8.72 / 21.65 |
| GLiNER2.5-Decide, without instructions | Closed world | scored run | 2026-10-02 12:47 UTC | 2026-10-02 12:50 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 4.89 / 6.81 / 8.62 |
| GLiNER2.5-Decide, without instructions | Open world | scored run | 2026-10-02 12:44 UTC | 2026-10-02 12:47 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 5.87 / 7.8 / 9.29 |
| Kev-4B | Closed world | scored run | 2026-10-01 12:19 UTC | 2026-10-01 12:58 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | not recorded |
| Kev-9B | Closed world | scored run | 2026-10-01 15:34 UTC | 2026-10-01 16:18 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 12.73 / 13.39 / 11.19 |
| Kev-9B | Open world | scored run | 2026-10-01 14:43 UTC | 2026-10-01 14:44 UTC | 20 of 20 | 1 | Apple M1 Max, 32 GB | 7.95 / 7.82 / 8.05 |
| Kev-9B | Open world | scored run | 2026-10-01 14:44 UTC | 2026-10-01 15:34 UTC | 1,780 of 1,780 | 1 | Apple M1 Max, 32 GB | 7.72 / 7.82 / 8.04 |
| GLiNER2.5-Decide | Closed world | rerun (timing, retest) | 2026-10-02 14:01 UTC | 2026-10-02 14:06 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 8.3 / 6.69 / 7.01 |
| GLiNER2.5-Decide | Open world | rerun (timing, retest) | 2026-10-02 13:57 UTC | 2026-10-02 14:01 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 4.82 / 5.74 / 6.97 |
| GLiNER2.5-Decide, without instructions | Closed world | rerun (timing, retest) | 2026-10-02 12:53 UTC | 2026-10-02 12:56 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 7.65 / 6.92 / 7.97 |
| GLiNER2.5-Decide, without instructions | Open world | rerun (timing, retest) | 2026-10-02 12:50 UTC | 2026-10-02 12:53 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 5.5 / 6.39 / 8.11 |
| Jev | Closed world | rerun (timing, retest) | 2026-10-01 12:04 UTC | 2026-10-01 12:09 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | not recorded |
| Jev | Open world | rerun (timing, retest) | 2026-10-01 11:58 UTC | 2026-10-01 12:04 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | not recorded |
| Kev-0.8B | Closed world | rerun (timing, retest) | 2026-10-01 16:32 UTC | 2026-10-01 16:36 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 5.2 / 5.56 / 6.33 |
| Kev-0.8B | Open world | rerun (timing, retest) | 2026-10-01 16:28 UTC | 2026-10-01 16:32 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 5.84 / 5.5 / 6.55 |
| Kev-4B | Closed world | rerun (timing, retest) | 2026-10-01 13:56 UTC | 2026-10-01 14:21 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 11.14 / 8.15 / 7.02 |
| Kev-4B | Open world | rerun (timing, retest) | 2026-10-01 13:29 UTC | 2026-10-01 13:56 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 7.26 / 6.97 / 7.22 |
| Laya | Closed world | rerun (timing, retest) | 2026-10-01 14:28 UTC | 2026-10-01 14:34 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 7.12 / 8.46 / 8.5 |
| Laya | Open world | rerun (timing, retest) | 2026-10-01 14:21 UTC | 2026-10-01 14:28 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 10.03 / 9.27 / 8.55 |
| GPT-6 Luna | Closed world | rerun (timing, retest) | 2026-10-01 15:57 UTC | 2026-10-01 16:27 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 7.88 / 8.32 / 8.96 |
| GPT-6 Luna | Open world | rerun (timing, retest) | 2026-10-01 15:30 UTC | 2026-10-01 15:57 UTC | 1,800 of 1,800 | 1 | Apple M1 Max, 32 GB | 15.51 / 10.17 / 9.47 |
Check it yourself
Every score on this site is computed from the saved answers, offline, without calling a model. The harness rebuilds them and this site from the same files:
hd verify # recompute every ProofWriter gold label with the independent solver
hd replay # rescore every saved answer: studies/*.jsonl
hd report # write RESULTS.md
cd site && npm run build # rebuild this site from studies/, engines.yaml and the run manifestsEvery number is also in one data file.
Version v0.1.0, unreleased.
Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.
Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.