Breakdowns · ProofWriter results
Where decision models fail: every breakdown
Accuracy broken down by every property ProofWriter records about a problem. Proof depth is the controlled difficulty axis; the others are descriptive.
At the deep end: accuracy at proof depth 5
- Jev76.5 to 85.5 (95%), open world81.0%
- GLiDE, Oct 3 rerun56.1 to 67.5 (95%), open world61.9%
- GLiDE, Oct 269.6 to 79.9 (95%), open world75.1%
- GPT-6 Luna40.1 to 51.6 (95%), open world46.0%
- Kev-9B35.3 to 46.7 (95%), open world41.2%
- Kev-4B31.1 to 41.9 (95%), open world36.3%
- Kev-0.8B45.3 to 56.7 (95%), open world50.9%
- GLiNER2.5-Decide24.9 to 35.3 (95%), open world30.1%
- GLiNER2.5-Decide, without instructions38.4 to 50.5 (95%), open world44.6%
- Laya29.1 to 39.8 (95%), open world34.3%
Fully scored models only. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.
Every breakdown, with each model's weakest slice
Open-world task; weakest and strongest slices of at least 50 items.
| Breakdown | Jev | GLiDE, Oct 3 rerun | GLiDE, Oct 2 | GPT-6 Luna | Kev-9B | Kev-4B | Kev-0.8B | GLiNER2.5-Decide | GLiNER2.5-Decide, without instructions | Laya |
|---|---|---|---|---|---|---|---|---|---|---|
| Proof depth | depth 4 71.9% · depth 0 97.7% | depth 4 61.4% · depth 0 97.7% | depth 4 73.9% · depth 0 97.7% | depth 4 44.6% · depth 0 93.3% | depth 4 38.6% · depth 0 89.3% | depth 5 36.3% · depth 0 87.7% | depth 2 46.9% · depth 0 70.3% | depth 5 30.1% · depth 0 33.3% | depth 2 44.2% · depth 4 50.5% | depth 5 34.3% · depth 0 62.7% |
| Gold answer | unknown 74.2% · true 88.8% | false 65.5% · unknown 93.7% | false 76.0% · unknown 95.9% | false 50.9% · unknown 83.4% | false 48.6% · unknown 68.1% | false 35.9% · unknown 78.8% | unknown 11.9% · false 89.6% | false 0.0% · unknown 100.0% | unknown 0.7% · false 69.4% | unknown 1.0% · false 85.1% |
| Theory kind | attribute 82.8% · relation 85.3% | relation 73.3% · attribute 78.6% | relation 80.0% · attribute 84.9% | attribute 62.5% · relation 66.3% | relation 57.5% · attribute 59.3% | relation 52.4% · attribute 54.4% | relation 53.2% · attribute 53.8% | relation 31.3% · attribute 33.8% | relation 43.3% · attribute 47.0% | attribute 41.2% · relation 44.3% |
| Negation in the theory | without negation 80.3% · with negation 87.5% | without negation 73.6% · with negation 79.3% | without negation 80.1% · with negation 85.8% | without negation 61.2% · with negation 67.1% | without negation 56.8% · with negation 60.4% | without negation 49.9% · with negation 57.3% | without negation 50.6% · with negation 56.6% | with negation 30.0% · without negation 35.5% | with negation 31.0% · without negation 59.4% | without negation 38.8% · with negation 46.2% |
| Negated statement | negated statement 78.6% · plain statement 89.1% | plain statement 75.0% · negated statement 77.8% | plain statement 81.6% · negated statement 84.1% | plain statement 62.3% · negated statement 65.8% | negated statement 57.4% · plain statement 59.7% | negated statement 53.5% · plain statement 53.6% | plain statement 48.5% · negated statement 58.6% | plain statement 32.4% · negated statement 33.2% | plain statement 43.4% · negated statement 47.6% | plain statement 38.7% · negated statement 46.2% |
| Question strategy | inv-rconc 50.2% · random 98.0% | inv-proof 65.5% · inv-random 98.0% | inv-proof 76.0% · inv-random 98.0% | inv-proof 50.9% · inv-random 100.0% | inv-proof 48.6% · random 78.0% | inv-proof 35.9% · random 88.0% | inv-rconc 0.8% · inv-proof 89.6% | inv-proof 0.0% · inv-random 100.0% | inv-rconc 0.0% · inv-proof 69.4% | inv-rconc 0.8% · inv-proof 85.1% |
| Paraphrased rules | paraphrased (NatLang) 78.1% · templated 86.1% | paraphrased (NatLang) 68.9% · templated 79.3% | paraphrased (NatLang) 74.9% · templated 86.0% | paraphrased (NatLang) 54.8% · templated 67.6% | paraphrased (NatLang) 55.8% · templated 59.6% | paraphrased (NatLang) 47.6% · templated 55.9% | paraphrased (NatLang) 53.0% · templated 53.8% | paraphrased (NatLang) 28.3% · templated 34.5% | templated 38.2% · paraphrased (NatLang) 64.3% | templated 41.7% · paraphrased (NatLang) 44.4% |
| Theory's deepest proof | 4 67.9% · 0 90.1% | 4 56.8% · 0 98.6% | 4 66.7% · 0 98.6% | 4 40.7% · 0 87.3% | 4 42.6% · 0 87.3% | 5 38.0% · 0 85.9% | 0 42.3% · 5 58.0% | 5 23.5% · 0 60.6% | 0 18.3% · 4 61.7% | 0 39.4% · 1 48.3% |
| Theory length | 80-109 79.8% · 0-49 93.1% | 110+ 65.2% · 0-49 95.1% | 110+ 74.3% · 0-49 95.5% | 110+ 52.6% · 0-49 91.5% | 110+ 47.9% · 0-49 77.7% | 110+ 41.7% · 0-49 75.7% | 110+ 52.6% · 0-49 56.3% | 110+ 26.1% · 0-49 39.7% | 50-79 36.8% · 110+ 53.7% | 80-109 40.3% · 0-49 47.8% |
| Number of rules | 8+ 80.0% · 0-2 92.2% | 3-5 68.5% · 0-2 93.1% | 3-5 75.1% · 0-2 94.1% | 3-5 59.1% · 0-2 90.2% | 3-5 51.6% · 0-2 78.9% | 3-5 45.9% · 0-2 78.9% | 8+ 49.8% · 0-2 58.8% | 3-5 24.0% · 8+ 40.7% | 8+ 32.0% · 3-5 61.2% | 8+ 35.7% · 0-2 53.9% |
| Number of facts | 8-12 80.3% · 0-3 90.4% | 13+ 62.9% · 0-3 88.9% | 13+ 74.2% · 0-3 92.2% | 13+ 56.6% · 0-3 81.4% | 13+ 46.5% · 0-3 71.9% | 13+ 40.3% · 0-3 69.8% | 0-3 52.4% · 13+ 56.6% | 13+ 23.9% · 0-3 42.8% | 0-3 32.9% · 8-12 53.6% | 4-7 41.6% · 13+ 44.0% |
| Proof size | 0-1 80.4% · 2 98.1% | 5+ 50.4% · 0-1 94.8% | 5+ 65.6% · 0-1 96.5% | 5+ 33.6% · 0-1 85.2% | 5+ 31.8% · 2 84.1% | 5+ 20.3% · 0-1 81.4% | 0-1 32.0% · 2 77.6% | 2 0.0% · 0-1 74.7% | 0-1 17.1% · 5+ 73.3% | 0-1 24.3% · 2 63.6% |
How many rule applications the shortest proof of the answer needs. Depth 0 is a fact stated in the text; depth 5 needs five chained inferences. This is the benchmark's difficulty axis, and the sample was built to hold it fixed: about 300 items at each depth.
The correct answer. On the open-world task a third of the items are unknown: neither the statement nor its negation follows. That class is where several models fail.
Two kinds of "not": in the theory's facts and rules, and in the statement to judge.
Whether the theory comes from ProofWriter's NatLang set, where people reworded the templated sentences, or from the templated sets.
Every other property ProofWriter records about a problem: how long the theory is, how many rules and facts it has, how big the proof is, how deep the theory goes, whether it states attributes or relations, and how the question was generated. None of them was controlled when the problems were sampled.
Version v0.1.0, unreleased.
Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.
Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.