Breakdown · the controlled difficulty axis
Accuracy by proof depth: where multi-hop reasoning breaks
How many rule applications the shortest proof of the answer needs. Depth 0 is a fact stated in the text; depth 5 needs five chained inferences. This is the benchmark's difficulty axis, and the sample was built to hold it fixed: about 300 items at each depth.
25.7On the open-world task, Jev's accuracy ranges from 71.9% (depth 4) to 97.7% (depth 0), a spread of 25.7 points. Jev is the most accurate model overall; every model is below.
Proof depth, open world task
A statement is true if it follows from the facts and rules, false if its negation follows, and unknown if neither does. Each line is one model: its accuracy, with the 95% interval as a bar. The dashed line is chance (33%); the grey tick is the score of always giving the slice's most common answer.
- Jev
- GLiDE, Oct 3 rerun
- GLiDE, Oct 2
- GPT-6 Luna(reasoning off)
- Kev-9B
- Kev-4B
- Kev-0.8B
- GLiNER2.5-Decide
- GLiNER2.5-Decide, without instructions
- Laya
| Proof depth | Items | Jev | GLiDE, Oct 3 rerun | GLiDE, Oct 2 | GPT-6 Luna | Kev-9B | Kev-4B | Kev-0.8B | GLiNER2.5-Decide | GLiNER2.5-Decide, without instructions | Laya | Best constant |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| depth 0 | 300 | 97.7% 96.0–99.3 | 97.7% 95.7–99.0 | 97.7% 95.7–99.0 | 93.3% 90.3–96.0 | 89.3% 85.7–92.7 | 87.7% 84.0–91.3 | 70.3% 65.3–75.7 | 33.3% 27.7–38.7 | 45.0% 39.7–51.0 | 62.7% 57.0–68.3 | 33.3% |
| depth 1 | 302 | 87.7% 83.8–91.4 | 87.1% 83.4–90.7 | 87.7% 84.1–91.4 | 77.8% 73.2–82.8 | 73.8% 68.9–78.8 | 62.9% 57.3–68.2 | 54.0% 48.3–59.6 | 33.1% 27.8–38.7 | 44.4% 38.7–49.7 | 42.1% 36.8–47.7 | 33.4% |
| depth 2 | 303 | 84.8% 80.2–88.4 | 79.9% 75.2–84.2 | 84.2% 80.2–88.1 | 60.4% 54.5–66.0 | 58.4% 53.1–63.7 | 52.1% 46.5–57.8 | 46.9% 41.3–52.8 | 33.3% 27.7–38.6 | 44.2% 38.6–50.2 | 41.3% 36.0–46.9 | 33.3% |
| depth 3 | 303 | 79.9% 75.2–84.5 | 70.0% 64.4–74.9 | 78.5% 73.6–83.2 | 61.7% 56.8–67.7 | 49.5% 44.2–55.4 | 44.6% 39.3–49.8 | 48.8% 43.6–54.5 | 33.3% 27.7–38.6 | 44.2% 38.6–50.2 | 38.3% 33.0–44.6 | 33.3% |
| depth 4 | 303 | 71.9% 66.7–77.2 | 61.4% 55.8–66.3 | 73.9% 69.3–78.5 | 44.6% 38.6–49.5 | 38.6% 33.3–43.6 | 37.3% 31.7–42.2 | 50.5% 44.9–56.1 | 33.3% 27.7–38.6 | 50.5% 45.2–56.4 | 36.0% 31.0–41.6 | 33.3% |
| depth 5 | 289 | 81.0% 76.5–85.5 | 61.9% 56.1–67.5 | 75.1% 69.6–79.9 | 46.0% 40.1–51.6 | 41.2% 35.3–46.7 | 36.3% 31.1–41.9 | 50.9% 45.3–56.7 | 30.1% 24.9–35.3 | 44.6% 38.4–50.5 | 34.3% 29.1–39.8 | 34.9% |
Proof depth, closed world task
Anything that cannot be derived is false, so there is no unknown answer. Each line is one model: its accuracy, with the 95% interval as a bar. The dashed line is chance (50%); the grey tick is the score of always giving the slice's most common answer.
- Jev
- GLiDE, Oct 3 rerun
- GLiDE, Oct 2
- GPT-6 Luna(reasoning off)
- Kev-9B
- Kev-4B
- Kev-0.8B
- GLiNER2.5-Decide
- GLiNER2.5-Decide, without instructions
- Laya
| Proof depth | Items | Jev | GLiDE, Oct 3 rerun | GLiDE, Oct 2 | GPT-6 Luna | Kev-9B | Kev-4B | Kev-0.8B | GLiNER2.5-Decide | GLiNER2.5-Decide, without instructions | Laya | Best constant |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| depth 0 | 300 | 99.3% 98.3–100.0 | 99.0% 97.7–100.0 | 99.3% 98.3–100.0 | 98.0% 96.3–99.3 | 89.7% 86.0–93.0 | 85.7% 81.7–89.7 | 68.0% 62.7–73.3 | 49.7% 44.0–55.0 | 52.3% 46.7–58.0 | 72.0% 66.7–76.7 | 50.0% |
| depth 1 | 300 | 93.7% 91.0–96.3 | 88.0% 84.3–91.7 | 90.0% 86.7–93.3 | 76.7% 72.3–81.0 | 77.0% 72.7–81.7 | 68.3% 63.0–73.7 | 59.7% 54.0–65.7 | 50.7% 45.0–56.3 | 54.3% 49.0–59.7 | 51.0% 45.0–56.0 | 50.0% |
| depth 2 | 300 | 92.0% 88.7–94.7 | 68.3% 63.0–73.7 | 72.0% 67.0–77.3 | 65.3% 60.0–70.7 | 64.0% 59.0–69.3 | 54.3% 49.0–59.7 | 51.3% 45.7–57.0 | 49.0% 43.0–54.3 | 54.0% 48.3–60.0 | 54.7% 48.7–59.7 | 50.0% |
| depth 3 | 300 | 85.0% 81.3–89.3 | 61.3% 55.7–66.7 | 64.3% 59.0–69.7 | 57.0% 51.7–63.0 | 58.7% 53.0–64.3 | 56.0% 50.7–61.3 | 53.0% 47.7–58.7 | 51.0% 45.3–56.3 | 53.3% 47.3–59.0 | 51.3% 45.7–56.7 | 50.0% |
| depth 4 | 300 | 76.3% 71.7–81.0 | 52.3% 47.0–58.0 | 55.7% 50.7–61.0 | 48.3% 43.0–53.7 | 50.3% 44.7–55.7 | 45.0% 39.0–50.3 | 51.0% 45.7–57.0 | 51.7% 46.0–57.0 | 49.7% 44.3–55.3 | 58.7% 53.0–64.3 | 50.0% |
| depth 5 | 300 | 89.3% 85.7–92.3 | 44.0% 38.3–49.3 | 49.3% 43.7–55.0 | 44.7% 38.7–50.3 | 43.0% 37.3–48.7 | 43.3% 38.3–49.0 | 51.7% 46.0–57.3 | 50.0% 44.3–55.7 | 62.0% 56.3–67.3 | 46.0% 40.3–51.0 | 50.0% |
What a deeper proof looks like
Three real open-world problems, each proof drawn as a chain from a stated fact to the answer. They are the shortest problems at their depth, so easier than typical. What proof depth means, with a worked example at every depth.
Depth 0 · statement: Gary is not white.
- Gary is not white.stated in the text
Depth 2 · statement: Harry is not nice.
- Harry is big.If someone is big then they are quiet.Harry is quiet.
- Quiet people are not nice.Harry is not nice.
Depth 5 · statement: Harry is smart.
- Harry is kind.Harry is quiet.
- + Harry is kind.Harry is young.
- + Harry is kind.Harry is blue.
- Harry is green.
- Harry is smart.
The gap to Jev grows with depth
Paired differences on the same items: Jev minus each model, by depth. Positive means Jev was more accurate.
Jev minus GLiDE, Oct 3 rerun, open world
Jev minus GLiDE, Oct 3 rerun, closed world
Jev minus GLiDE, Oct 2, open world
Jev minus GLiDE, Oct 2, closed world
Jev minus GPT-6 Luna, open world
Jev minus GPT-6 Luna, closed world
Jev minus Kev-9B, open world
Jev minus Kev-9B, closed world
Jev minus Kev-4B, open world
Jev minus Kev-4B, closed world
Jev minus Kev-0.8B, open world
Jev minus Kev-0.8B, closed world
Jev minus GLiNER2.5-Decide, open world
Jev minus GLiNER2.5-Decide, closed world
Jev minus GLiNER2.5-Decide, without instructions, open world
Jev minus GLiNER2.5-Decide, without instructions, closed world
Jev minus Laya, open world
Jev minus Laya, closed world
The unknown answer at each depth
Recall on each correct answer, by depth, on the open-world task: the share of items with that answer each model got right. Counted from the saved answers of every fully scored model.
| Model | Correct answer | depth 0 | depth 1 | depth 2 | depth 3 | depth 4 | depth 5 |
|---|---|---|---|---|---|---|---|
| Jev | true | 98% | 89% | 86% | 86% | 80% | 93% |
| false | 99% | 92% | 87% | 80% | 79% | 92% | |
| unknown | 96% | 82% | 81% | 73% | 56% | 54% | |
| GLiDE, Oct 3 rerun | true | 98% | 85% | 78% | 61% | 45% | 55% |
| false | 98% | 79% | 64% | 57% | 49% | 46% | |
| unknown | 97% | 97% | 97% | 91% | 91% | 89% | |
| GLiDE, Oct 2 | true | 98% | 84% | 79% | 70% | 63% | 67% |
| false | 98% | 81% | 75% | 71% | 65% | 65% | |
| unknown | 97% | 98% | 98% | 94% | 93% | 95% | |
| GPT-6 Luna | true | 96% | 77% | 56% | 52% | 33% | 36% |
| false | 85% | 61% | 46% | 52% | 27% | 35% | |
| unknown | 99% | 95% | 79% | 80% | 74% | 71% | |
| Kev-9B | true | 97% | 78% | 61% | 53% | 30% | 36% |
| false | 94% | 73% | 44% | 40% | 19% | 23% | |
| unknown | 77% | 70% | 70% | 55% | 67% | 69% | |
| Kev-4B | true | 96% | 64% | 44% | 35% | 20% | 22% |
| false | 82% | 48% | 32% | 27% | 16% | 12% | |
| unknown | 85% | 77% | 81% | 72% | 76% | 82% | |
| Kev-0.8B | true | 91% | 61% | 48% | 48% | 50% | 52% |
| false | 92% | 88% | 87% | 89% | 92% | 89% | |
| unknown | 28% | 12% | 6% | 10% | 10% | 5% | |
| GLiNER2.5-Decide | true | 0% | 0% | 0% | 0% | 0% | 0% |
| false | 0% | 0% | 0% | 0% | 0% | 0% | |
| unknown | 100% | 100% | 100% | 100% | 100% | 100% | |
| GLiNER2.5-Decide, without instructions | true | 67% | 64% | 63% | 64% | 73% | 59% |
| false | 64% | 68% | 69% | 68% | 78% | 68% | |
| unknown | 4% | 0% | 0% | 0% | 0% | 0% | |
| Laya | true | 89% | 37% | 38% | 33% | 29% | 17% |
| false | 97% | 88% | 84% | 81% | 79% | 81% | |
| unknown | 2% | 1% | 2% | 1% | 0% | 0% |
About 100 items per cell, so each number is uncertain by roughly ±10 points.
Version v0.1.0, unreleased.
Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.
Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.