Skip to main content
HardDecisions

Breakdown · the controlled difficulty axis

Accuracy by proof depth: where multi-hop reasoning breaks

How many rule applications the shortest proof of the answer needs. Depth 0 is a fact stated in the text; depth 5 needs five chained inferences. This is the benchmark's difficulty axis, and the sample was built to hold it fixed: about 300 items at each depth.

25.7On the open-world task, Jev's accuracy ranges from 71.9% (depth 4) to 97.7% (depth 0), a spread of 25.7 points. Jev is the most accurate model overall; every model is below.

Proof depth, open world task

A statement is true if it follows from the facts and rules, false if its negation follows, and unknown if neither does. Each line is one model: its accuracy, with the 95% interval as a bar. The dashed line is chance (33%); the grey tick is the score of always giving the slice's most common answer.

  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
  • GPT-6 Luna(reasoning off)
  • Kev-9B
  • Kev-4B
  • Kev-0.8B
  • GLiNER2.5-Decide
  • GLiNER2.5-Decide, without instructions
  • Laya
depth 0300 items
depth 1302 items
depth 2303 items
depth 3303 items
depth 4303 items
depth 5289 items
Proof depth, Open world
Proof depthItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2GPT-6 LunaKev-9BKev-4BKev-0.8BGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsLayaBest constant
depth 030097.7% 96.0–99.397.7% 95.7–99.097.7% 95.7–99.093.3% 90.3–96.089.3% 85.7–92.787.7% 84.0–91.370.3% 65.3–75.733.3% 27.7–38.745.0% 39.7–51.062.7% 57.0–68.333.3%
depth 130287.7% 83.8–91.487.1% 83.4–90.787.7% 84.1–91.477.8% 73.2–82.873.8% 68.9–78.862.9% 57.3–68.254.0% 48.3–59.633.1% 27.8–38.744.4% 38.7–49.742.1% 36.8–47.733.4%
depth 230384.8% 80.2–88.479.9% 75.2–84.284.2% 80.2–88.160.4% 54.5–66.058.4% 53.1–63.752.1% 46.5–57.846.9% 41.3–52.833.3% 27.7–38.644.2% 38.6–50.241.3% 36.0–46.933.3%
depth 330379.9% 75.2–84.570.0% 64.4–74.978.5% 73.6–83.261.7% 56.8–67.749.5% 44.2–55.444.6% 39.3–49.848.8% 43.6–54.533.3% 27.7–38.644.2% 38.6–50.238.3% 33.0–44.633.3%
depth 430371.9% 66.7–77.261.4% 55.8–66.373.9% 69.3–78.544.6% 38.6–49.538.6% 33.3–43.637.3% 31.7–42.250.5% 44.9–56.133.3% 27.7–38.650.5% 45.2–56.436.0% 31.0–41.633.3%
depth 528981.0% 76.5–85.561.9% 56.1–67.575.1% 69.6–79.946.0% 40.1–51.641.2% 35.3–46.736.3% 31.1–41.950.9% 45.3–56.730.1% 24.9–35.344.6% 38.4–50.534.3% 29.1–39.834.9%

Proof depth, closed world task

Anything that cannot be derived is false, so there is no unknown answer. Each line is one model: its accuracy, with the 95% interval as a bar. The dashed line is chance (50%); the grey tick is the score of always giving the slice's most common answer.

  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
  • GPT-6 Luna(reasoning off)
  • Kev-9B
  • Kev-4B
  • Kev-0.8B
  • GLiNER2.5-Decide
  • GLiNER2.5-Decide, without instructions
  • Laya
depth 0300 items
depth 1300 items
depth 2300 items
depth 3300 items
depth 4300 items
depth 5300 items
Proof depth, Closed world
Proof depthItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2GPT-6 LunaKev-9BKev-4BKev-0.8BGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsLayaBest constant
depth 030099.3% 98.3–100.099.0% 97.7–100.099.3% 98.3–100.098.0% 96.3–99.389.7% 86.0–93.085.7% 81.7–89.768.0% 62.7–73.349.7% 44.0–55.052.3% 46.7–58.072.0% 66.7–76.750.0%
depth 130093.7% 91.0–96.388.0% 84.3–91.790.0% 86.7–93.376.7% 72.3–81.077.0% 72.7–81.768.3% 63.0–73.759.7% 54.0–65.750.7% 45.0–56.354.3% 49.0–59.751.0% 45.0–56.050.0%
depth 230092.0% 88.7–94.768.3% 63.0–73.772.0% 67.0–77.365.3% 60.0–70.764.0% 59.0–69.354.3% 49.0–59.751.3% 45.7–57.049.0% 43.0–54.354.0% 48.3–60.054.7% 48.7–59.750.0%
depth 330085.0% 81.3–89.361.3% 55.7–66.764.3% 59.0–69.757.0% 51.7–63.058.7% 53.0–64.356.0% 50.7–61.353.0% 47.7–58.751.0% 45.3–56.353.3% 47.3–59.051.3% 45.7–56.750.0%
depth 430076.3% 71.7–81.052.3% 47.0–58.055.7% 50.7–61.048.3% 43.0–53.750.3% 44.7–55.745.0% 39.0–50.351.0% 45.7–57.051.7% 46.0–57.049.7% 44.3–55.358.7% 53.0–64.350.0%
depth 530089.3% 85.7–92.344.0% 38.3–49.349.3% 43.7–55.044.7% 38.7–50.343.0% 37.3–48.743.3% 38.3–49.051.7% 46.0–57.350.0% 44.3–55.762.0% 56.3–67.346.0% 40.3–51.050.0%

What a deeper proof looks like

Three real open-world problems, each proof drawn as a chain from a stated fact to the answer. They are the shortest problems at their depth, so easier than typical. What proof depth means, with a worked example at every depth.

Depth 0 · statement: Gary is not white.

  1. Gary is not white.stated in the text

Depth 2 · statement: Harry is not nice.

  1. Harry is big.If someone is big then they are quiet.Harry is quiet.
  2. Quiet people are not nice.Harry is not nice.

Depth 5 · statement: Harry is smart.

  1. Harry is kind.Harry is quiet.
  2. + Harry is kind.Harry is young.
  3. + Harry is kind.Harry is blue.
  4. Harry is green.
  5. Harry is smart.

The gap to Jev grows with depth

Paired differences on the same items: Jev minus each model, by depth. Positive means Jev was more accurate.

Jev minus GLiDE, Oct 3 rerun, open world

depth 0300 items
0.0
depth 1302 items
+0.7
depth 2303 items
+5.0
depth 3303 items
+9.9
depth 4303 items
+10.6
depth 5289 items
+19.0

Jev minus GLiDE, Oct 3 rerun, closed world

depth 0300 items
+0.3
depth 1300 items
+5.7
depth 2300 items
+23.7
depth 3300 items
+23.7
depth 4300 items
+24.0
depth 5300 items
+45.3

Jev minus GLiDE, Oct 2, open world

depth 0300 items
0.0
depth 1302 items
0.0
depth 2303 items
+0.7
depth 3303 items
+1.3
depth 4303 items
−2.0
depth 5289 items
+5.9

Jev minus GLiDE, Oct 2, closed world

depth 0300 items
0.0
depth 1300 items
+3.7
depth 2300 items
+20.0
depth 3300 items
+20.7
depth 4300 items
+20.7
depth 5300 items
+40.0

Jev minus GPT-6 Luna, open world

depth 0300 items
+4.3
depth 1302 items
+9.9
depth 2303 items
+24.4
depth 3303 items
+18.2
depth 4303 items
+27.4
depth 5289 items
+34.9

Jev minus GPT-6 Luna, closed world

depth 0300 items
+1.3
depth 1300 items
+17.0
depth 2300 items
+26.7
depth 3300 items
+28.0
depth 4300 items
+28.0
depth 5300 items
+44.7

Jev minus Kev-9B, open world

depth 0300 items
+8.3
depth 1302 items
+13.9
depth 2303 items
+26.4
depth 3303 items
+30.4
depth 4303 items
+33.3
depth 5289 items
+39.8

Jev minus Kev-9B, closed world

depth 0300 items
+9.7
depth 1300 items
+16.7
depth 2300 items
+28.0
depth 3300 items
+26.3
depth 4300 items
+26.0
depth 5300 items
+46.3

Jev minus Kev-4B, open world

depth 0300 items
+10.0
depth 1302 items
+24.8
depth 2303 items
+32.7
depth 3303 items
+35.3
depth 4303 items
+34.7
depth 5289 items
+44.6

Jev minus Kev-4B, closed world

depth 0300 items
+13.7
depth 1300 items
+25.3
depth 2300 items
+37.7
depth 3300 items
+29.0
depth 4300 items
+31.3
depth 5300 items
+46.0

Jev minus Kev-0.8B, open world

depth 0300 items
+27.3
depth 1302 items
+33.8
depth 2303 items
+38.0
depth 3303 items
+31.0
depth 4303 items
+21.5
depth 5289 items
+30.1

Jev minus Kev-0.8B, closed world

depth 0300 items
+31.3
depth 1300 items
+34.0
depth 2300 items
+40.7
depth 3300 items
+32.0
depth 4300 items
+25.3
depth 5300 items
+37.7

Jev minus GLiNER2.5-Decide, open world

depth 0300 items
+64.3
depth 1302 items
+54.6
depth 2303 items
+51.5
depth 3303 items
+46.5
depth 4303 items
+38.6
depth 5289 items
+50.9

Jev minus GLiNER2.5-Decide, closed world

depth 0300 items
+49.7
depth 1300 items
+43.0
depth 2300 items
+43.0
depth 3300 items
+34.0
depth 4300 items
+24.7
depth 5300 items
+39.3

Jev minus GLiNER2.5-Decide, without instructions, open world

depth 0300 items
+52.7
depth 1302 items
+43.4
depth 2303 items
+40.6
depth 3303 items
+35.6
depth 4303 items
+21.5
depth 5289 items
+36.3

Jev minus GLiNER2.5-Decide, without instructions, closed world

depth 0300 items
+47.0
depth 1300 items
+39.3
depth 2300 items
+38.0
depth 3300 items
+31.7
depth 4300 items
+26.7
depth 5300 items
+27.3

Jev minus Laya, open world

depth 0300 items
+35.0
depth 1302 items
+45.7
depth 2303 items
+43.6
depth 3303 items
+41.6
depth 4303 items
+36.0
depth 5289 items
+46.7

Jev minus Laya, closed world

depth 0300 items
+27.3
depth 1300 items
+42.7
depth 2300 items
+37.3
depth 3300 items
+33.7
depth 4300 items
+17.7
depth 5300 items
+43.3

The unknown answer at each depth

Recall on each correct answer, by depth, on the open-world task: the share of items with that answer each model got right. Counted from the saved answers of every fully scored model.

ModelCorrect answerdepth 0depth 1depth 2depth 3depth 4depth 5
Jevtrue98%89%86%86%80%93%
false99%92%87%80%79%92%
unknown96%82%81%73%56%54%
GLiDE, Oct 3 reruntrue98%85%78%61%45%55%
false98%79%64%57%49%46%
unknown97%97%97%91%91%89%
GLiDE, Oct 2true98%84%79%70%63%67%
false98%81%75%71%65%65%
unknown97%98%98%94%93%95%
GPT-6 Lunatrue96%77%56%52%33%36%
false85%61%46%52%27%35%
unknown99%95%79%80%74%71%
Kev-9Btrue97%78%61%53%30%36%
false94%73%44%40%19%23%
unknown77%70%70%55%67%69%
Kev-4Btrue96%64%44%35%20%22%
false82%48%32%27%16%12%
unknown85%77%81%72%76%82%
Kev-0.8Btrue91%61%48%48%50%52%
false92%88%87%89%92%89%
unknown28%12%6%10%10%5%
GLiNER2.5-Decidetrue0%0%0%0%0%0%
false0%0%0%0%0%0%
unknown100%100%100%100%100%100%
GLiNER2.5-Decide, without instructionstrue67%64%63%64%73%59%
false64%68%69%68%78%68%
unknown4%0%0%0%0%0%
Layatrue89%37%38%33%29%17%
false97%88%84%81%79%81%
unknown2%1%2%1%0%0%

About 100 items per cell, so each number is uncertain by roughly ±10 points.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.