Skip to main content
HardDecisions

Benchmark for decision models and LLM classifiers · ProofWriter

Decision model benchmark: how accuracy falls as reasoning gets deeper

Fast decision models and LLM classifiers answer a question about a text in one step. We gave them 3,600 multi-hop reasoning problems from ProofWriter, from statements read straight off the page to ones that need five chained inferences, and measured where each one breaks.

Accuracy by proof depth, both tasksEach line is one model; each point is about 300 problems that need that many inference steps, with its 95% interval. The dashed line is chance: one in three on the open-world task, one in two on the closed-world task. Every number, both tasks.

Depth 0: the answer is written in the text. Depth 5: five rules must be chained to reach it. What proof depth means, with an example at every depth.

  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
  • GPT-6 Luna(reasoning off)
  • Kev-9B
  • Kev-4B
  • Kev-0.8B
  • GLiNER2.5-Decide
  • GLiNER2.5-Decide, without instructions
  • Laya

Open world: true, false or unknown

0%25%50%75%100%012345proof depth (inference steps)chance 33%

Closed world: true or false

0%25%50%75%100%012345proof depth (inference steps)chance 50%

83.8%Jev answered 83.8% of open-world problems correctly and 89.3% of closed-world ones, the highest of the models tested. Its lead over GPT-6 Luna (reasoning off) is +4.3 points at depth 0 and +34.9 points at depth 5 on the same problems (+1.3 and +44.7 on the closed-world task). The closest is GLiDE, Oct 2: level with it on the open-world task (+0.9 points, within noise), but 17.5 points behind on the closed-world one.

10 models so far: hosted and open decision models, and a general LLM used as a classifier. Every result below links to the problems and the numbers behind it.

What proof depth means

Every problem is a short text of facts and if-then rules, plus a statement to judge as true, false or unknown. Proof depth counts how many rules you have to chain together to reach the answer. Two real problems from the benchmark:

Depth 0: the answer is written down

  • fact Gary is not white.
  • rule If something is cold and big then it is not young.

Statement: Gary is not white. Answer: true.

The text states it outright, so no rule is needed. Across all open-world problems at depth 0, the models we tested score from 33.3% to 97.7%.

Depth 5: 5 rules in a chain

  • fact Anne is blue.
  • fact Anne is kind.
  • fact Anne is smart.
  • fact Anne is young.
  • fact Charlie is blue.
  • fact Charlie is green.
  • fact Charlie is kind.
  • fact Charlie is red.
  • fact Dave is quiet.
  • fact Harry is kind.
  • rule All kind, quiet people are young.
  • rule All kind people are quiet.
  • rule Kind, young people are blue.
  • rule All young, blue people are green.
  • rule Blue, green people are smart.

Statement: Harry is smart. Answer: true.

  1. Harry is kind.Harry is quiet.
  2. + Harry is kind.Harry is young.
  3. + Harry is kind.Harry is blue.
  4. Harry is green.
  5. Harry is smart.

Each rule produces a new fact that the next rule needs, from "Harry is kind" to "Harry is smart". Miss one link and the answer is out of reach. Across all open-world problems at depth 5, the models score from 30.1% to 81.0%.

Highlighted lines are the ones the proof uses; the rest are distractors. These are the shortest problems at their depth, so easier than typical. What proof depth means, with a worked example at every depth.

Overall accuracy on both tasks

The same 1,800 problems per task for every model. Models are ordered by accuracy; a model still being scored is listed last and marked partial.

Open world: true, false or unknown

A statement is true if it follows from the facts and rules, false if its negation follows, and unknown if neither does.

Overall accuracy, Open world: true, false or unknown
ModelAccuracy95% intervalMacro-F1Recall: trueRecall: falseRecall: unknownItems
Jev83.8%82.2–85.50.83888.8%88.3%74.2%1,800
GLiDE, Oct 282.9%81.1–84.60.83377.0%76.0%95.9%1,800
GLiDE, Oct 3 rerun76.4%74.4–78.30.77070.4%65.5%93.7%1,800
GPT-6 Luna64.1%61.7–66.20.64758.3%50.9%83.4%1,800
Kev-9B58.6%56.3–60.90.58859.2%48.6%68.1%1,800
Kev-4B53.6%51.4–56.00.53246.6%35.9%78.8%1,800
Kev-0.8B53.6%51.4–55.90.47758.2%89.6%11.9%1,800
GLiNER2.5-Decide, without instructions45.5%43.1–47.80.36665.3%69.4%0.7%1,800
Laya42.4%40.3–44.50.33740.2%85.1%1.0%1,800
GLiNER2.5-Decide32.8%30.7–34.90.1650.0%0.0%100.0%1,800

Chance is 33.3%; always giving the most common answer scores 33.6% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.

Closed world: true or false

Anything that cannot be derived is false, so there is no unknown answer.

Overall accuracy, Closed world: true or false
ModelAccuracy95% intervalMacro-F1Recall: trueRecall: falseItems
Jev89.3%87.8–90.70.89389.7%88.9%1,800
GLiDE, Oct 271.8%69.6–73.80.71775.7%67.9%1,800
GLiDE, Oct 3 rerun68.8%66.7–71.00.68872.8%64.9%1,800
GPT-6 Luna65.0%62.8–67.40.65066.2%63.8%1,800
Kev-9B63.8%61.5–66.10.63864.7%62.9%1,800
Kev-4B58.8%56.6–61.10.57642.2%75.3%1,800
Kev-0.8B55.8%53.5–57.90.55345.4%66.1%1,800
Laya55.6%53.3–58.10.50924.8%86.4%1,800
GLiNER2.5-Decide, without instructions54.3%52.0–56.60.54147.4%61.1%1,800
GLiNER2.5-Decide50.3%48.1–52.60.3532.1%98.6%1,800

Chance is 50.0%; always giving the most common answer scores 50.0% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.

Every model gets worse as proofs get deeper

Proof depth is the number of rules that must be chained to reach the answer. It is the one difficulty axis the sample controls.

Open-world task, change from depth 0 to depth 5 in points. Accuracy by proof depth has every depth, both tasks and the paired gaps.

Head-to-head

Where they fail

Accuracy by every property ProofWriter records. Only proof depth was controlled when the problems were sampled; the others move with depth and with each other, so they describe where errors fall rather than what causes them.

How we measured, consistency, speed and size

For teams evaluating or tuning a decision model

Not on the board yet

A model that has not been scored is never shown as zero. Every model.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.