Skip to main content
HardDecisions

Breakdown · descriptive

Accuracy on true, false and unknown answers

The correct answer. On the open-world task a third of the items are unknown: neither the statement nor its negation follows. That class is where several models fail.

14.5On the open-world task, Jev's accuracy ranges from 74.2% (unknown) to 88.8% (true), a spread of 14.5 points. Jev is the most accurate model overall; every model is below.

Descriptive, not causal: the sample controls proof depth and the answer, not gold answer. Across these slices the average proof depth runs from 2.4 to 2.5, so depth differs between slices and can add to or mask any gap. The answer mix and mean depth of every slice are in the tables.

Every model's answers as a confusion matrix

A confusion matrix sorts every problem by its correct answer (the rows) and what the model said (the columns). Right answers sit on the diagonal; everything off it is an error, and where the errors land shows which answer a model reaches for when it is wrong. Fully scored models, most accurate first, on the same 1,800 open-world problems.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
GLiDE, Oct 2, open-world task
Correct ↓ · said →truefalseunknown
true4660139
false2460143
unknown177566
GLiDE, Oct 3 rerun, open-world task
Correct ↓ · said →truefalseunknown
true4260179
false3396206
unknown2314553
GPT-6 Luna (reasoning off), open-world task
Correct ↓ · said →truefalseunknown
true3533249
false7308290
unknown4553492
Kev-9B, open-world task
Correct ↓ · said →truefalseunknown
true35837210
false74294237
unknown13157402
Kev-4B, open-world task
Correct ↓ · said →truefalseunknown
true28212311
false45217343
unknown8936465
Kev-0.8B, open-world task
Correct ↓ · said →truefalseunknown
true35220944
false595424
unknown15037070
GLiNER2.5-Decide, without instructions, open-world task
Correct ↓ · said →truefalseunknown
true3952100
false1854200
unknown2663204
Laya, open-world task
Correct ↓ · said →truefalseunknown
true2433548
false895151
unknown1274576
GLiNER2.5-Decide, open-world task
Correct ↓ · said →truefalseunknown
true00605
false00605
unknown00590

Jev got 74% of unknown items right; it said true on 21% of them. GLiDE, Oct 2 got 76% of false items right; it said unknown on 24% of them. GLiDE, Oct 3 rerun got 65% of false items right; it said unknown on 34% of them. GPT-6 Luna (reasoning off) got 51% of false items right; it said unknown on 48% of them. Kev-9B got 49% of false items right; it said unknown on 39% of them. Kev-4B got 36% of false items right; it said unknown on 57% of them. Kev-0.8B got 12% of unknown items right; it said false on 63% of them. GLiNER2.5-Decide, without instructions got 1% of unknown items right; it said false on 54% of them. Laya got 1% of unknown items right; it said false on 77% of them. GLiNER2.5-Decide got 0% of true items right; it said unknown on 100% of them.

The closed-world task (true or false)
Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
GLiDE, Oct 2, closed-world task
Correct ↓ · said →truefalse
true681219
false289611
GLiDE, Oct 3 rerun, closed-world task
Correct ↓ · said →truefalse
true655245
false316584
GPT-6 Luna (reasoning off), closed-world task
Correct ↓ · said →truefalse
true596304
false326574
Kev-9B, closed-world task
Correct ↓ · said →truefalse
true582318
false334566
Kev-4B, closed-world task
Correct ↓ · said →truefalse
true380520
false222678
Kev-0.8B, closed-world task
Correct ↓ · said →truefalse
true409491
false305595
Laya, closed-world task
Correct ↓ · said →truefalse
true223677
false122778
GLiNER2.5-Decide, without instructions, closed-world task
Correct ↓ · said →truefalse
true427473
false350550
GLiNER2.5-Decide, closed-world task
Correct ↓ · said →truefalse
true19881
false13887

Jev said true on 50% of problems; the correct answer is true on 50%. GLiDE, Oct 2 said true on 54% of problems; the correct answer is true on 50%. GLiDE, Oct 3 rerun said true on 54% of problems; the correct answer is true on 50%. GPT-6 Luna (reasoning off) said true on 51% of problems; the correct answer is true on 50%. Kev-9B said true on 51% of problems; the correct answer is true on 50%. Kev-4B said false on 67% of problems; the correct answer is false on 50%. Kev-0.8B said false on 60% of problems; the correct answer is false on 50%. Laya said false on 81% of problems; the correct answer is false on 50%. GLiNER2.5-Decide, without instructions said false on 57% of problems; the correct answer is false on 50%. GLiNER2.5-Decide said false on 98% of problems; the correct answer is false on 50%.

Gold answer, open world task

A statement is true if it follows from the facts and rules, false if its negation follows, and unknown if neither does. Each line is one model: its accuracy, with the 95% interval as a bar. The dashed line is chance (33%); the grey tick is the score of always giving the slice's most common answer.

  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
  • GPT-6 Luna(reasoning off)
  • Kev-9B
  • Kev-4B
  • Kev-0.8B
  • GLiNER2.5-Decide
  • GLiNER2.5-Decide, without instructions
  • Laya
false605 items
true605 items
unknown590 items
Gold answer, Open world
Gold answerItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2GPT-6 LunaKev-9BKev-4BKev-0.8BGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsLayaBest constantAnswer mix · mean depth
false60588.3% 85.5–90.965.5% 61.5–69.376.0% 72.4–79.550.9% 46.8–54.948.6% 44.6–52.635.9% 31.9–39.589.6% 87.1–91.70.0% 0.0–0.069.4% 65.3–72.985.1% 82.1–87.9one answerfalse 100% · 2.5
true60588.8% 86.1–91.170.4% 66.6–73.977.0% 73.6–80.558.3% 54.0–62.159.2% 55.4–63.146.6% 42.8–50.658.2% 54.0–62.00.0% 0.0–0.065.3% 61.7–69.140.2% 36.2–44.1one answertrue 100% · 2.5
unknown59074.2% 70.7–77.693.7% 91.7–95.695.9% 94.2–97.583.4% 80.5–86.668.1% 64.6–71.778.8% 75.4–81.911.9% 9.3–14.6100.0% 100.0–100.00.7% 0.2–1.51.0% 0.3–1.9one answerunknown 100% · 2.4

Gold answer, closed world task

Anything that cannot be derived is false, so there is no unknown answer. Each line is one model: its accuracy, with the 95% interval as a bar. The dashed line is chance (50%); the grey tick is the score of always giving the slice's most common answer.

  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
  • GPT-6 Luna(reasoning off)
  • Kev-9B
  • Kev-4B
  • Kev-0.8B
  • GLiNER2.5-Decide
  • GLiNER2.5-Decide, without instructions
  • Laya
false900 items
true900 items
Gold answer, Closed world
Gold answerItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2GPT-6 LunaKev-9BKev-4BKev-0.8BGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsLayaBest constantAnswer mix · mean depth
false90088.9% 86.9–90.964.9% 61.9–68.167.9% 64.9–71.063.8% 60.6–66.962.9% 59.9–65.975.3% 72.8–78.266.1% 63.0–69.398.6% 97.8–99.261.1% 58.0–64.386.4% 84.1–88.6one answerfalse 100% · 2.5
true90089.7% 87.7–91.672.8% 69.8–75.775.7% 72.7–78.366.2% 63.1–69.264.7% 61.7–67.642.2% 39.2–45.445.4% 42.1–48.62.1% 1.2–3.147.4% 44.2–50.624.8% 21.9–27.6one answertrue 100% · 2.5

How often each model says unknown

On the open-world task, about a third of the correct answers are unknown. A model that rarely says unknown cannot get those items right.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.