Breakdown · descriptive
Negation: in the rules and in the statement
Two kinds of "not": in the theory's facts and rules, and in the statement to judge.
Descriptive, not causal: the sample controls proof depth and the answer, not these properties. Several of them grow with depth and with each other, so each section gives the average proof depth of its slices; the answer mix and mean depth of every slice are in the tables.
Does negation in the rules lower accuracy?
Whether the theory's facts and rules contain "not".
On the open-world task, Jev's accuracy ranges from 80.3% (without negation) to 87.5% (with negation), a spread of 7.2 points.
Descriptive, not causal. The average proof depth of these slices runs from 2.5 to 2.5, so depth differs between them and can add to or mask any gap.
Negation in the theory, open world task
- Jev
- GLiDE, Oct 3 rerun
- GLiDE, Oct 2
- GPT-6 Luna(reasoning off)
- Kev-9B
- Kev-4B
- Kev-0.8B
- GLiNER2.5-Decide
- GLiNER2.5-Decide, without instructions
- Laya
| Negation in the theory | Items | Jev | GLiDE, Oct 3 rerun | GLiDE, Oct 2 | GPT-6 Luna | Kev-9B | Kev-4B | Kev-0.8B | GLiNER2.5-Decide | GLiNER2.5-Decide, without instructions | Laya | Best constant | Answer mix · mean depth |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| with negation | 881 | 87.5% 85.4–89.7 | 79.3% 76.8–82.1 | 85.8% 83.8–88.1 | 67.1% 63.9–70.3 | 60.4% 57.1–63.8 | 57.3% 54.0–60.7 | 56.6% 53.6–59.6 | 30.0% 27.0–32.8 | 31.0% 28.1–33.9 | 46.2% 43.0–49.5 | 35.1% | false 35% · true 35% · unknown 30% · 2.5 |
| without negation | 919 | 80.3% 77.7–83.1 | 73.6% 70.7–76.2 | 80.1% 77.5–82.6 | 61.2% 58.1–64.1 | 56.8% 53.5–59.8 | 49.9% 46.9–53.1 | 50.6% 47.3–54.1 | 35.5% 32.6–38.5 | 59.4% 56.1–62.5 | 38.8% 35.5–41.9 | 35.5% | false 32% · true 32% · unknown 35% · 2.5 |
Negation in the theory, closed world task
- Jev
- GLiDE, Oct 3 rerun
- GLiDE, Oct 2
- GPT-6 Luna(reasoning off)
- Kev-9B
- Kev-4B
- Kev-0.8B
- GLiNER2.5-Decide
- GLiNER2.5-Decide, without instructions
- Laya
| Negation in the theory | Items | Jev | GLiDE, Oct 3 rerun | GLiDE, Oct 2 | GPT-6 Luna | Kev-9B | Kev-4B | Kev-0.8B | GLiNER2.5-Decide | GLiNER2.5-Decide, without instructions | Laya | Best constant | Answer mix · mean depth |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| with negation | 870 | 93.7% 92.1–95.3 | 73.9% 70.9–76.9 | 77.5% 74.6–80.2 | 68.6% 65.4–71.8 | 66.4% 63.3–69.7 | 61.4% 57.9–64.6 | 57.4% 54.1–60.7 | 49.9% 46.8–53.0 | 53.7% 50.3–56.7 | 57.4% 54.1–60.5 | 50.1% | false 50% · true 50% · 2.5 |
| without negation | 930 | 85.2% 82.8–87.5 | 64.1% 61.0–67.2 | 66.5% 63.3–69.5 | 61.6% 58.6–64.7 | 61.3% 58.3–64.6 | 56.3% 53.0–59.2 | 54.3% 51.0–57.4 | 50.8% 47.7–54.0 | 54.8% 51.6–58.1 | 54.0% 50.5–57.3 | 50.1% | false 50% · true 50% · 2.5 |
Accuracy on negated statements
Whether the statement to judge itself contains "not" ("The lion is not blue").
On the open-world task, Jev's accuracy ranges from 78.6% (negated statement) to 89.1% (plain statement), a spread of 10.5 points.
Descriptive, not causal. The average proof depth of these slices runs from 2.5 to 2.5, so depth differs between them and can add to or mask any gap.
Negated statement, open world task
- Jev
- GLiDE, Oct 3 rerun
- GLiDE, Oct 2
- GPT-6 Luna(reasoning off)
- Kev-9B
- Kev-4B
- Kev-0.8B
- GLiNER2.5-Decide
- GLiNER2.5-Decide, without instructions
- Laya
| Negated statement | Items | Jev | GLiDE, Oct 3 rerun | GLiDE, Oct 2 | GPT-6 Luna | Kev-9B | Kev-4B | Kev-0.8B | GLiNER2.5-Decide | GLiNER2.5-Decide, without instructions | Laya | Best constant | Answer mix · mean depth |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| plain statement | 899 | 89.1% 87.1–91.0 | 75.0% 72.0–77.8 | 81.6% 78.9–84.1 | 62.3% 59.2–65.3 | 59.7% 56.4–63.0 | 53.6% 50.7–56.6 | 48.5% 45.4–51.7 | 32.4% 29.4–35.4 | 43.4% 40.3–46.6 | 38.7% 35.5–42.0 | 41.5% | false 26% · true 41% · unknown 32% · 2.5 |
| negated statement | 901 | 78.6% 75.9–81.2 | 77.8% 75.1–80.2 | 84.1% 81.7–86.3 | 65.8% 62.9–68.8 | 57.4% 53.8–60.4 | 53.5% 50.2–56.8 | 58.6% 55.5–61.7 | 33.2% 30.1–36.2 | 47.6% 44.4–50.9 | 46.2% 43.1–49.5 | 41.1% | false 41% · true 26% · unknown 33% · 2.5 |
Negated statement, closed world task
- Jev
- GLiDE, Oct 3 rerun
- GLiDE, Oct 2
- GPT-6 Luna(reasoning off)
- Kev-9B
- Kev-4B
- Kev-0.8B
- GLiNER2.5-Decide
- GLiNER2.5-Decide, without instructions
- Laya
| Negated statement | Items | Jev | GLiDE, Oct 3 rerun | GLiDE, Oct 2 | GPT-6 Luna | Kev-9B | Kev-4B | Kev-0.8B | GLiNER2.5-Decide | GLiNER2.5-Decide, without instructions | Laya | Best constant | Answer mix · mean depth |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| plain statement | 882 | 89.6% 87.5–91.6 | 71.4% 68.5–74.5 | 74.3% 71.7–77.1 | 67.5% 64.4–70.5 | 65.6% 62.6–68.7 | 58.6% 55.3–61.7 | 53.6% 50.3–56.9 | 46.0% 42.9–49.2 | 54.1% 51.1–57.4 | 56.7% 53.5–59.8 | 54.8% | false 45% · true 55% · 2.5 |
| negated statement | 918 | 89.0% 87.0–91.0 | 66.3% 63.2–69.5 | 69.4% 66.4–72.4 | 62.6% 59.4–65.8 | 62.0% 58.9–65.0 | 58.9% 55.7–62.2 | 57.8% 54.7–60.9 | 54.5% 51.4–57.5 | 54.5% 51.3–57.4 | 54.6% 51.3–57.7 | 54.6% | false 55% · true 45% · 2.5 |
In each chart a line is one model: its accuracy, with the 95% interval as a bar. The dashed line is chance; the grey tick is the score of always giving the slice's most common answer.
Version v0.1.0, unreleased.
Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.
Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.