Guide · aligning decision models
Aligning decision models with the decision you mean
A decision model can be accurate and still answer a different question from the one you meant. The open-world and closed-world tasks ask the same problems under two rules for what "unknown" means; the gap between them is an alignment problem you can measure.
Two rules for what cannot be proved
Under the open-world rule, a statement nothing supports is unknown. Under the closed-world rule, it is false. The two tasks state their rule in the question, and each model got the rule for its task.
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
Accuracy under each rule
- Jevopen world 83.8%, closed world 89.3%+5.4
- GLiDE, Oct 3 rerunopen world 76.4%, closed world 68.8%−7.6
- GLiDE, Oct 2open world 82.9%, closed world 71.8%−11.1
- GPT-6 Lunaopen world 64.1%, closed world 65.0%+0.9
- Kev-9Bopen world 58.6%, closed world 63.8%+5.2
- Kev-4Bopen world 53.6%, closed world 58.8%+5.2
- Kev-0.8Bopen world 53.6%, closed world 55.8%+2.2
- GLiNER2.5-Decideopen world 32.8%, closed world 50.3%+17.6
- GLiNER2.5-Decide, without instructionsopen world 45.5%, closed world 54.3%+8.8
- Layaopen world 42.4%, closed world 55.6%+13.2
Closed-world minus open-world accuracy, in points. Chance is 50% on the closed-world task and 33% on the open-world one, so higher closed-world numbers are expected; the gap above chance is what to compare.
Open world: true, false or unknown
| Model | Accuracy | 95% interval | Macro-F1 | Recall: true | Recall: false | Recall: unknown | Items |
|---|---|---|---|---|---|---|---|
| Jev | 83.8% | 82.2–85.5 | 0.838 | 88.8% | 88.3% | 74.2% | 1,800 |
| GLiDE, Oct 2 | 82.9% | 81.1–84.6 | 0.833 | 77.0% | 76.0% | 95.9% | 1,800 |
| GLiDE, Oct 3 rerun | 76.4% | 74.4–78.3 | 0.770 | 70.4% | 65.5% | 93.7% | 1,800 |
| GPT-6 Luna | 64.1% | 61.7–66.2 | 0.647 | 58.3% | 50.9% | 83.4% | 1,800 |
| Kev-9B | 58.6% | 56.3–60.9 | 0.588 | 59.2% | 48.6% | 68.1% | 1,800 |
| Kev-4B | 53.6% | 51.4–56.0 | 0.532 | 46.6% | 35.9% | 78.8% | 1,800 |
| Kev-0.8B | 53.6% | 51.4–55.9 | 0.477 | 58.2% | 89.6% | 11.9% | 1,800 |
| GLiNER2.5-Decide, without instructions | 45.5% | 43.1–47.8 | 0.366 | 65.3% | 69.4% | 0.7% | 1,800 |
| Laya | 42.4% | 40.3–44.5 | 0.337 | 40.2% | 85.1% | 1.0% | 1,800 |
| GLiNER2.5-Decide | 32.8% | 30.7–34.9 | 0.165 | 0.0% | 0.0% | 100.0% | 1,800 |
Chance is 33.3%; always giving the most common answer scores 33.6% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.
Closed world: true or false
| Model | Accuracy | 95% interval | Macro-F1 | Recall: true | Recall: false | Items |
|---|---|---|---|---|---|---|
| Jev | 89.3% | 87.8–90.7 | 0.893 | 89.7% | 88.9% | 1,800 |
| GLiDE, Oct 2 | 71.8% | 69.6–73.8 | 0.717 | 75.7% | 67.9% | 1,800 |
| GLiDE, Oct 3 rerun | 68.8% | 66.7–71.0 | 0.688 | 72.8% | 64.9% | 1,800 |
| GPT-6 Luna | 65.0% | 62.8–67.4 | 0.650 | 66.2% | 63.8% | 1,800 |
| Kev-9B | 63.8% | 61.5–66.1 | 0.638 | 64.7% | 62.9% | 1,800 |
| Kev-4B | 58.8% | 56.6–61.1 | 0.576 | 42.2% | 75.3% | 1,800 |
| Kev-0.8B | 55.8% | 53.5–57.9 | 0.553 | 45.4% | 66.1% | 1,800 |
| Laya | 55.6% | 53.3–58.1 | 0.509 | 24.8% | 86.4% | 1,800 |
| GLiNER2.5-Decide, without instructions | 54.3% | 52.0–56.6 | 0.541 | 47.4% | 61.1% | 1,800 |
| GLiNER2.5-Decide | 50.3% | 48.1–52.6 | 0.353 | 2.1% | 98.6% | 1,800 |
Chance is 50.0%; always giving the most common answer scores 50.0% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.
Does the model follow the rule it was given?
On the open-world task, 590 items have unknown as the correct answer. What each model said on them:
| Model | Said unknown | Said true | Said false |
|---|---|---|---|
| Jev | 74.2% | 21.0% | 4.7% |
| GLiDE, Oct 3 rerun | 93.7% | 3.9% | 2.4% |
| GLiDE, Oct 2 | 95.9% | 2.9% | 1.2% |
| GPT-6 Luna | 83.4% | 7.6% | 9.0% |
| Kev-9B | 68.1% | 22.2% | 9.7% |
| Kev-4B | 78.8% | 15.1% | 6.1% |
| Kev-0.8B | 11.9% | 25.4% | 62.7% |
| GLiNER2.5-Decide | 100.0% | 0.0% | 0.0% |
| GLiNER2.5-Decide, without instructions | 0.7% | 45.1% | 54.2% |
| Laya | 1.0% | 21.5% | 77.5% |
Kev-0.8B, GLiNER2.5-Decide, without instructions and Laya answer these items as if the closed-world rule applied, even though the question defines unknown and offers it: Kev-0.8B said false on 62.7%, GLiNER2.5-Decide, without instructions said false on 54.2% and Laya said false on 77.5%. That is not a reasoning failure so much as a different decision from the one asked for, and it is the kind of misalignment that a single accuracy number hides.
Aligning Jev
Jev scored 83.8% under the open-world rule and 89.3% under the closed-world one, the highest of the models here under both. On unknown items it said true 21.0% of the time and false 4.7%: when it errs on "not enough information", it leans toward yes. If your decision treats "cannot tell" as its own outcome, that is the error to test and correct for first. Jev's full results.
How to check alignment on your own decisions
- Write the decision rule down, including what "unknown" or "not enough information" means, and put it in the instructions the model gets.
- Build test items where the rule matters: the same facts that should lead to different answers under different rules.
- Look at what the model says on each correct answer, not only its accuracy; a lean toward one answer is a misalignment you can see in one table.
- Change the rule and check the answers move the way they should. A model that answers the same under both rules is not reading the rule.
- Repeat the test after any change of model version, instructions or tuning.
Version v0.1.0, unreleased.
Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.
Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.