Guide · aligning decision models
Aligning decision models with the decision you mean
A decision model can be accurate and still answer a different question from the one you meant. The open-world and closed-world tasks ask the same problems under two rules for what "unknown" means; the gap between them is an alignment problem you can measure.
Two rules for what cannot be proved
Under the open-world rule, a statement nothing supports is unknown. Under the closed-world rule, it is false. The two tasks state their rule in the question, and each model got the rule for its task.
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
Accuracy under each rule
- Jevopen world 83.8%, closed world 89.3%+5.4
- GPT-6 Lunaopen world 64.1%, closed world 65.0%+0.9
- Kev-9Bopen world 58.6%, closed world 63.8%+5.2
- Kev-4Bopen world 53.6%, closed world 58.8%+5.2
- Kev-0.8Bopen world 53.6%, closed world 55.8%+2.2
- Layaopen world 42.4%, closed world 55.6%+13.2
Closed-world minus open-world accuracy, in points. Chance is 50% on the closed-world task and 33% on the open-world one, so higher closed-world numbers are expected; the gap above chance is what to compare.
Open world: true, false or unknown
| Model | Accuracy | 95% interval | Macro-F1 | Recall: true | Recall: false | Recall: unknown | Items |
|---|---|---|---|---|---|---|---|
| Jev | 83.8% | 82.2–85.5 | 0.838 | 88.8% | 88.3% | 74.2% | 1,800 |
| GPT-6 Luna | 64.1% | 61.7–66.2 | 0.647 | 58.3% | 50.9% | 83.4% | 1,800 |
| Kev-9B | 58.6% | 56.3–60.9 | 0.588 | 59.2% | 48.6% | 68.1% | 1,800 |
| Kev-4B | 53.6% | 51.4–56.0 | 0.532 | 46.6% | 35.9% | 78.8% | 1,800 |
| Kev-0.8B | 53.6% | 51.4–55.9 | 0.477 | 58.2% | 89.6% | 11.9% | 1,800 |
| Laya | 42.4% | 40.3–44.5 | 0.337 | 40.2% | 85.1% | 1.0% | 1,800 |
Chance is 33.3%; always giving the most common answer scores 33.6% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.
Closed world: true or false
| Model | Accuracy | 95% interval | Macro-F1 | Recall: true | Recall: false | Items |
|---|---|---|---|---|---|---|
| Jev | 89.3% | 87.8–90.7 | 0.893 | 89.7% | 88.9% | 1,800 |
| GPT-6 Luna | 65.0% | 62.8–67.4 | 0.650 | 66.2% | 63.8% | 1,800 |
| Kev-9B | 63.8% | 61.5–66.1 | 0.638 | 64.7% | 62.9% | 1,800 |
| Kev-4B | 58.8% | 56.6–61.1 | 0.576 | 42.2% | 75.3% | 1,800 |
| Kev-0.8B | 55.8% | 53.5–57.9 | 0.553 | 45.4% | 66.1% | 1,800 |
| Laya | 55.6% | 53.3–58.1 | 0.509 | 24.8% | 86.4% | 1,800 |
Chance is 50.0%; always giving the most common answer scores 50.0% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.
Does the model follow the rule it was given?
On the open-world task, 590 items have unknown as the correct answer. What each model said on them:
| Model | Said unknown | Said true | Said false |
|---|---|---|---|
| Jev | 74.2% | 21.0% | 4.7% |
| GPT-6 Luna | 83.4% | 7.6% | 9.0% |
| Kev-9B | 68.1% | 22.2% | 9.7% |
| Kev-4B | 78.8% | 15.1% | 6.1% |
| Kev-0.8B | 11.9% | 25.4% | 62.7% |
| Laya | 1.0% | 21.5% | 77.5% |
Kev-0.8B and Laya answer these items as if the closed-world rule applied, even though the question defines unknown and offers it: Kev-0.8B said false on 62.7% and Laya said false on 77.5%. That is not a reasoning failure so much as a different decision from the one asked for, and it is the kind of misalignment that a single accuracy number hides.
Aligning Jev
Jev scored 83.8% under the open-world rule and 89.3% under the closed-world one, the highest of the models here under both. On unknown items it said true 21.0% of the time and false 4.7%: when it errs on "not enough information", it leans toward yes. If your decision treats "cannot tell" as its own outcome, that is the error to test and correct for first. Jev's full results.
How to check alignment on your own decisions
- Write the decision rule down, including what "unknown" or "not enough information" means, and put it in the instructions the model gets.
- Build test items where the rule matters: the same facts that should lead to different answers under different rules.
- Look at what the model says on each correct answer, not only its accuracy; a lean toward one answer is a misalignment you can see in one table.
- Change the rule and check the answers move the way they should. A model that answers the same under both rules is not reading the rule.
- Repeat the test after any change of model version, instructions or tuning.