Skip to main content
HardDecisions

Models · comparisons

Every decision model and LLM classifier we tested, compared with Jev

Each model answered the same 1,800 problems on each task, one request per problem. Here is how they compare overall and head to head with Jev, with what each one is and where it ran. Every rival has one page: its comparison with Jev.

Jev and its rivals, one page each

Jev accuracy on multi-hop reasoning

The reference model: every rival is compared with it on the same items.

Jev
Jev vs GLiDE: two hosted decision models on deep reasoning

GLiDE is Fastino's hosted decision model, released on 1 October 2026, which Fastino calls the first thinking decision model. It takes the same request as Jev, so both got exactly the same text, question and options for the same 3,600 problems. We ran GLiDE twice. The first run was on October 2, the day after launch. On October 3 Fastino's CEO asked us to re-evaluate GLiDE, saying Fastino had problems during the launch, so we ran exactly the same benchmark again with the same request. Both runs are shown side by side. Jev answers every decision in one parallel pass; GLiDE, by Fastino's own description, spends extra time and tokens on decisions it is unsure of. Fastino reports that GLiDE beats Jev on its Decision Index 0.2.1 (64.81 to 57.91); that is Fastino's benchmark, not this one.

Jev · GLiDE, Oct 3 rerun · GLiDE, Oct 2
Jev vs GPT-6 Luna: a decision model against an LLM classifier

Jev is built to make one decision per request. GPT-6 Luna is a general large language model, used here as a classifier with reasoning off. Both answered the same 3,600 problems. OpenAI's new Decisions API is built on a version of GPT-6 Luna, so these results preview the accuracy of a Luna-based decision model on multi-step reasoning. They are not a test of the API itself.

Jev · GPT-6 Luna
Kev vs Jev: is an open-source decision model a Jev alternative?

Kev is an open decision model you can run on your own hardware; Jev is hosted. We ran every Kev size that fits a 32 GB laptop against Jev on the same problems, and asked whether a bigger Kev helps.

Jev · Kev-9B · Kev-4B · Kev-0.8B
Jev vs GLiNER2.5-Decide: a hosted decision model against Fastino's open one

GLiNER2.5-Decide is Fastino's open 340M-parameter decision model, released under Apache 2.0, that answers in one pass. We ran it on a laptop against Jev on the same 3,600 problems. Every model received the same problem, the same question and the same meaning for each answer. GLiNER2.5-Decide is shown two ways. GLiNER2.5-Decide: the instructions and the meaning of each answer are in its text, after the problem, because Fastino's API for this model has no separate field for them; given them, it answered unknown to every open-world problem and false to almost every closed-world one. GLiNER2.5-Decide, without instructions: only the problem and the three answer words; its answers vary but stay near chance, even at depth 0, where the answer is written in the text. Fastino's hosted API for this model gave the same answer as our copy on 7,197 of the 7,200 problems in both forms, and the same setup scores 63.7% on Fastino's own benchmark data (Fastino reports 60.2%), so the model is working as published. Asked a plain yes-or-no question in the form its model card shows (“Is it true that …?”, with the facts as the passage), it does better: 76% at depth 0 on held-out ProofWriter problems whose answer is true or false, against Jev's 98% on the benchmark. That form leaves out the instructions, so it is not a benchmark entry.

Jev · GLiNER2.5-Decide · GLiNER2.5-Decide, without instructions
Jev vs Laya: hosted and open decision models on deep reasoning

Laya is a small open decision model built on an encoder. Both it and Jev answered the same problems.

Jev · Laya

The models

  1. Jev
    hosted decision model · ran on vendor's servers
    open world 83.8% · closed world 89.3%

    Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

  2. GLiDE, Oct 3 rerun
    hosted decision model · ran on vendor's servers
    open world 76.4% · closed world 68.8%

    The same GLiDE, run again on October 3 with exactly the same request and settings, because Fastino's CEO asked us to re-evaluate it: he said Fastino had problems during the launch. The October 2 run is kept and shown beside it. Fastino's API reports the model as "glide" both times, so the responses do not show what changed.

  3. GLiDE, Oct 2
    hosted decision model · ran on vendor's servers
    open world 82.9% · closed world 71.8%

    GLiDE is a hosted decision model made by Fastino Labs, which describes it as a "thinking" decision model: it forms a probability over the options and spends more computation when the leading option is uncertain. Fastino serves it on the same System One request format as Jev, so it got exactly Jev's request: the text, the question and the options, one request per item. Its size and architecture are not published.

  4. GPT-6 Luna
    LLM classifier · ran on vendor's servers
    open world 64.1% · closed world 65.0%

    GPT-6 Luna is a hosted large language model made by OpenAI. We used it as a classifier: one request per item through the Chat Completions API, with reasoning effort set to none (its lowest setting) and a strict JSON schema that allows only the task's options. Its size and architecture are not published.

  5. Kev-9B
    open decision model · ran on our laptop (Apple M1 Max, 32 GB)
    open world 58.6% · closed world 63.8%

    Kev-9B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 9-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

  6. Kev-4B
    open decision model · ran on our laptop (Apple M1 Max, 32 GB)
    open world 53.6% · closed world 58.8%

    Kev-4B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 4-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

  7. Kev-0.8B
    open decision model · ran on our laptop (Apple M1 Max, 32 GB)
    open world 53.6% · closed world 55.8%

    Kev-0.8B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 0.8-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

  8. GLiNER2.5-Decide
    open decision model · ran on our laptop (Apple M1 Max, 32 GB)
    open world 32.8% · closed world 50.3%

    GLiNER2.5-Decide is an open decision model made by Fastino Labs: a 340M-parameter schema classifier on a DeBERTa-v3-large encoder, released under Apache 2.0. It answers in one pass; its model card says it "does not reason." We ran it on our own laptop with Fastino's gliner2 package, PyTorch on Apple's GPU, with everything every other model gets: the problem, then the question's instructions and each option with its description, in the message, and the options as its answer labels. On the same inputs, our copy gave Fastino's hosted model's answers.

  9. GLiNER2.5-Decide, without instructions
    open decision model · ran on our laptop (Apple M1 Max, 32 GB)
    open world 45.5% · closed world 54.3%

    The same GLiNER2.5-Decide checkpoint asked in the bare form Fastino's hosted API documents: the problem text, which ends with the statement, and the answer options as labels, without the question's instructions or the option descriptions. It is shown beside the full form to show what those add.

  10. Laya
    open decision model · ran on our laptop (Apple M1 Max, 32 GB)
    open world 42.4% · closed world 55.6%

    Laya is an open decision model made by Convai Innovations: a 421M-parameter ModernBERT-large encoder with decision heads. We ran it on our own laptop with the laya package, PyTorch on Apple's GPU.

  11. Kev-27B
    Not run

    Kev-27B is the largest Kev: 27B, every weight fine-tuned from Qwen3.8-27B (post-trained), with 51.26 GB full weights (jaredpalmer/kev-27b). It was not run: does not fit this 32 GB machine; would need a rented 80 GB GPU.

Accuracy side by side

Open world: true, false or unknown

Overall accuracy, Open world: true, false or unknown
ModelAccuracy95% intervalMacro-F1Recall: trueRecall: falseRecall: unknownItems
Jev83.8%82.2–85.50.83888.8%88.3%74.2%1,800
GLiDE, Oct 282.9%81.1–84.60.83377.0%76.0%95.9%1,800
GLiDE, Oct 3 rerun76.4%74.4–78.30.77070.4%65.5%93.7%1,800
GPT-6 Luna64.1%61.7–66.20.64758.3%50.9%83.4%1,800
Kev-9B58.6%56.3–60.90.58859.2%48.6%68.1%1,800
Kev-4B53.6%51.4–56.00.53246.6%35.9%78.8%1,800
Kev-0.8B53.6%51.4–55.90.47758.2%89.6%11.9%1,800
GLiNER2.5-Decide, without instructions45.5%43.1–47.80.36665.3%69.4%0.7%1,800
Laya42.4%40.3–44.50.33740.2%85.1%1.0%1,800
GLiNER2.5-Decide32.8%30.7–34.90.1650.0%0.0%100.0%1,800

Chance is 33.3%; always giving the most common answer scores 33.6% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.

Closed world: true or false

Overall accuracy, Closed world: true or false
ModelAccuracy95% intervalMacro-F1Recall: trueRecall: falseItems
Jev89.3%87.8–90.70.89389.7%88.9%1,800
GLiDE, Oct 271.8%69.6–73.80.71775.7%67.9%1,800
GLiDE, Oct 3 rerun68.8%66.7–71.00.68872.8%64.9%1,800
GPT-6 Luna65.0%62.8–67.40.65066.2%63.8%1,800
Kev-9B63.8%61.5–66.10.63864.7%62.9%1,800
Kev-4B58.8%56.6–61.10.57642.2%75.3%1,800
Kev-0.8B55.8%53.5–57.90.55345.4%66.1%1,800
Laya55.6%53.3–58.10.50924.8%86.4%1,800
GLiNER2.5-Decide, without instructions54.3%52.0–56.60.54147.4%61.1%1,800
GLiNER2.5-Decide50.3%48.1–52.60.3532.1%98.6%1,800

Chance is 50.0%; always giving the most common answer scores 50.0% on these items. Recall is the share of items with that correct answer that the model got right. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.

Jev against every model, on the same items

Jev minus each model, in points, with the 95% interval of the paired difference. Positive means Jev was more accurate. Because both answered exactly the same items, the interval is narrower than comparing two separate scores.

ModelOpen world: overalldepth 0depth 5Closed world: overalldepth 0depth 5
GLiDE, Oct 3 rerun+7.4 +5.2 to +9.90.0 −1.7 to +2.0+19.0 +12.5 to +26.6+20.4 +18.2 to +22.7+0.3 −1.0 to +1.7+45.3 +39.3 to +51.3
GLiDE, Oct 2+0.9 −1.2 to +3.30.0 −1.7 to +2.0+5.9 −0.7 to +13.1+17.5 +15.4 to +19.60.0 −1.0 to +1.0+40.0 +33.7 to +46.0
GPT-6 Luna+19.8 +17.2 to +22.4+4.3 +1.0 to +7.7+34.9 +27.3 to +42.9+24.3 +21.9 to +26.6+1.3 0.0 to +3.0+44.7 +38.0 to +51.3
Kev-9B+25.3 +22.7 to +27.8+8.3 +5.3 to +11.7+39.8 +32.9 to +47.8+25.5 +23.0 to +27.8+9.7 +6.3 to +13.3+46.3 +40.3 to +52.3
Kev-4B+30.3 +27.4 to +32.9+10.0 +6.3 to +13.7+44.6 +37.0 to +51.9+30.5 +27.9 to +33.1+13.7 +9.7 to +18.0+46.0 +39.7 to +52.3
Kev-0.8B+30.3 +27.7 to +32.7+27.3 +21.7 to +32.7+30.1 +23.5 to +36.0+33.5 +30.7 to +36.3+31.3 +26.0 to +36.7+37.7 +31.0 to +44.0
GLiNER2.5-Decide+51.1 +48.3 to +54.1+64.3 +59.0 to +70.0+50.9 +42.6 to +59.2+38.9 +36.1 to +41.7+49.7 +44.0 to +55.7+39.3 +33.3 to +46.0
GLiNER2.5-Decide, without instructions+38.3 +35.4 to +41.3+52.7 +47.0 to +58.7+36.3 +30.4 to +42.2+35.0 +32.1 to +37.9+47.0 +41.3 to +52.7+27.3 +21.0 to +34.3
Laya+41.4 +38.8 to +44.0+35.0 +29.3 to +40.7+46.7 +40.1 to +52.9+33.7 +30.7 to +36.5+27.3 +22.3 to +33.0+43.3 +37.0 to +50.3

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.