About

Who makes this and how we tested each model

Who makes this site, how it is funded, and how each model was reached.

Who makes this

Hard-Decisions is made by Anthus AI Solutions, an independent AI engineering consultancy. The work is self-funded. It follows the design of its sibling, Biased-Decisions, which measures whether the same models change their decisions when a text changes who a person is.

The models and how we reached them

Jev

Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

GPT-6 Luna

GPT-6 Luna is a hosted large language model made by OpenAI. We used it as a classifier: one request per item through the Chat Completions API, with reasoning effort set to none (its lowest setting) and a strict JSON schema that allows only the task's options. Its size and architecture are not published.

Kev-9B

Kev-9B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 9-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

Kev-4B

Kev-4B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 4-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

Kev-0.8B

Kev-0.8B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 0.8-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

Laya

Laya is an open decision model made by Convai Innovations: a 421M-parameter ModernBERT-large encoder with decision heads. We ran it on our own laptop with the laya package, PyTorch on Apple's GPU.

Kev-27B

Kev-27B is the largest Kev: 27B, every weight fine-tuned from Qwen3.8-27B (post-trained), with 51.26 GB full weights (jaredpalmer/kev-27b). It was not run: does not fit this 32 GB machine; would need a rented 80 GB GPU.

The dataset

ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. The example problems shown on this site are quoted from it. How we sampled and scored it.

Reporting an error in a result

For general contact, visit the contact page on anth.us.

Analytics and privacy

The site uses Google Analytics. To turn off tracking for your browser, visit any page with ?notrack=1 added to the address. Add ?notrack=0 to turn it back on.

Six things to know before you read the results

ProofWriter is syntheticIts theories are generated from templates (some reworded by people), and models may have seen it in training. It measures one kind of reasoning, not decisions in your domain.
One protocol for every modelEach model answered each item once, with the same question and the same options. GPT-6 Luna ran with reasoning off, its lowest setting, as a direct classifier; another setting would be another model.
Latency is not like for likeHosted models' times include the network and the vendor's servers. Open models ran on one laptop. Compare speed within a setting, not across settings.
Only depth is controlledThe sample balances proof depth and the answer. Other breakdowns, such as negation, length or paraphrase, move with depth and with each other: they show where errors fall, not why.
Predictions came firstWe wrote down what we expected before any model answered, and report each prediction as held or failed, including the ones we got wrong.
Every number re-runs offlineScores are computed from the saved answers; one command rebuilds them without calling a model. Results still being scored are marked as such, never filled in.