Skip to main content
HardDecisions

Comparison · Jev vs LLM

Jev vs GPT-6 Luna: a decision model against an LLM classifier

Jev is built to make one decision per request. GPT-6 Luna is a general large language model, used here as a classifier with reasoning off. Both answered the same 3,600 problems.

GPT-6 Luna ran with reasoning effort set to none, its lowest setting, under the same one-request protocol as every model here: the same question, the same options, one answer per item. A different reasoning setting would be a different model, scored separately.

OpenAI's new Decisions API is built on a version of GPT-6 Luna, so these results preview the accuracy of a Luna-based decision model on multi-step reasoning. They are not a test of the API itself. What this means for the OpenAI Decisions API.

Overall: Jev and GPT-6 Luna

Open world: true, false or unknown

  • Jev82.2 to 85.5 (95%), 1,800 items83.8%
  • GPT-6 Luna61.7 to 66.2 (95%), 1,800 items64.1%

Jev minus GPT-6 Luna: +19.8 points on the same 1,800 items, 95% interval +17.2 to +22.4: Jev is clearly ahead.

Closed world: true or false

  • Jev87.8 to 90.7 (95%), 1,800 items89.3%
  • GPT-6 Luna62.8 to 67.4 (95%), 1,800 items65.0%

Jev minus GPT-6 Luna: +24.3 points on the same 1,800 items, 95% interval +21.9 to +26.6: Jev is clearly ahead.

Jev vs GPT-6 Luna by proof depth

Each point is about 300 problems that need that many inference steps, with its 95% interval.

Open world: true, false or unknown
  • Jev
  • GPT-6 Luna(reasoning off)
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
  • GPT-6 Luna(reasoning off)
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Jev minus GPT-6 Luna by depth, open world

depth 0300 items
+4.3
depth 1302 items
+9.9
depth 2303 items
+24.4
depth 3303 items
+18.2
depth 4303 items
+27.4
depth 5289 items
+34.9

Jev minus GPT-6 Luna by depth, closed world

depth 0300 items
+1.3
depth 1300 items
+17.0
depth 2300 items
+26.7
depth 3300 items
+28.0
depth 4300 items
+28.0
depth 5300 items
+44.7

Paired differences: positive means Jev was more accurate on the same items.

Accuracy by depth, open world

By depth, Open world
Proof depthItemsJevGPT-6 LunaBest constant
depth 030097.7% 96.0–99.393.3% 90.3–96.033.3%
depth 130287.7% 83.8–91.477.8% 73.2–82.833.4%
depth 230384.8% 80.2–88.460.4% 54.5–66.033.3%
depth 330379.9% 75.2–84.561.7% 56.8–67.733.3%
depth 430371.9% 66.7–77.244.6% 38.6–49.533.3%
depth 528981.0% 76.5–85.546.0% 40.1–51.634.9%

Accuracy by depth, closed world

By depth, Closed world
Proof depthItemsJevGPT-6 LunaBest constant
depth 030099.3% 98.3–100.098.0% 96.3–99.350.0%
depth 130093.7% 91.0–96.376.7% 72.3–81.050.0%
depth 230092.0% 88.7–94.765.3% 60.0–70.750.0%
depth 330085.0% 81.3–89.357.0% 51.7–63.050.0%
depth 430076.3% 71.7–81.048.3% 43.0–53.750.0%
depth 530089.3% 85.7–92.344.7% 38.7–50.350.0%

What GPT-6 Luna got right and wrong, depth by depth

At each proof depth, GPT-6 Luna's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

93.3%
right on 280 of 300 problems wrong; chance 33% (dashed line)
right

Alan is not young.

Correct answer: false · Model said: false (100.0% sure, probe)

paraphrased
The problem and its proof

Alan is young and green. This makes him cold to others. Eric seems to be round. Even though Gary is big and rough he is sometimes round and kind. Harry is a young, round shaped young man who is also very cold. Truth be told, every big, rough person out there that is green, will also be colored blue. Big people who are red in color are almost always young. If you come across a young, blue person you can count on them being nice to you. People turn green when they are feeling blue. Nice people that are very green and even round shaped will be very young. Any person who can be red and blue at the same time is cold. If you ever meet someone who is both green and blue while being nice, then they'll be big, too.

Stated in the text: "Alan is young."

right

The lion does not visit the bear.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The bald eagle visits the tiger. The bear visits the bald eagle. The lion does not visit the bear. The tiger needs the lion. If someone needs the lion then they do not need the tiger. If someone sees the bear and they see the lion then they do not need the bear.

Stated in the text: "The lion does not visit the bear."

right

Bob is blue.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

paraphrased
The problem and its proof

Alan woke up one day to find he had become round. He was still rough, red, and green, but now he was also big. Bob seems to be round. The young person who is always feeling cold is named Charlie. Dave who is young and nice is also quite big and feels green. Not a single young person is not also kind. Even though young people are rough, they are still very red. Watch out for young people that are kind for they are rough. Nice people with red and rough skin are green with envy. A young person who is big and rough and big is also usually round. When someone is blue, nice and young they will also be as big as the sun.

No proof: neither the statement nor its negation follows.

right

Charlie is round.

Correct answer: true · Model said: true (100.0% sure, probe)

paraphrased
The problem and its proof

Charlie is a round and rough around the edges, and he is also big. Eric is young and green. Even though he is round he is cold outside. Fred is a young, round shaped young man who is also very cold. Gary may be round, but he is also kind. Do not fear big cold people for they are often kind. Someone who is a young age and looks like they are round are also red. Any human who is rough, green, and kind, has a blue personality. Any red sort of person is a nice person. Nice and green people are often found to be big as well.

Stated in the text: "Charlie is round."

Depth 1, open-world task

77.8%
right on 235 of 302 problems wrong; chance 33% (dashed line)
right

The tiger is not kind.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The tiger is nice. The tiger is red. The tiger is round. If someone is round then they are not kind.

  1. The tiger is round.If someone is round then they are not kind.The tiger is not kind.
right

Gary is rough.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

The problem and its proof

Bob is red. Erin is not green. Gary is young. Rough people are green. If someone is young then they are green. All furry people are big. Cold, furry people are young. If someone is red then they are furry. All furry, red people are rough.

No proof: neither the statement nor its negation follows.

wrong

Eric is not round.

Correct answer: false · Model said: unknown (98.9% sure, probe)

paraphrased
The problem and its proof

Alan, who is relatively young, is also pretty big and tends to be cold. No one knows Charlie like me and he is a kind guy, very round in the belly and green as grass. Eric has big, red cheeks with a rough texture. He always wear green sweaters because he is easily cold. That guy Gary sure is nice. Rough people who are red in color and big are usally round in shape. A rather big person is a red person. Young people that are turning blue from being cold will be rough looking. When someone is green and round, they can be rough but kind. People with big smiles and round eyes will have red hair. Is a kind person if feeling cold and looking red then they are usually young. Red people, who are also round and young, are often quite blue as well.

The proof that the statement's negation follows:

  1. Eric is red. + Eric is rough. + Eric is big.If someone is red and rough and big then they are round.Eric is round.
right

Harry is white.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

The problem and its proof

Bob is not smart. Bob is white. Harry is green. Round people are white. White people are blue. If Bob is white and Bob is blue then Bob is round. Round, white people are not quiet. If someone is blue and not white then they are smart. If someone is white then they are not smart. If someone is smart and not quiet then they are green. All smart, round people are not green.

No proof: neither the statement nor its negation follows.

Depth 2, open-world task

60.4%
right on 183 of 303 problems wrong; chance 33% (dashed line)
right

The squirrel is young.

Correct answer: false · Model said: false (84.2% sure, probe)

The problem and its proof

The dog sees the squirrel. The dog visits the rabbit. The rabbit likes the squirrel. The rabbit sees the dog. The squirrel likes the dog. The squirrel sees the dog. The squirrel does not visit the rabbit. If someone sees the dog and the dog visits the rabbit then they visit the dog. If someone visits the dog then they are not young.

The proof that the statement's negation follows:

  1. The squirrel sees the dog. + The dog visits the rabbit.If someone sees the dog and the dog visits the rabbit then they visit the dog.The squirrel visits the dog.
  2. If someone visits the dog then they are not young.The squirrel is not young.
right

The bear eats the mouse.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The bear sees the mouse. The mouse chases the bear. If something is big then it eats the mouse. If something sees the mouse then it is big.

  1. The bear sees the mouse.If something sees the mouse then it is big.The bear is big.
  2. If something is big then it eats the mouse.The bear eats the mouse.
wrong

The lion visits the mouse.

Correct answer: true · Model said: unknown

The problem and its proof

The lion does not need the tiger. The mouse visits the tiger. The squirrel visits the mouse. The tiger eats the lion. If someone visits the mouse then they visit the lion. If someone visits the lion then the lion visits the mouse.

  1. The squirrel visits the mouse.If someone visits the mouse then they visit the lion.The squirrel visits the lion.
  2. If someone visits the lion then the lion visits the mouse.The lion visits the mouse.
wrong

Gary is not blue.

Correct answer: true · Model said: unknown (97.6% sure, probe)

The problem and its proof

Erin is cold. Fiona is cold. Fiona is not quiet. Fiona is rough. Gary is nice. Gary is not quiet. Gary is smart. Smart things are big. Big things are smart. If something is cold and not quiet then it is smart. All big, quiet things are smart. If something is nice and cold then it is not smart. All quiet, big things are not nice. Rough things are not nice. All big things are not blue.

  1. Gary is smart.Smart things are big.Gary is big.
  2. All big things are not blue.Gary is not blue.

Depth 3, open-world task

61.7%
right on 187 of 303 problems wrong; chance 33% (dashed line)
right

Harry is green.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

paraphrased
The problem and its proof

That guy Alan sure is nice. Bob is round but he is also nice and kind. People that know Charlie will tell you he is rough and cold. But he can also be kind and red despite being blue because he is so young. That guy Harry sure is nice. People who are very nice and easily flush red are often wearing green concealer. If you're truly nice, cold, and blue, you're kind, too. Big, kind folks are green ones. Round people who are kind tend to be nice. Young and red people look round. A nice and green man or woman is also red in color.

No proof: neither the statement nor its negation follows.

wrong

Eric is not green.

Correct answer: false · Model said: unknown (98.3% sure, probe)

paraphrased
The problem and its proof

Bob is cold to his ex-wife. She knows he is round and red. He is kind to everyone else. Eric is known for being big and rough, but few know he is also always red with cold. For being so cold, it's good Gary can remain nice. For being so cold, it's good Harry can remain nice. Round people who feel blue and are green in color are often young in age. Rough people who are red in color and big are usally round in shape. Kind red people are green on the inside. A rather big person is a red person. A nice but rough person will also be green and cold. A big and round individual is sure to be a kind individual, too. A person that is red, rough and kind often feels cold.

The proof that the statement's negation follows:

  1. Eric is red. + Eric is rough. + Eric is big.If someone is red and rough and big then they are round.Eric is round.
  2. + Eric is big.If someone is big and round then they are kind.Eric is kind.
  3. + Eric is red.If someone is kind and red then they are green.Eric is green.
wrong

The cow does not see the cat.

Correct answer: unknown · Model said: false (97.6% sure, probe)

The problem and its proof

The bear does not chase the cat. The bear eats the cat. The bear eats the cow. The bear is green. The bear is not nice. The bear sees the dog. The cat is nice. The cat is red. The cat sees the bear. The cow eats the dog. The dog is kind. If someone is kind and they see the dog then the dog chases the cow. If someone eats the bear and the bear chases the dog then the dog eats the cow. If someone sees the bear then they see the cow. If someone sees the cat and the cat sees the bear then they eat the dog. If someone is nice and they eat the cow then the cow does not chase the cat. If someone sees the cow then they see the cat. If someone is green then they see the bear. If someone eats the dog then they are red.

No proof: neither the statement nor its negation follows.

right

Charlie is not young.

Correct answer: true · Model said: true (99.9% sure, probe)

The problem and its proof

Charlie is smart. Fiona is not young. All white, round people are big. If someone is young then they are big. All young, round people are big. If Fiona is white then Fiona is big. All round people are not young. If Charlie is cold then Charlie is round. If someone is smart then they are cold. Smart people are blue.

  1. Charlie is smart.If someone is smart then they are cold.Charlie is cold.
  2. If Charlie is cold then Charlie is round.Charlie is round.
  3. All round people are not young.Charlie is not young.

Depth 4, open-world task

44.6%
right on 135 of 303 problems wrong; chance 33% (dashed line)
right

Anne is young.

Correct answer: true · Model said: true

The problem and its proof

Anne is big. Charlie is big. Charlie is blue. Charlie is cold. Charlie is green. Charlie is nice. Charlie is rough. Charlie is young. Fiona is big. Fiona is green. Gary is big. Gary is cold. Gary is green. Gary is nice. Gary is rough. Gary is young. All rough, young people are nice. All big, cold people are rough. If someone is rough and blue then they are young. Big people are cold. Rough people are blue.

  1. Anne is big.Big people are cold.Anne is cold.
  2. + Anne is big.All big, cold people are rough.Anne is rough.
  3. Rough people are blue.Anne is blue.
  4. If someone is rough and blue then they are young.Anne is young.
wrong

Charlie is not kind.

Correct answer: false · Model said: unknown (100.0% sure, probe)

paraphrased
The problem and its proof

Bob is a round and rough around the edges, and he is also big. Young Charlie was rough in demeanor. He was cold, which made his skin blue despite being big. Gary seems big and rough, but he's actually quite kind. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Folks who are rough and tumble yet gentle and kind are blue. Someone who has young, nice and cold qualities will possess a red quality, too. Anyone feeling cold is bound to act nice. A round person who is on the young side is also likely to treat people in a kind manner. Tom is a rough, young person to know, he is very green at his job but he is very round from his weight. Nice people who feel blue often look green when ill.

The proof that the statement's negation follows:

  1. Charlie is cold.If someone is cold then they are nice.Charlie is nice.
  2. + Charlie is blue.If someone is nice and blue then they are green.Charlie is green.
  3. + Charlie is rough. + Charlie is young.If someone is rough and young and green then they are round.Charlie is round.
  4. + Charlie is young.If someone is round and young then they are kind.Charlie is kind.
right

Charlie is not blue.

Correct answer: true · Model said: true

The problem and its proof

Bob is cold. Bob is quiet. Charlie is white. Dave is quiet. Dave is white. Gary is blue. Gary is quiet. All white, green things are cold. White things are cold. Red, smart things are not blue. All red things are green. Green, white things are quiet. Smart, white things are red. If Charlie is quiet then Charlie is red. If something is blue and cold then it is smart. All white, cold things are smart.

  1. Charlie is white.White things are cold.Charlie is cold.
  2. + Charlie is white.All white, cold things are smart.Charlie is smart.
  3. + Charlie is white.Smart, white things are red.Charlie is red.
  4. Red, smart things are not blue.Charlie is not blue.
wrong

Bob is nice.

Correct answer: true · Model said: unknown (100.0% sure, probe)

The problem and its proof

Anne is furry. Anne is red. Anne is smart. Bob is red. Dave is blue. Dave is kind. Erin is white. Furry people are kind. Furry, blue people are kind. All red people are furry. All nice, red people are smart. Kind, nice people are smart. All furry people are kind. White people are kind. If someone is white and red then they are nice. If Bob is kind then Bob is white.

  1. Bob is red.All red people are furry.Bob is furry.
  2. Furry people are kind.Bob is kind.
  3. If Bob is kind then Bob is white.Bob is white.
  4. + Bob is red.If someone is white and red then they are nice.Bob is nice.

Depth 5, open-world task

46.0%
right on 133 of 289 problems wrong; chance 33% (dashed line)
wrong

Anne is not big.

Correct answer: false · Model said: unknown (100.0% sure, probe)

The problem and its proof

Anne is blue. Anne is nice. Bob is kind. Bob is nice. Fiona is blue. Fiona is furry. Fiona is kind. Fiona is nice. Fiona is smart. Gary is blue. Gary is furry. Gary is smart. Big, furry things are green. Kind, nice things are big. All smart, green things are kind. All nice things are furry. If Anne is kind and Anne is big then Anne is green. If Fiona is blue then Fiona is green. Furry, green things are smart. Furry things are green.

The proof that the statement's negation follows:

  1. Anne is nice.All nice things are furry.Anne is furry.
  2. Furry things are green.Anne is green.
  3. Furry, green things are smart.Anne is smart.
  4. All smart, green things are kind.Anne is kind.
  5. + Anne is nice.Kind, nice things are big.Anne is big.
right

The cat does not see the cat.

Correct answer: false · Model said: false (95.2% sure, probe)

The problem and its proof

The cat does not chase the rabbit. The cat is cold. The cat is red. The cat sees the rabbit. The cat sees the tiger. The mouse is big. The mouse is red. The rabbit does not chase the cat. The rabbit likes the cat. The rabbit does not like the mouse. The tiger likes the mouse. If something likes the mouse then it chases the cat. If something sees the mouse then it chases the cat. If something chases the mouse then the mouse does not like the rabbit. If something chases the cat then it sees the cat. If the tiger sees the cat then the cat sees the mouse.

The proof that the statement's negation follows:

  1. The tiger likes the mouse.If something likes the mouse then it chases the cat.The tiger chases the cat.
  2. If something chases the cat then it sees the cat.The tiger sees the cat.
  3. If the tiger sees the cat then the cat sees the mouse.The cat sees the mouse.
  4. If something sees the mouse then it chases the cat.The cat chases the cat.
  5. If something chases the cat then it sees the cat.The cat sees the cat.
right

The cow is not big.

Correct answer: true · Model said: true

The problem and its proof

The cat is not round. The cow is blue. The cow is kind. The cow likes the cat. The cow sees the rabbit. The rabbit chases the cow. The rabbit is nice. The tiger does not chase the cow. The tiger does not like the rabbit. The tiger sees the rabbit. If something chases the cow then the cow chases the cat. If something chases the rabbit then it is not big. If something likes the cow then the cow is nice. If something is kind then it sees the rabbit. If something chases the cat then the cat likes the cow. If the rabbit chases the tiger then the rabbit does not see the cow. If something is nice and it sees the rabbit then it chases the rabbit. If something is big then it is round.

  1. The rabbit chases the cow.If something chases the cow then the cow chases the cat.The cow chases the cat.
  2. If something chases the cat then the cat likes the cow.The cat likes the cow.
  3. If something likes the cow then the cow is nice.The cow is nice.
  4. + The cow sees the rabbit.If something is nice and it sees the rabbit then it chases the rabbit.The cow chases the rabbit.
  5. If something chases the rabbit then it is not big.The cow is not big.
wrong

Anne is quiet.

Correct answer: false · Model said: unknown (100.0% sure, probe)

The problem and its proof

Anne is smart. Charlie is not blue. Charlie is quiet. Erin is not nice. Erin is round. Harry is green. Harry is young. Smart things are green. If Anne is round then Anne is young. If something is blue and green then it is not quiet. If something is green and smart then it is round. If something is young then it is blue. Smart things are nice. If Harry is quiet and Harry is not smart then Harry is young.

The proof that the statement's negation follows:

  1. Anne is smart.Smart things are green.Anne is green.
  2. + Anne is smart.If something is green and smart then it is round.Anne is round.
  3. If Anne is round then Anne is young.Anne is young.
  4. If something is young then it is blue.Anne is blue.
  5. If something is blue and green then it is not quiet.Anne is not quiet.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

98.0%
right on 294 of 300 problems wrong; chance 50% (dashed line)
right

Charlie is not red.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Charlie is red. If something is nice and not young then it is blue. If Charlie is red then Charlie is kind. All kind things are big.

Stated in the text: "Charlie is red."

right

The rabbit does not eat the bear.

Correct answer: false · Model said: false (99.9% sure, probe)

The problem and its proof

The bear is rough. The rabbit eats the bear. The squirrel visits the rabbit. If someone eats the bear then they need the bear. If someone needs the squirrel and they are not big then they need the bear. If someone eats the squirrel and they are not big then the squirrel is blue. If someone needs the rabbit then they visit the rabbit. If someone needs the bear then the bear visits the rabbit. If someone eats the squirrel then they are blue. If someone needs the rabbit then they eat the bear. If someone eats the squirrel and the squirrel is not rough then they are cold.

Stated in the text: "The rabbit eats the bear."

right

The rabbit is not cold.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The dog needs the rabbit. The dog visits the rabbit. The rabbit eats the dog. If something visits the dog then it is young. Round things are young. If something needs the dog and it visits the dog then the dog eats the rabbit. If something is nice then it visits the dog. If something visits the rabbit and the rabbit is young then it eats the rabbit. If something is young then it eats the rabbit. If something is cold and it eats the dog then it needs the rabbit. If something visits the rabbit and the rabbit eats the dog then it is round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The squirrel chases the cow.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

The cow chases the lion. The cow likes the rabbit. The cow sees the lion. The cow sees the squirrel. The lion chases the squirrel. The lion likes the cow. The lion likes the squirrel. The rabbit chases the lion. The rabbit likes the lion. The squirrel is big. The squirrel is rough. The squirrel likes the lion. The squirrel sees the lion. The squirrel sees the rabbit. If something sees the rabbit and the rabbit sees the cow then the rabbit chases the cow. If something is rough then it likes the rabbit. If something is cold and it chases the lion then the lion likes the squirrel. If something likes the cow then it is rough. If the lion is big and the lion chases the rabbit then the rabbit chases the cow. If something chases the rabbit then it is rough. If the lion sees the rabbit and the lion chases the cow then the cow is cold. If the lion is rough and the lion likes the rabbit then the rabbit likes the cow. If something is red and it sees the cow then the cow likes the rabbit.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

76.7%
right on 230 of 300 problems wrong; chance 50% (dashed line)
right

The rabbit is not green.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

The rabbit is red. If the rabbit is not blue then the rabbit is green. Green things are round. Green, red things are cold.

  1. If the rabbit is not blue then the rabbit is green.The rabbit is green.
right

Fred is round.

Correct answer: false · Model said: false (100.0% sure, probe)

paraphrased
The problem and its proof

Charlie is a red, rough, green and blue man. For being so young, Dave is nice. He has rough, red hands from working all day and it makes him feel blue. Gary seems to be round. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Some cold people can be nice at times,and red at at other times. Anyone who is both red and green must be as cold as Christmas. I've noticed that nice people who have rough, green skin have a tendancy to be round. Most big and cold people are also rough. A big round young person is often blue. When a person is blue, rough and kind then it means they also are cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Harry is furry.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

Bob is nice. Bob is young. Charlie is kind. Charlie is smart. Dave is kind. Dave is white. Harry is nice. All young people are white. All young people are furry. All nice people are furry. If someone is nice then they are blue. Blue people are nice. If someone is smart and kind then they are nice. If someone is white then they are kind. All kind, furry people are smart. All kind, white people are young.

  1. Harry is nice.All nice people are furry.Harry is furry.
wrong

Eric is nice.

Correct answer: true · Model said: false (41.6% sure, probe)

paraphrased
The problem and its proof

Charlie can be rough and cold, but he is also green, an avid gardener. Eric is green, but often kind, even when he is blue and cold. The young person who is always feeling cold is named Fred. Gary is kind and nice and looks green. If a kind person is green and a bit blue then they are also nice. A blue colored person who is nice is a red person. A person who is kind and rough and blue is young. If someone is cold and blue, you will find that they are also quite young. A person who is green, blue, and red in color will be round shaped.

  1. Eric is green. + Eric is kind. + Eric is blue.If someone is green and kind and blue then they are nice.Eric is nice.

Depth 2, closed-world task

65.3%
right on 196 of 300 problems wrong; chance 50% (dashed line)
wrong

The lion does not see the bear.

Correct answer: false · Model said: true (92.2% sure, probe)

The problem and its proof

The bear is rough. The lion chases the bear. If someone chases the lion and the lion is not young then the lion chases the bear. If someone sees the bear then the bear sees the lion. If the bear chases the lion then the bear is blue. If someone chases the bear then they eat the bear. If someone eats the bear then they see the bear. If the bear is red then the bear chases the lion. If someone chases the bear and they are young then the bear chases the lion. If someone eats the bear then the bear is big.

  1. The lion chases the bear.If someone chases the bear then they eat the bear.The lion eats the bear.
  2. If someone eats the bear then they see the bear.The lion sees the bear.
right

The mouse does not chase the cow.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

The cow eats the lion. The lion visits the mouse. The mouse is big. If something chases the cow then the cow chases the mouse. If something chases the mouse then the mouse chases the cow. If something visits the mouse then it chases the mouse.

  1. The lion visits the mouse.If something visits the mouse then it chases the mouse.The lion chases the mouse.
  2. If something chases the mouse then the mouse chases the cow.The mouse chases the cow.
right

The cow is young.

Correct answer: true · Model said: true (99.7% sure, probe)

The problem and its proof

The bald eagle chases the tiger. The bald eagle needs the tiger. The bald eagle sees the cow. The bald eagle sees the tiger. The cow chases the tiger. The cow is kind. The cow is red. The cow sees the bald eagle. The cow sees the tiger. The tiger chases the bald eagle. The tiger chases the cow. The tiger is blue. The tiger is kind. The tiger is red. The tiger needs the bald eagle. Young, kind people are green. If someone needs the tiger and the tiger needs the cow then the cow is young. If someone chases the tiger and they chase the bald eagle then the bald eagle sees the tiger. If someone sees the tiger then the tiger needs the cow. If someone needs the bald eagle and the bald eagle chases the tiger then the bald eagle sees the cow. If someone is red and they need the tiger then they are blue.

  1. The bald eagle sees the tiger.If someone sees the tiger then the tiger needs the cow.The tiger needs the cow.
  2. + The bald eagle needs the tiger.If someone needs the tiger and the tiger needs the cow then the cow is young.The cow is young.
right

Eric is nice.

Correct answer: true · Model said: true (99.9% sure, probe)

paraphrased
The problem and its proof

For being so cold, it's good Alan can remain nice. Eric is young and green. This makes him cold to others. Fred is a kind person and he is also often cold. That guy Harry sure is nice. All those who are nice and big are red. If you come across a young, blue person you can count on them being nice to you. People who are said to be big and nice are round. People who are young are also blue. A kind person will certainly be young.

  1. Eric is young.If someone is young then they are blue.Eric is blue.
  2. + Eric is young.If someone is blue and young then they are nice.Eric is nice.

Depth 3, closed-world task

57.0%
right on 171 of 300 problems wrong; chance 50% (dashed line)
right

Fiona is blue.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Anne is red. Erin is young. Fiona is kind. All white things are red. If something is quiet and white then it is kind. If something is quiet then it is red. If something is red then it is rough. All rough things are blue. All young things are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Gary is not cold.

Correct answer: false · Model said: true (99.9% sure, probe)

paraphrased
The problem and its proof

Red Charlie is still young while also very blue and green. Dave seems to be round. Eric, who is relatively young, is also pretty big and tends to be cold. Gary who is young and nice is also quite big and feels green. Not a single young person is not also kind. When you meet someone who is nice and kind, they wont feel blue. All those who are nice and big are red. A young person who is big and rough and big is also usually round. Young and red people look round. If a person is blue and red and big, then that person is cold.

  1. Gary is young.If someone is young then they are kind.Gary is kind.
  2. + Gary is nice.If someone is kind and nice then they are blue.Gary is blue.
  3. + Gary is nice. + Gary is big.If someone is nice and big then they are red.Gary is red.
  4. + Gary is big.If someone is blue and big and red then they are cold.Gary is cold.
wrong

Eric is nice.

Correct answer: true · Model said: false (100.0% sure, probe)

paraphrased
The problem and its proof

Charlie is green and cold too. Eric is a big guy who is colored red from a sunburn. He also has a cold. No one knows Fred like me and he is a kind guy, very round in the belly and green as grass. People here that are both big and kind while being cold are always red. Red people are almost always rough. A person who is cold and blue is nice. If you're truly nice, cold, and blue, you're kind, too. A big, round person who is kind is also red. Rough, cold people are blue. Nice people with red and rough skin are green with envy.

  1. Eric is red.If someone is red then they are rough.Eric is rough.
  2. + Eric is cold.If someone is rough and cold then they are blue.Eric is blue.
  3. + Eric is cold.If someone is cold and blue then they are nice.Eric is nice.
right

Gary is not green.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Bob is quiet. Bob is smart. Dave is rough. Erin is big. Erin is rough. Gary is round. Gary is white. Smart people are green. All rough people are smart. If someone is round then they are rough.

  1. Gary is round.If someone is round then they are rough.Gary is rough.
  2. All rough people are smart.Gary is smart.
  3. Smart people are green.Gary is green.

Depth 4, closed-world task

48.3%
right on 145 of 300 problems wrong; chance 50% (dashed line)
right

Fiona is not smart.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

Bob is cold. Dave is big. Dave is furry. Dave is rough. Dave is round. Dave is smart. Fiona is round. If someone is smart then they are big. Furry people are smart. All furry people are smart. Smart people are furry. If someone is cold then they are furry. All smart people are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Bob is not nice.

Correct answer: true · Model said: false (99.0% sure, probe)

The problem and its proof

Bob is big. Charlie is blue. Gary is nice. Harry is kind. If someone is blue and big then they are cold. Quiet people are nice. Blue people are cold. All cold people are green. All cold, green people are nice. If someone is nice then they are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The lion is young.

Correct answer: true · Model said: false

The problem and its proof

The bear eats the cow. The bear likes the lion. The bear likes the mouse. The cow chases the bear. The cow chases the mouse. The cow eats the bear. The cow is nice. The lion chases the cow. The lion chases the mouse. The lion likes the bear. The mouse chases the cow. The mouse chases the lion. If someone is young then they are round. If someone eats the lion then they are young. If someone eats the bear then they eat the lion. If someone is nice and they chase the lion then they like the bear. If someone eats the lion then the lion eats the bear.

  1. The cow eats the bear.If someone eats the bear then they eat the lion.The cow eats the lion.
  2. If someone eats the lion then the lion eats the bear.The lion eats the bear.
  3. If someone eats the bear then they eat the lion.The lion eats the lion.
  4. If someone eats the lion then they are young.The lion is young.
right

Charlie is young.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Charlie is rough. Dave is white. Erin is blue. Harry is kind. If something is white then it is cold. Big things are white. If something is cold then it is young. If something is young then it is blue. If something is rough then it is blue. White, cold things are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 5, closed-world task

44.7%
right on 134 of 300 problems wrong; chance 50% (dashed line)
right

Erin is rough.

Correct answer: false · Model said: false (94.4% sure, probe)

The problem and its proof

Dave is blue. Dave is rough. Dave is young. Erin is blue. Erin is round. Erin is young. Fiona is rough. Gary is green. Gary is nice. Gary is young. Rough things are round. Green, nice things are quiet. Round, rough things are green. All quiet things are green. All green things are nice. All quiet things are rough. All rough, nice things are quiet. All quiet, green things are rough. If something is nice and quiet then it is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The rabbit likes the bear.

Correct answer: true · Model said: false (100.0% sure, probe)

The problem and its proof

The bear is red. The rabbit is red. The rabbit likes the tiger. The squirrel is cold. The tiger eats the bear. The tiger is big. The tiger likes the rabbit. All cold people are kind. If someone is kind then they like the bear. If someone chases the rabbit and they are kind then the rabbit is kind. If someone likes the bear then they chase the rabbit. If someone chases the tiger and the tiger is red then the tiger chases the bear. If the squirrel is kind and the squirrel likes the rabbit then the rabbit is green. If the tiger eats the bear and the bear chases the rabbit then the bear is green.

  1. The squirrel is cold.All cold people are kind.The squirrel is kind.
  2. If someone is kind then they like the bear.The squirrel likes the bear.
  3. If someone likes the bear then they chase the rabbit.The squirrel chases the rabbit.
  4. If someone chases the rabbit and they are kind then the rabbit is kind.The rabbit is kind.
  5. If someone is kind then they like the bear.The rabbit likes the bear.
wrong

The tiger does not see the tiger.

Correct answer: false · Model said: true (100.0% sure, probe)

The problem and its proof

The lion visits the mouse. The mouse sees the lion. The mouse sees the rabbit. The mouse visits the rabbit. The rabbit chases the lion. The rabbit chases the tiger. The rabbit is nice. The rabbit visits the lion. The tiger chases the lion. The tiger chases the mouse. If someone is green then they visit the tiger. If someone is nice then they visit the mouse. If someone sees the rabbit then they are kind. If someone sees the tiger then the tiger sees the rabbit. If someone is kind then they see the tiger. If the lion chases the mouse then the mouse sees the rabbit. If the rabbit does not chase the tiger then the tiger is cold.

  1. The mouse sees the rabbit.If someone sees the rabbit then they are kind.The mouse is kind.
  2. If someone is kind then they see the tiger.The mouse sees the tiger.
  3. If someone sees the tiger then the tiger sees the rabbit.The tiger sees the rabbit.
  4. If someone sees the rabbit then they are kind.The tiger is kind.
  5. If someone is kind then they see the tiger.The tiger sees the tiger.
right

Fiona is rough.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Bob is cold. Dave is big. Dave is furry. Dave is rough. Dave is round. Dave is smart. Fiona is round. If someone is smart then they are big. Furry people are smart. All furry people are smart. Smart people are furry. If someone is cold then they are furry. All smart people are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

What this means for the OpenAI Decisions API

At OpenAI DevDay on 29 September 2026, OpenAI announced the Decisions API, in limited preview with broader availability "in the coming days". OpenAI says it "uses a version of GPT-6 Luna": the model picks among answers the developer defines in advance, to classify, route or choose an agent's next action (The Decoder, Axios).

That is the shape of task we ran: GPT-6 Luna with reasoning off, one request per problem, and a fixed list of answers enforced with a strict JSON schema. So these results preview the accuracy a Luna-based decision model gets on multi-step reasoning. GPT-6 Luna answered 64.1% of open-world problems and 65.0% of closed-world ones correctly. On the open-world task it scored 93.3% when the answer is stated outright (depth 0) and 46.0% when it takes five chained inferences, where Jev scored 81.0% (+34.9 points on the same items). If your decisions need several steps of reasoning, the depth results above are the ones to read.

These are not results for the Decisions API itself. OpenAI uses a specialized version of Luna, and its accuracy may differ. As of 1 October 2026 OpenAI had published no Decisions API documentation. We will benchmark the API itself when we can get access.

On speed, OpenAI's own chart shows 150 ms per decision for the Decisions API, against 1.6 s for Luna through the regular API (The Decoder). Those are OpenAI's figures, not our measurements.

Confidence and log-probabilities: things to know

Where each one is better

Open world, Jev vs GPT-6 Luna. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GPT-6 Luna is clearly ahead on unknown answers (+9.2 points).

Closed world, Jev vs GPT-6 Luna. Jev is clearly ahead on depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GPT-6 Luna is not clearly ahead on any depth or answer class. Too close to call: depth 0.

A slice counts as clearly ahead when its paired 95% interval excludes zero.

True, false and unknown, open world

By correct answer, Open world
Correct answerItemsJevGPT-6 LunaBest constant
false60588.3% 85.5–90.950.9% 46.8–54.9one answer
true60588.8% 86.1–91.158.3% 54.0–62.1one answer
unknown59074.2% 70.7–77.683.4% 80.5–86.6one answer

True, false and unknown, closed world

By correct answer, Closed world
Correct answerItemsJevGPT-6 LunaBest constant
false90088.9% 86.9–90.963.8% 60.6–66.9one answer
true90089.7% 87.7–91.666.2% 63.1–69.2one answer

Paraphrased vs templated rules, open world

By wording, Open world
WordingItemsJevGPT-6 LunaBest constant
templated1,29886.1% 84.2–87.967.6% 64.8–70.234.5%
paraphrased (NatLang)50278.1% 74.5–82.154.8% 50.4–59.435.9%

Paraphrased vs templated rules, closed world

By wording, Closed world
WordingItemsJevGPT-6 LunaBest constant
templated1,30694.0% 92.8–95.367.3% 64.8–69.950.2%
paraphrased (NatLang)49476.7% 73.1–80.458.9% 54.7–63.450.6%

Answer by answer: Jev and GPT-6 Luna on the same problems

Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
GPT-6 Luna (reasoning off), open-world task
Correct ↓ · said →truefalseunknown
true3533249
false7308290
unknown4553492

Jev said true on 37% of problems; the correct answer is true on 34%. GPT-6 Luna (reasoning off) said unknown on 57% of problems; the correct answer is unknown on 33%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
GPT-6 Luna (reasoning off), closed-world task
Correct ↓ · said →truefalse
true596304
false326574

Jev said true on 50% of problems; the correct answer is true on 50%. GPT-6 Luna (reasoning off) said true on 51% of problems; the correct answer is true on 50%.

Where GPT-6 Luna errs

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

GPT-6 Luna, open-world task

GPT-6 Luna (reasoning off), open-world task
Correct ↓ · said →truefalseunknownRecall
true353324958.3%
false730829050.9%
unknown455349283.4%

GPT-6 Luna answered false 20.2%, true 22.5%, unknown 57.3% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

GPT-6 Luna, closed-world task

GPT-6 Luna (reasoning off), closed-world task
Correct ↓ · said →truefalseRecall
true59630466.2%
false32657463.8%

GPT-6 Luna answered false 48.8%, true 51.2% of the time; the correct answers are false 50.0%, true 50.0%.

GPT-6 Luna at depth 0 and depth 5

GPT-6 Luna (reasoning off), open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9604
false28513
unknown1099
GPT-6 Luna (reasoning off), open-world task, depth 5
Correct ↓ · said →truefalseunknown
true36065
false03566
unknown131262

Open-world task. At depth 0, GPT-6 Luna was right on 93% of problems (the green diagonal); at depth 5, on 46%. The answer whose share moved most is unknown: 39% of its answers at depth 0, 67% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
GPT-6 Luna (reasoning off), closed-world task, depth 0
Correct ↓ · said →truefalse
true1482
false4146
GPT-6 Luna (reasoning off), closed-world task, depth 5
Correct ↓ · said →truefalse
true7377
false8961

Closed-world task. At depth 0, GPT-6 Luna was right on 98% of problems (the green diagonal); at depth 5, on 45%. The answer whose share moved most is true: 51% of its answers at depth 0, 54% at depth 5, against 50% and 50% of the correct answers.

GPT-6 Luna across every breakdown

For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownWeakest · strongest
Proof depthdepth 4 44.6% · depth 0 93.3% (48.8 points)
Gold answerfalse 50.9% · unknown 83.4% (32.5 points)
Theory kindattribute 62.5% · relation 66.3% (3.8 points)
Negation in the theorywithout negation 61.2% · with negation 67.1% (5.9 points)
Negated statementplain statement 62.3% · negated statement 65.8% (3.5 points)
Question strategyinv-proof 50.9% · inv-random 100.0% (49.1 points)
Paraphrased rulesparaphrased (NatLang) 54.8% · templated 67.6% (12.9 points)
Theory's deepest proof4 40.7% · 0 87.3% (46.6 points)
Theory length110+ 52.6% · 0-49 91.5% (38.9 points)
Number of rules3-5 59.1% · 0-2 90.2% (31.1 points)
Number of facts13+ 56.6% · 0-3 81.4% (24.8 points)
Proof size5+ 33.6% · 0-1 85.2% (51.6 points)

Exactly what GPT-6 Luna was asked

One open-world problem at depth 0, as each model received it. The correct answer is unknown.

The bear is green. Green things are not young.

Statement: The bear is not big.
Jevright answered unknown (100.0%); correct answer unknown

POST https://api.typesafe.ai/v1/systemone

Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "jev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "jev-1.13.0",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 1,
      "probabilities": {
        "false": 0,
        "true": 0,
        "unknown": 1
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 460,
    "output_tokens": 38
  }
}
GPT-6 Lunaright answered unknown; correct answer unknown

POST https://api.openai.com/v1/chat/completions

GPT-6 Luna with reasoning off (reasoning_effort none, its lowest setting), as a direct classifier. One chat message carries the problem, the question and every option with its meaning; the strict JSON schema only allows one of the options as the answer.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

Options:
- true: The statement follows from the facts and rules.
- false: The negation of the statement follows from the facts and rules.
- unknown: Neither the statement nor its negation follows from the facts and rules.

Reply with a JSON object {"answer": <option>} where <option> is exactly one of: true, false, unknown.

The answers it can give

true
false
unknown
The full request and response, exactly as sent and recorded

Request

{
  "model": "gpt-6-luna",
  "max_completion_tokens": 1024,
  "reasoning_effort": "none",
  "messages": [
    {
      "role": "user",
      "content": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.\n\nReply with a JSON object {\"answer\": <option>} where <option> is exactly one of: true, false, unknown."
    }
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "decision",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "answer": {
            "type": "string",
            "enum": [
              "true",
              "false",
              "unknown"
            ]
          }
        },
        "required": [
          "answer"
        ],
        "additionalProperties": false
      }
    }
  }
}

Response (the saved record)

{
  "model": "gpt-6-luna",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "raw": null,
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 221,
    "output_tokens": 11,
    "stop_reason": "stop"
  }
}

Time and money

GPT-6 Luna's median open-world decision took 825 ms against Jev's 189 ms, and 1 of its 3,600 decisions took ten seconds or more (the slowest 10 s). At list price GPT-6 Luna cost $37 per million decisions against Jev's $23: 1.6 times as much per decision and 2.1 times as much per correct decision.

Every decision's time

  • Jev
  • GPT-6 Luna(reasoning off)
05001,0001,5002,0000.1 s0.3 s1 s3 s10 s30 s100 stime per decision (log scale)1 GPT-6 Luna decisions ≥ 10 sJev median 179 msGPT-6 Luna median 875 ms

Average time per decision, by proof depth

Open world: true, false or unknown

0 s1 s2 s012345proof depth (inference steps)207ms202ms209ms203ms207ms209ms894ms890ms902ms920ms876ms894ms

Closed world: true or false

0 s1 s2 s012345proof depth (inference steps)177ms177ms175ms174ms174ms178ms993ms1.0s974ms1.0s990ms981ms
ModelTaskDepth 0Depth 1Depth 2Depth 3Depth 4Depth 5
JevOpen world207 ms202 ms209 ms203 ms207 ms209 ms
JevClosed world177 ms177 ms175 ms174 ms174 ms178 ms
GPT-6 LunaOpen world894 ms890 ms902 ms920 ms876 ms894 ms
GPT-6 LunaClosed world993 ms1.0 s974 ms1.0 s990 ms981 ms

List-price cost per million decisions, by proof depth

Open world: true, false or unknown

$0$10$20$30$40012345proof depth (inference steps)$23$23$23$24$24$24$36$37$37$38$39$39

Closed world: true or false

$0$10$20$30$40012345proof depth (inference steps)$22$22$22$23$23$23$34$35$35$36$37$37
ModelTaskDepth 0Depth 1Depth 2Depth 3Depth 4Depth 5
JevOpen world$23$23$23$24$24$24
JevClosed world$22$22$22$23$23$23
GPT-6 LunaOpen world$36$37$37$38$39$39
GPT-6 LunaClosed world$34$35$35$36$37$37

Times are wall time per request, one request at a time. Costs are list prices times the tokens each API reported. Cost per decision for every hosted model · Speed, size and memory.

Consistency and calibration

Asked every open world question a second time, GPT-6 Luna gave the same answer on 89.7% of 1,800 items (185 changed; Gwet's AC1 0.855, interval 0.837 to 0.876). Accuracy was 64.1% the first time and 62.8% the second.

Asked every closed world question a second time, GPT-6 Luna gave the same answer on 90.6% of 1,800 items (169 changed; Gwet's AC1 0.812, interval 0.784 to 0.838). Accuracy was 65.0% the first time and 64.8% the second.

We asked GPT-6 Luna, like every model, for its answer only, so no confidence was recorded and its calibration is not measured here. A separate probe read its log-probabilities: see its calibration on the OpenAI Decisions API preview page.

Size, hardware and speed

ModelKindParametersRan onMedian time per decisionMemory peak
Jevhosted decision modelundisclosedvendor's servers189 msnot applicable (hosted)
GPT-6 LunaLLM classifierundisclosedvendor's servers825 msnot applicable (hosted)

Times are open-world, one request at a time. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.

GPT-6 Luna

Parameters
undisclosed
Architecture
undisclosed
Weights
not available (hosted)
Where it ran
OpenAI Chat Completions API, reasoning_effort none, strict JSON schema
Maker's stated hardware
not applicable (hosted)
Price
$0.10 input / $0.50 output per million tokens
Time per decision, open world
median 825 ms, 90th percentile 1,165 ms, over the network
Time per decision, closed world
median 929 ms, 90th percentile 1,268 ms, over the network
Source
developers.openai.com/api/docs/models/gpt-6-luna

The models

Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

GPT-6 Luna GPT-6 Luna is a hosted large language model made by OpenAI. We used it as a classifier: one request per item through the Chat Completions API, with reasoning effort set to none (its lowest setting) and a strict JSON schema that allows only the task's options. Its size and architecture are not published.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.