Skip to main content
HardDecisions

Comparison · OpenAI Decisions API vs Jev

Jev vs the OpenAI Decisions API: a decision model against OpenAI's decision endpoint

OpenAI's Decisions API (POST /v1/decisions, model gpt-6-luna) answers a question with a choice, probabilities and a confidence. Jev and the Decisions API answered the same 3,600 problems, and GPT-6 Luna through the chat endpoint is shown beside them.

GPT-6 Luna ran through OpenAI's chat endpoint with reasoning effort set to none, its lowest setting, under the same one-request protocol as every model here: the same question, the same options, one answer per item. A different reasoning setting would be a different model, scored separately.

The Decisions API entry is gpt-6-luna through OpenAI's dedicated endpoint, sent the benchmark's identical instructions and option descriptions. The GPT-6 Luna entry is the same model name through the chat endpoint with reasoning off. We did not test other wordings of the instructions. We do not know why the two endpoints differ. The OpenAI Decisions API, as we tested it.

Overall: Jev, OpenAI Decisions API (GPT-6 Luna) and GPT-6 Luna

Open world: true, false or unknown

  • Jev82.2 to 85.5 (95%), 1,800 items83.8%
  • OpenAI Decisions API (GPT-6 Luna)51.6 to 56.4 (95%), 1,800 items54.0%
  • GPT-6 Luna61.7 to 66.2 (95%), 1,800 items64.1%

Jev minus OpenAI Decisions API (GPT-6 Luna): +29.8 points on the same 1,800 items, 95% interval +27.1 to +32.5: Jev is clearly ahead.

Jev minus GPT-6 Luna: +19.8 points on the same 1,800 items, 95% interval +17.2 to +22.4: Jev is clearly ahead.

Closed world: true or false

  • Jev87.8 to 90.7 (95%), 1,800 items89.3%
  • OpenAI Decisions API (GPT-6 Luna)54.1 to 58.6 (95%), 1,800 items56.3%
  • GPT-6 Luna62.8 to 67.4 (95%), 1,800 items65.0%

Jev minus OpenAI Decisions API (GPT-6 Luna): +33.0 points on the same 1,800 items, 95% interval +30.4 to +35.3: Jev is clearly ahead.

Jev minus GPT-6 Luna: +24.3 points on the same 1,800 items, 95% interval +21.9 to +26.6: Jev is clearly ahead.

Jev vs OpenAI Decisions API (GPT-6 Luna) and GPT-6 Luna by proof depth

Each point is about 300 problems that need that many inference steps, with its 95% interval.

Open world: true, false or unknown
  • Jev
  • OpenAI Decisions API (GPT-6 Luna)
  • GPT-6 Luna(reasoning off)
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
  • OpenAI Decisions API (GPT-6 Luna)
  • GPT-6 Luna(reasoning off)
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Jev minus OpenAI Decisions API (GPT-6 Luna) by depth, open world

depth 0300 items
+8.0
depth 1302 items
+26.5
depth 2303 items
+33.3
depth 3303 items
+32.0
depth 4303 items
+36.6
depth 5289 items
+42.9

Jev minus OpenAI Decisions API (GPT-6 Luna) by depth, closed world

depth 0300 items
+17.3
depth 1300 items
+28.0
depth 2300 items
+37.3
depth 3300 items
+33.7
depth 4300 items
+30.3
depth 5300 items
+51.3

Jev minus GPT-6 Luna by depth, open world

depth 0300 items
+4.3
depth 1302 items
+9.9
depth 2303 items
+24.4
depth 3303 items
+18.2
depth 4303 items
+27.4
depth 5289 items
+34.9

Jev minus GPT-6 Luna by depth, closed world

depth 0300 items
+1.3
depth 1300 items
+17.0
depth 2300 items
+26.7
depth 3300 items
+28.0
depth 4300 items
+28.0
depth 5300 items
+44.7

Paired differences: positive means Jev was more accurate on the same items.

Accuracy by depth, open world

By depth, Open world
Proof depthItemsJevOpenAI Decisions API (GPT-6 Luna)GPT-6 LunaBest constant
depth 030097.7% 96.0–99.389.7% 86.0–93.393.3% 90.3–96.033.3%
depth 130287.7% 83.8–91.461.3% 56.0–66.277.8% 73.2–82.833.4%
depth 230384.8% 80.2–88.451.5% 45.5–57.160.4% 54.5–66.033.3%
depth 330379.9% 75.2–84.547.9% 42.2–54.161.7% 56.8–67.733.3%
depth 430371.9% 66.7–77.235.3% 29.7–40.644.6% 38.6–49.533.3%
depth 528981.0% 76.5–85.538.1% 32.2–43.346.0% 40.1–51.634.9%

Accuracy by depth, closed world

By depth, Closed world
Proof depthItemsJevOpenAI Decisions API (GPT-6 Luna)GPT-6 LunaBest constant
depth 030099.3% 98.3–100.082.0% 77.7–86.398.0% 96.3–99.350.0%
depth 130093.7% 91.0–96.365.7% 60.7–71.076.7% 72.3–81.050.0%
depth 230092.0% 88.7–94.754.7% 48.7–60.065.3% 60.0–70.750.0%
depth 330085.0% 81.3–89.351.3% 45.3–56.757.0% 51.7–63.050.0%
depth 430076.3% 71.7–81.046.0% 40.3–51.748.3% 43.0–53.750.0%
depth 530089.3% 85.7–92.338.0% 32.7–43.344.7% 38.7–50.350.0%

What OpenAI Decisions API (GPT-6 Luna) got right and wrong, depth by depth

At each proof depth, OpenAI Decisions API (GPT-6 Luna)'s accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

89.7%
right on 269 of 300 problems wrong; chance 33% (dashed line)
right

Anne is kind.

Correct answer: false · Model said: false (71.0% sure)

The problem and its proof

Anne is big. Anne is not kind. Anne is rough. Dave is nice. Fiona is kind. Fiona is not quiet. Fiona is not smart. Harry is not kind. Harry is rough. Harry is not smart. If Anne is quiet then Anne is not kind. Quiet things are rough. Nice, quiet things are rough. Nice things are kind. All big things are green. All kind things are green. If something is green then it is big. If something is kind and not rough then it is not big.

Stated in the text: "Anne is not kind."

right

Charlie is cold.

Correct answer: unknown · Model said: unknown (100.0% sure)

The problem and its proof

Bob is cold. Bob is not nice. Bob is not quiet. Charlie is blue. Charlie is round. Gary is rough. Harry is cold. Rough things are blue. Red, quiet things are blue. All rough, blue things are nice. If something is quiet and round then it is not nice. If Harry is red then Harry is rough. If something is nice then it is round. If something is round then it is red. Red things are not quiet.

No proof: neither the statement nor its negation follows.

right

Charlie is cold.

Correct answer: false · Model said: false (74.0% sure)

The problem and its proof

Charlie is not cold. Harry is green. Kind things are green. If something is big then it is not green. If something is cold and red then it is green. All green things are kind. Kind, quiet things are nice. All green things are cold. Nice things are quiet. If Harry is cold then Harry is red.

Stated in the text: "Charlie is not cold."

right

Harry is blue.

Correct answer: unknown · Model said: unknown (100.0% sure)

paraphrased
The problem and its proof

Charlie rather resembles the rainbow, as he is green, red and blue. Charlie is also said to be very kind. Thankfully Dave is nice. People have realized that he is big and rough. Since he is young he has tendency to become red when they mention it. That guy Eric sure is nice. Harry seems to be round. A person who is kind and rough and blue is young. One who is young, red and also cold will definitely be round, too. Nice people with blue and rough skin are very young and cannot be trusted with responsibility. Someone who has a round shape with a nice personality with big size are always kind. Nice and green people are often found to be big as well. Nice people who are also red are going to be cold too.

No proof: neither the statement nor its negation follows.

Depth 1, open-world task

61.3%
right on 185 of 302 problems wrong; chance 33% (dashed line)
right

The mouse sees the mouse.

Correct answer: unknown · Model said: unknown (100.0% sure)

The problem and its proof

The cat needs the mouse. The cow does not see the cat. The mouse is big. If something chases the cat then the cat is not nice. If something needs the mouse then the mouse is not nice. If something chases the cow and it is not nice then it sees the mouse.

No proof: neither the statement nor its negation follows.

wrong

The tiger eats the tiger.

Correct answer: false · Model said: unknown (100.0% sure)

The problem and its proof

The bald eagle chases the tiger. The bald eagle eats the tiger. The bald eagle is big. The bald eagle is blue. The bald eagle is not cold. The bald eagle is green. The bald eagle is round. The bald eagle likes the tiger. The tiger chases the bald eagle. The tiger eats the bald eagle. The tiger is not big. The tiger does not like the bald eagle. If the tiger likes the bald eagle then the tiger eats the bald eagle. If something chases the tiger then it likes the tiger. If something chases the tiger and it likes the bald eagle then the tiger likes the bald eagle. If the tiger likes the bald eagle and the tiger eats the bald eagle then the tiger chases the bald eagle. If something is cold and it likes the bald eagle then the bald eagle likes the tiger. If something eats the bald eagle and it does not like the bald eagle then it does not eat the tiger.

The proof that the statement's negation follows:

  1. The tiger eats the bald eagle. + The tiger does not like the bald eagle.If something eats the bald eagle and it does not like the bald eagle then it does not eat the tiger.The tiger does not eat the tiger.
wrong

The bald eagle does not need the squirrel.

Correct answer: false · Model said: unknown (49.0% sure)

The problem and its proof

The bald eagle is big. The bald eagle is kind. The bald eagle does not like the rabbit. The bald eagle needs the cat. The bald eagle needs the rabbit. The bald eagle does not see the squirrel. The cat is not green. The cat likes the rabbit. The rabbit likes the bald eagle. The squirrel is not blue. The squirrel is not cold. The squirrel is kind. The squirrel needs the cat. The squirrel needs the rabbit. If something likes the squirrel then the squirrel is big. All big things are green. If something is kind and big then it needs the squirrel. If something sees the rabbit and it needs the cat then it does not like the rabbit. If something needs the rabbit and the rabbit is kind then it needs the cat. If the bald eagle likes the squirrel and the bald eagle needs the squirrel then the squirrel does not like the bald eagle. If something sees the cat and it is blue then the cat does not see the bald eagle. If something needs the squirrel then it likes the squirrel.

The proof that the statement's negation follows:

  1. The bald eagle is kind. + The bald eagle is big.If something is kind and big then it needs the squirrel.The bald eagle needs the squirrel.
right

Charlie is not kind.

Correct answer: unknown · Model said: unknown (100.0% sure)

paraphrased
The problem and its proof

Dave may be round, but he is also kind. Eric is rather round shaped nice fellow who is feeling blue and looking green today. Harry is big, round and rough, but he is nice and kind. Young people who are cold to others and green with envy are actually nice. Every single big person is a little green in some areas. You'll always find rough, cold, green people to also be red people. Someone that is cold rough and red is also considered to be kind. A round shaped kind person who is colored green will be cold natured.

No proof: neither the statement nor its negation follows.

Depth 2, open-world task

51.5%
right on 156 of 303 problems wrong; chance 33% (dashed line)
wrong

The mouse does not need the mouse.

Correct answer: true · Model said: unknown (99.0% sure)

The problem and its proof

The cat is young. The cow is young. The cow visits the mouse. The mouse eats the cat. The mouse needs the cat. The mouse needs the rabbit. The mouse does not visit the cow. The mouse visits the rabbit. The rabbit is round. The rabbit does not need the cat. The rabbit visits the mouse. If someone eats the cat and they eat the cow then they do not need the mouse. If someone visits the cat then the cat eats the cow. All nice people are red. If the mouse needs the rabbit and the mouse eats the cat then the mouse eats the cow. If someone is red and they visit the mouse then the mouse needs the cow. If someone needs the cow then the cow visits the cat. All round people are nice. If someone needs the cat and the cat visits the rabbit then they do not eat the rabbit.

  1. The mouse needs the rabbit. + The mouse eats the cat.If the mouse needs the rabbit and the mouse eats the cat then the mouse eats the cow.The mouse eats the cow.
  2. + The mouse eats the cat.If someone eats the cat and they eat the cow then they do not need the mouse.The mouse does not need the mouse.
right

Charlie is not red.

Correct answer: false · Model said: false (97.0% sure)

paraphrased
The problem and its proof

Charlie is young, round, and kind, but that doesn't mean he isn't also rough and cold at times, as well. The young person who is always feeling cold is named Dave. Eric is cold to his ex-wife. She knows he is round and red. He is kind to everyone else. That guy Harry sure is nice. Kind red people are green on the inside. Rough and big people are always also cold people. Any person that's blue, young and green will turn out to be a nice person, too. All young and kind people that feel blue are described as red. A big and round individual is sure to be a kind individual, too. Any person who can be red and blue at the same time is cold. People who are young are also blue.

The proof that the statement's negation follows:

  1. Charlie is young.If someone is young then they are blue.Charlie is blue.
  2. + Charlie is young. + Charlie is kind.If someone is blue and young and kind then they are red.Charlie is red.
right

The bear needs the dog.

Correct answer: unknown · Model said: unknown (51.0% sure)

The problem and its proof

The bear likes the dog. The bear needs the mouse. The bear sees the dog. The dog is kind. The dog is not rough. The mouse is not round. The mouse likes the bear. If something is red then it needs the dog. If something is round then it needs the dog. If something is kind then it is round.

No proof: neither the statement nor its negation follows.

wrong

The rabbit is red.

Correct answer: false · Model said: unknown (48.0% sure)

The problem and its proof

The rabbit is blue. All nice things are not red. Nice, kind things are red. Nice things are green. All green things are not kind. All green things are nice. All blue things are nice.

The proof that the statement's negation follows:

  1. The rabbit is blue.All blue things are nice.The rabbit is nice.
  2. All nice things are not red.The rabbit is not red.

Depth 3, open-world task

47.9%
right on 145 of 303 problems wrong; chance 33% (dashed line)
wrong

Dave is not kind.

Correct answer: false · Model said: unknown (100.0% sure)

paraphrased
The problem and its proof

Dave is red, but he has turned blue as he is cold. He is nice but can be rough. Gary seems to be round. Harry is a big fellow, often blue and sad, but he is nice. People here that are both big and kind while being cold are always red. A very nice person who is also green is certainly kind. Every single big person is a little green in some areas. People who are young and blue are also red. People that are rough and red in the face, are usually big in size. People who are round and behave in a cold way are surely blue.

The proof that the statement's negation follows:

  1. Dave is red. + Dave is rough.If someone is red and rough then they are big.Dave is big.
  2. If someone is big then they are green.Dave is green.
  3. + Dave is nice.If someone is green and nice then they are kind.Dave is kind.
wrong

Bob is red.

Correct answer: true · Model said: unknown (88.0% sure)

paraphrased
The problem and its proof

For being so cold, it's good Alan can remain nice. Bob is green, but often kind, even when he is blue and cold. Eric is a big guy who is colored red from a sunburn. He also has a cold. After Harry got wet in the rain, he feels cold. He also looks green but big. People here that are both big and kind while being cold are always red. A person who is kind, cold, and rough is big. Young, red people are usually quite blue too. A blue, cold natured person will be rough textured. I've noticed that nice people who have rough, green skin have a tendancy to be round. If someone is green and naive they may also have red, rough skin. It is a safe guess then that they are also round. Nice people who feel blue often look green when ill.

  1. Bob is blue. + Bob is cold.If someone is blue and cold then they are rough.Bob is rough.
  2. + Bob is kind. + Bob is cold.If someone is kind and cold and rough then they are big.Bob is big.
  3. + Bob is kind. + Bob is cold.If someone is big and kind and cold then they are red.Bob is red.
right

Gary is not smart.

Correct answer: true · Model said: true (97.0% sure)

The problem and its proof

Bob is not rough. Dave is furry. Gary is rough. If Gary is big then Gary is not nice. Rough people are big. If someone is rough and not nice then they are not smart.

  1. Gary is rough.Rough people are big.Gary is big.
  2. If Gary is big then Gary is not nice.Gary is not nice.
  3. + Gary is rough.If someone is rough and not nice then they are not smart.Gary is not smart.
right

The rabbit is red.

Correct answer: true · Model said: true (99.0% sure)

The problem and its proof

The rabbit is cold. All cold things are green. All green things are round. If something is young and not green then it is round. If something is round and not cold then it is red. All young things are red. Round things are red.

  1. The rabbit is cold.All cold things are green.The rabbit is green.
  2. All green things are round.The rabbit is round.
  3. Round things are red.The rabbit is red.

Depth 4, open-world task

35.3%
right on 107 of 303 problems wrong; chance 33% (dashed line)
wrong

Charlie is not nice.

Correct answer: false · Model said: unknown (91.0% sure)

The problem and its proof

Anne is red. Anne is rough. Anne is young. Charlie is rough. Fiona is not kind. Fiona is nice. Gary is big. Young, red people are big. Nice people are green. If someone is big and not kind then they are green. All rough people are young. All nice people are young. All rough people are young. All young people are red. If someone is big then they are nice. If Charlie is kind then Charlie is young.

The proof that the statement's negation follows:

  1. Charlie is rough.All rough people are young.Charlie is young.
  2. All young people are red.Charlie is red.
  3. Young, red people are big.Charlie is big.
  4. If someone is big then they are nice.Charlie is nice.
wrong

Fiona is not cold.

Correct answer: true · Model said: unknown (99.0% sure)

The problem and its proof

Charlie is not red. Erin is big. Erin is not red. Erin is not white. Fiona is nice. Harry is not cold. Harry is white. Cold people are not big. If Erin is big then Erin is not green. Nice people are quiet. If someone is green and quiet then they are white. All white people are not cold. If Erin is nice then Erin is white. All quiet people are green. If someone is quiet and not cold then they are red.

  1. Fiona is nice.Nice people are quiet.Fiona is quiet.
  2. All quiet people are green.Fiona is green.
  3. If someone is green and quiet then they are white.Fiona is white.
  4. All white people are not cold.Fiona is not cold.
wrong

Erin is furry.

Correct answer: false · Model said: unknown (100.0% sure)

The problem and its proof

Charlie is furry. Erin is quiet. Erin is smart. Fiona is kind. Fiona is not quiet. Fiona is rough. Gary is not cold. Gary is furry. Gary is green. Gary is kind. Gary is quiet. Gary is rough. All cold, kind things are not furry. Smart things are cold. If something is cold then it is rough. If something is rough and cold then it is kind. If Fiona is kind then Fiona is rough. All furry, smart things are green. If something is cold and not furry then it is green.

The proof that the statement's negation follows:

  1. Erin is smart.Smart things are cold.Erin is cold.
  2. If something is cold then it is rough.Erin is rough.
  3. If something is rough and cold then it is kind.Erin is kind.
  4. All cold, kind things are not furry.Erin is not furry.
right

The bald eagle chases the cow.

Correct answer: unknown · Model said: unknown (51.0% sure)

The problem and its proof

The bald eagle is young. The bald eagle needs the lion. The cow eats the bald eagle. The cow needs the bald eagle. The lion chases the cow. The lion eats the cow. The lion needs the bald eagle. If someone needs the cow and they eat the bald eagle then the bald eagle eats the lion. If someone needs the lion and they need the bald eagle then they chase the cow. If someone is cold then they need the bald eagle. If someone is round then they need the bald eagle. If someone eats the lion then they are round. If someone needs the bald eagle and the bald eagle is round then the bald eagle is rough. If someone needs the bald eagle then they are round. If someone is round then they need the lion.

No proof: neither the statement nor its negation follows.

Depth 5, open-world task

38.1%
right on 110 of 289 problems wrong; chance 33% (dashed line)
right

Gary is not round.

Correct answer: unknown · Model said: unknown (66.0% sure)

The problem and its proof

Erin is nice. Erin is white. Fiona is quiet. Gary is white. Harry is cold. Harry is green. Harry is nice. Harry is quiet. Harry is round. Harry is white. All quiet people are round. If someone is quiet then they are round. If Erin is round then Erin is green. Nice people are cold. If someone is quiet and nice then they are cold. If someone is round then they are nice. If someone is cold then they are quiet. If Gary is quiet then Gary is not white. If someone is nice and green then they are smart.

No proof: neither the statement nor its negation follows.

right

Charlie is not white.

Correct answer: unknown · Model said: unknown (98.0% sure)

The problem and its proof

Anne is big. Anne is furry. Anne is nice. Anne is round. Bob is furry. Bob is kind. Bob is round. Bob is smart. Bob is white. Charlie is nice. Harry is furry. Harry is nice. If Bob is big then Bob is nice. If Bob is round then Bob is furry. If someone is furry then they are big. Big people are round. If someone is kind and white then they are smart. Nice, round people are kind. Kind, white people are nice. If someone is smart then they are white. Kind, nice people are white.

No proof: neither the statement nor its negation follows.

wrong

Bob is young.

Correct answer: true · Model said: unknown (95.0% sure)

The problem and its proof

Anne is blue. Anne is nice. Anne is quiet. Anne is rough. Anne is white. Bob is big. Charlie is rough. Erin is big. Erin is nice. Erin is young. Quiet, nice things are white. If something is rough then it is quiet. If Bob is white then Bob is young. If Anne is big and Anne is blue then Anne is rough. If Bob is rough and Bob is quiet then Bob is nice. Big things are rough. If Erin is nice and Erin is young then Erin is big.

  1. Bob is big.Big things are rough.Bob is rough.
  2. If something is rough then it is quiet.Bob is quiet.
  3. If Bob is rough and Bob is quiet then Bob is nice.Bob is nice.
  4. Quiet, nice things are white.Bob is white.
  5. If Bob is white then Bob is young.Bob is young.
wrong

The cow is not big.

Correct answer: true · Model said: unknown (100.0% sure)

The problem and its proof

The cat is not round. The cow is blue. The cow is kind. The cow likes the cat. The cow sees the rabbit. The rabbit chases the cow. The rabbit is nice. The tiger does not chase the cow. The tiger does not like the rabbit. The tiger sees the rabbit. If something chases the cow then the cow chases the cat. If something chases the rabbit then it is not big. If something likes the cow then the cow is nice. If something is kind then it sees the rabbit. If something chases the cat then the cat likes the cow. If the rabbit chases the tiger then the rabbit does not see the cow. If something is nice and it sees the rabbit then it chases the rabbit. If something is big then it is round.

  1. The rabbit chases the cow.If something chases the cow then the cow chases the cat.The cow chases the cat.
  2. If something chases the cat then the cat likes the cow.The cat likes the cow.
  3. If something likes the cow then the cow is nice.The cow is nice.
  4. + The cow sees the rabbit.If something is nice and it sees the rabbit then it chases the rabbit.The cow chases the rabbit.
  5. If something chases the rabbit then it is not big.The cow is not big.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

82.0%
right on 246 of 300 problems wrong; chance 50% (dashed line)
right

Harry is round.

Correct answer: false · Model said: false (99.0% sure)

paraphrased
The problem and its proof

That guy Bob sure is nice. Dave is young, round and kind with green ideas and wears blue clothes. Eric is somehow red and kind and also green. That guy Harry sure is nice. People who are very nice and easily flush red are often wearing green concealer. All those who are nice and big are red. Watch out for young people that are kind for they are rough. Blue eyed people, green with sickness and rough around the edges quickly turn red and blush when stepping ashore. Any person who can be red and blue at the same time is cold. A person that is round and somewhat green while being nice tends to be red as well.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Fred is nice.

Correct answer: true · Model said: true (96.0% sure)

paraphrased
The problem and its proof

Green, big and red are qualities which all describe Alan. Green, big and red are qualities which all describe Charlie. That guy Fred sure is nice. Harry is a round and rough around the edges, and he is also big. Red people are almost always rough. If a person is described as being nice, cold and young it usually follow that they are red. Cold people that are big and red are usually nice. People who are feeling blue and green are said to be red. Rough people who look red are cold.

Stated in the text: "Fred is nice."

wrong

The bald eagle is not green.

Correct answer: false · Model said: true (97.0% sure)

The problem and its proof

The bald eagle is green. The cow chases the rabbit. The rabbit sees the squirrel. The squirrel sees the cow. If something needs the bald eagle then it is nice. If something chases the rabbit then it chases the cow. If something sees the bald eagle and the bald eagle needs the cow then the bald eagle chases the rabbit.

Stated in the text: "The bald eagle is green."

right

The bald eagle is nice.

Correct answer: false · Model said: false (96.0% sure)

The problem and its proof

The bald eagle is big. The bear eats the cow. The cow sees the mouse. The mouse likes the bald eagle. If something likes the bald eagle then the bald eagle sees the mouse.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

65.7%
right on 197 of 300 problems wrong; chance 50% (dashed line)
right

Alan is big.

Correct answer: true · Model said: true (51.0% sure)

paraphrased
The problem and its proof

Rough and round Alan is nice and kind and he's also cold. For being so cold, it's good Bob can remain nice. Gary seems to be round. That guy Harry sure is nice. A person who is feeling rough, has red cheeks and feeling green will also tend to feel blue. If someone is green and naive they may also have red, rough skin. It is a safe guess then that they are also round. Even though a person is big and rough when they are cold they turn red. Kind people that are round are on the big side. When you meet and big and nice man, they will be kind to the earth and live green lifestyle. People who are round and green while being cold are also red.

  1. Alan is kind. + Alan is round.If someone is kind and round then they are big.Alan is big.
right

Fred is red.

Correct answer: false · Model said: false (99.0% sure)

paraphrased
The problem and its proof

Dave is red, but he has turned blue as he is cold. He is nice but can be rough. Gary seems to be round. Harry is a big fellow, often blue and sad, but he is nice. People here that are both big and kind while being cold are always red. A very nice person who is also green is certainly kind. Every single big person is a little green in some areas. People who are young and blue are also red. People that are rough and red in the face, are usually big in size. People who are round and behave in a cold way are surely blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Dave is not young.

Correct answer: false · Model said: true (92.0% sure)

paraphrased
The problem and its proof

That guy Bob sure is nice. No one knows Dave like me and he is a kind guy, very round in the belly and green as grass. Fred seems to be round. That person with the super big round body wearing red and green clothes who is acting quite nice towards you while being cold and feeling blue is named Harry. A rather big person is a red person. An individual with the traits of being nice, red, and cold is also round. Every person I met that was cold and round was also red. Young people who are both nice and cold also tend to be rough. I bet anyone who is round and kind is also pretty young. If a person acts cold yet nice and green, they will be kind.

  1. Dave is round. + Dave is kind.If someone is round and kind then they are young.Dave is young.
right

The tiger does not like the tiger.

Correct answer: true · Model said: true (91.0% sure)

The problem and its proof

The bear likes the mouse. The mouse sees the tiger. The tiger sees the bear. If someone likes the mouse and they like the bear then the mouse eats the tiger. If someone likes the tiger then the tiger eats the bear. If someone eats the bear and they are green then they see the bear. If someone is green then they like the tiger. If someone sees the bear then they like the bear. If someone eats the tiger then they are rough. If someone is blue and they like the mouse then they are rough. If someone likes the mouse then the mouse is green.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 2, closed-world task

54.7%
right on 164 of 300 problems wrong; chance 50% (dashed line)
wrong

Bob is not blue.

Correct answer: false · Model said: true (98.0% sure)

The problem and its proof

Bob is cold. Bob is furry. Bob is rough. Bob is round. Bob is young. Dave is blue. Dave is rough. Erin is cold. Erin is furry. Erin is quiet. Erin is rough. Erin is round. Erin is young. Gary is round. Quiet, young things are blue. Round things are quiet. All furry things are young. All round, quiet things are rough. Blue, quiet things are rough. Rough things are furry. Blue things are furry. If Gary is young and Gary is quiet then Gary is rough.

  1. Bob is round.Round things are quiet.Bob is quiet.
  2. + Bob is young.Quiet, young things are blue.Bob is blue.
wrong

The mouse does not chase the cow.

Correct answer: false · Model said: true (56.0% sure)

The problem and its proof

The cow eats the lion. The lion visits the mouse. The mouse is big. If something chases the cow then the cow chases the mouse. If something chases the mouse then the mouse chases the cow. If something visits the mouse then it chases the mouse.

  1. The lion visits the mouse.If something visits the mouse then it chases the mouse.The lion chases the mouse.
  2. If something chases the mouse then the mouse chases the cow.The mouse chases the cow.
right

The mouse does not need the dog.

Correct answer: true · Model said: true (84.0% sure)

The problem and its proof

The bear is nice. The dog is young. The dog likes the mouse. The dog likes the squirrel. The dog needs the mouse. The dog needs the squirrel. The mouse is green. The mouse needs the bear. The squirrel is young. The squirrel likes the bear. The squirrel needs the bear. The squirrel visits the mouse. If someone likes the dog then they need the dog. If the dog does not need the mouse then the mouse visits the dog. If someone is nice then they visit the mouse. If the bear is nice then the bear visits the dog. If someone visits the squirrel and they are young then they are red. If someone visits the mouse then they are green. If someone likes the bear then they like the dog. If someone needs the dog and the dog needs the mouse then the dog likes the bear.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Bob is not young.

Correct answer: true · Model said: true (95.0% sure)

The problem and its proof

Bob is nice. Dave is kind. Dave is nice. Dave is smart. Dave is young. Fiona is kind. Fiona is nice. Gary is kind. Gary is red. Gary is smart. If someone is red then they are young. Nice people are kind. If someone is quiet and kind then they are red. All young people are rough. If Gary is not nice then Gary is quiet. If Gary is nice and Gary is not red then Gary is rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

51.3%
right on 154 of 300 problems wrong; chance 50% (dashed line)
right

Dave is not cold.

Correct answer: true · Model said: true (92.0% sure)

The problem and its proof

Charlie is smart. Dave is blue. Round things are cold. Young things are smart. If something is smart then it is round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cow is nice.

Correct answer: true · Model said: true (99.0% sure)

The problem and its proof

The bear chases the dog. The bear is cold. The bear is nice. The bear is red. The bear is young. The bear likes the dog. The cow chases the bear. The cow is kind. The cow is red. The cow likes the mouse. The cow visits the bear. The dog chases the mouse. The dog is kind. The dog is nice. The dog likes the bear. The mouse is cold. If something is nice then it chases the cow. If something visits the dog then it chases the bear. If the cow likes the dog and the dog chases the mouse then the dog likes the bear. If something likes the mouse then the mouse likes the cow. If something visits the mouse and it does not visit the bear then the bear visits the cow. If the bear visits the cow then the cow is nice. If the cow visits the bear and the bear visits the cow then the cow likes the mouse. If something is kind then it likes the dog. If something chases the cow then it visits the cow.

  1. The bear is nice.If something is nice then it chases the cow.The bear chases the cow.
  2. If something chases the cow then it visits the cow.The bear visits the cow.
  3. If the bear visits the cow then the cow is nice.The cow is nice.
wrong

Erin is not smart.

Correct answer: true · Model said: false (85.0% sure)

The problem and its proof

Bob is cold. Erin is quiet. Harry is round. If someone is round then they are cold. If someone is smart and cold then they are nice. Round, smart people are nice. If Erin is round then Erin is smart. Cold, nice people are big. If someone is smart and round then they are big. All nice people are round. If Bob is quiet and Bob is big then Bob is cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Dave is green.

Correct answer: false · Model said: true (72.0% sure)

The problem and its proof

Anne is cold. Bob is rough. Dave is white. All cold people are rough. If Anne is red then Anne is blue. If Dave is not white then Dave is green. If someone is red then they are blue. If someone is rough then they are red. All rough people are green. If someone is young and not cold then they are green. If someone is red and not blue then they are green.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 4, closed-world task

46.0%
right on 138 of 300 problems wrong; chance 50% (dashed line)
wrong

The tiger does not visit the bald eagle.

Correct answer: false · Model said: true (96.0% sure)

The problem and its proof

The bald eagle is green. The bald eagle needs the squirrel. The bald eagle sees the tiger. The bald eagle visits the mouse. The mouse is round. The mouse needs the squirrel. The mouse sees the squirrel. The squirrel is big. The squirrel is round. The squirrel needs the mouse. The squirrel visits the mouse. The squirrel visits the tiger. The tiger is big. The tiger is rough. The tiger needs the squirrel. The tiger visits the squirrel. If something visits the bald eagle and it is round then the bald eagle sees the squirrel. If something sees the tiger and it needs the squirrel then the tiger is round. If something is nice then it visits the bald eagle. If something is round then it sees the squirrel. If something sees the squirrel then it is nice.

  1. The bald eagle sees the tiger. + The bald eagle needs the squirrel.If something sees the tiger and it needs the squirrel then the tiger is round.The tiger is round.
  2. If something is round then it sees the squirrel.The tiger sees the squirrel.
  3. If something sees the squirrel then it is nice.The tiger is nice.
  4. If something is nice then it visits the bald eagle.The tiger visits the bald eagle.
right

The lion does not like the lion.

Correct answer: true · Model said: true (95.0% sure)

The problem and its proof

The dog eats the lion. The dog is big. The dog is rough. The dog is round. The dog likes the lion. The dog sees the lion. The lion likes the dog. If someone is round then they like the lion. If someone is green then they eat the dog. If someone likes the dog and they like the lion then they eat the dog. If the dog sees the lion and the lion is big then the lion sees the dog. If someone eats the dog then they are round. If someone eats the lion then they are green. If someone is nice and not round then they see the dog. If someone is rough then they see the dog.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The cow is rough.

Correct answer: false · Model said: true (58.0% sure)

The problem and its proof

The cow is blue. If something is blue then it is young. All rough, young things are blue. If the cow is round then the cow is cold. If the cow is blue then the cow is young. All cold things are round. If something is rough then it is round. All rough, round things are cold. All cold, blue things are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Charlie is big.

Correct answer: false · Model said: false (99.0% sure)

The problem and its proof

Charlie is nice. Dave is green. Erin is big. Erin is nice. Erin is rough. Fiona is quiet. Fiona is rough. All green people are rough. If someone is rough then they are quiet. If someone is quiet then they are young. Young people are big. If someone is big and quiet then they are nice. If someone is quiet and not rough then they are nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 5, closed-world task

38.0%
right on 114 of 300 problems wrong; chance 50% (dashed line)
wrong

The lion eats the dog.

Correct answer: true · Model said: false (95.0% sure)

The problem and its proof

The bald eagle eats the lion. The bald eagle needs the rabbit. The bald eagle sees the dog. The bald eagle sees the rabbit. The dog eats the bald eagle. The dog eats the rabbit. The dog needs the lion. The lion needs the bald eagle. The lion needs the dog. The rabbit eats the lion. The rabbit is green. The rabbit is round. If something sees the rabbit then it eats the rabbit. If something sees the lion and it is blue then the lion sees the dog. If something eats the rabbit then it eats the dog. If something is green and it sees the lion then the lion eats the rabbit. If something is rough then it eats the lion. If something eats the dog and it eats the lion then the lion sees the rabbit. If something is rough and it sees the lion then the lion is blue.

  1. The bald eagle sees the rabbit.If something sees the rabbit then it eats the rabbit.The bald eagle eats the rabbit.
  2. If something eats the rabbit then it eats the dog.The bald eagle eats the dog.
  3. + The bald eagle eats the lion.If something eats the dog and it eats the lion then the lion sees the rabbit.The lion sees the rabbit.
  4. If something sees the rabbit then it eats the rabbit.The lion eats the rabbit.
  5. If something eats the rabbit then it eats the dog.The lion eats the dog.
right

Harry is not big.

Correct answer: true · Model said: true (97.0% sure)

The problem and its proof

Bob is cold. Bob is smart. Fiona is cold. Fiona is quiet. Fiona is smart. Gary is nice. Harry is smart. If someone is quiet and cold then they are rough. All nice people are red. Big people are nice. All red people are quiet. If someone is quiet and rough then they are big. If Fiona is red then Fiona is cold. All red, nice people are smart. Quiet, smart people are rough. All cold people are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Anne is red.

Correct answer: false · Model said: false (98.0% sure)

The problem and its proof

Anne is round. Fiona is big. Fiona is round. Fiona is young. Gary is red. Gary is white. Harry is big. Harry is round. Harry is smart. Harry is white. Young things are smart. Young things are red. If something is red and smart then it is white. Big things are red. Big, rough things are round. All smart, white things are red. Smart things are big. If something is red then it is young. Big things are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Bob is white.

Correct answer: false · Model said: true (69.0% sure)

The problem and its proof

Bob is cold. Charlie is big. Charlie is blue. Charlie is cold. Charlie is furry. Charlie is white. Gary is big. Gary is cold. Harry is big. Harry is cold. Harry is quiet. Furry, blue people are big. All big, cold people are green. If someone is quiet and furry then they are blue. Furry, quiet people are white. If someone is quiet and furry then they are green. If Gary is white then Gary is quiet. If Bob is green and Bob is white then Bob is furry. If someone is furry then they are white. All green people are furry.

No proof to show: under the closed world this answer holds because something cannot be derived.

What GPT-6 Luna got right and wrong, depth by depth

At each proof depth, GPT-6 Luna's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

93.3%
right on 280 of 300 problems wrong; chance 33% (dashed line)
right

Alan is not young.

Correct answer: false · Model said: false (100.0% sure, probe)

paraphrased
The problem and its proof

Alan is young and green. This makes him cold to others. Eric seems to be round. Even though Gary is big and rough he is sometimes round and kind. Harry is a young, round shaped young man who is also very cold. Truth be told, every big, rough person out there that is green, will also be colored blue. Big people who are red in color are almost always young. If you come across a young, blue person you can count on them being nice to you. People turn green when they are feeling blue. Nice people that are very green and even round shaped will be very young. Any person who can be red and blue at the same time is cold. If you ever meet someone who is both green and blue while being nice, then they'll be big, too.

Stated in the text: "Alan is young."

right

The lion does not visit the bear.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The bald eagle visits the tiger. The bear visits the bald eagle. The lion does not visit the bear. The tiger needs the lion. If someone needs the lion then they do not need the tiger. If someone sees the bear and they see the lion then they do not need the bear.

Stated in the text: "The lion does not visit the bear."

right

Bob is blue.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

paraphrased
The problem and its proof

Alan woke up one day to find he had become round. He was still rough, red, and green, but now he was also big. Bob seems to be round. The young person who is always feeling cold is named Charlie. Dave who is young and nice is also quite big and feels green. Not a single young person is not also kind. Even though young people are rough, they are still very red. Watch out for young people that are kind for they are rough. Nice people with red and rough skin are green with envy. A young person who is big and rough and big is also usually round. When someone is blue, nice and young they will also be as big as the sun.

No proof: neither the statement nor its negation follows.

right

Charlie is round.

Correct answer: true · Model said: true (100.0% sure, probe)

paraphrased
The problem and its proof

Charlie is a round and rough around the edges, and he is also big. Eric is young and green. Even though he is round he is cold outside. Fred is a young, round shaped young man who is also very cold. Gary may be round, but he is also kind. Do not fear big cold people for they are often kind. Someone who is a young age and looks like they are round are also red. Any human who is rough, green, and kind, has a blue personality. Any red sort of person is a nice person. Nice and green people are often found to be big as well.

Stated in the text: "Charlie is round."

Depth 1, open-world task

77.8%
right on 235 of 302 problems wrong; chance 33% (dashed line)
right

The tiger is not kind.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The tiger is nice. The tiger is red. The tiger is round. If someone is round then they are not kind.

  1. The tiger is round.If someone is round then they are not kind.The tiger is not kind.
right

Gary is rough.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

The problem and its proof

Bob is red. Erin is not green. Gary is young. Rough people are green. If someone is young then they are green. All furry people are big. Cold, furry people are young. If someone is red then they are furry. All furry, red people are rough.

No proof: neither the statement nor its negation follows.

wrong

Eric is not round.

Correct answer: false · Model said: unknown (98.9% sure, probe)

paraphrased
The problem and its proof

Alan, who is relatively young, is also pretty big and tends to be cold. No one knows Charlie like me and he is a kind guy, very round in the belly and green as grass. Eric has big, red cheeks with a rough texture. He always wear green sweaters because he is easily cold. That guy Gary sure is nice. Rough people who are red in color and big are usally round in shape. A rather big person is a red person. Young people that are turning blue from being cold will be rough looking. When someone is green and round, they can be rough but kind. People with big smiles and round eyes will have red hair. Is a kind person if feeling cold and looking red then they are usually young. Red people, who are also round and young, are often quite blue as well.

The proof that the statement's negation follows:

  1. Eric is red. + Eric is rough. + Eric is big.If someone is red and rough and big then they are round.Eric is round.
right

Harry is white.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

The problem and its proof

Bob is not smart. Bob is white. Harry is green. Round people are white. White people are blue. If Bob is white and Bob is blue then Bob is round. Round, white people are not quiet. If someone is blue and not white then they are smart. If someone is white then they are not smart. If someone is smart and not quiet then they are green. All smart, round people are not green.

No proof: neither the statement nor its negation follows.

Depth 2, open-world task

60.4%
right on 183 of 303 problems wrong; chance 33% (dashed line)
right

The squirrel is young.

Correct answer: false · Model said: false (84.2% sure, probe)

The problem and its proof

The dog sees the squirrel. The dog visits the rabbit. The rabbit likes the squirrel. The rabbit sees the dog. The squirrel likes the dog. The squirrel sees the dog. The squirrel does not visit the rabbit. If someone sees the dog and the dog visits the rabbit then they visit the dog. If someone visits the dog then they are not young.

The proof that the statement's negation follows:

  1. The squirrel sees the dog. + The dog visits the rabbit.If someone sees the dog and the dog visits the rabbit then they visit the dog.The squirrel visits the dog.
  2. If someone visits the dog then they are not young.The squirrel is not young.
right

The bear eats the mouse.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The bear sees the mouse. The mouse chases the bear. If something is big then it eats the mouse. If something sees the mouse then it is big.

  1. The bear sees the mouse.If something sees the mouse then it is big.The bear is big.
  2. If something is big then it eats the mouse.The bear eats the mouse.
wrong

The lion visits the mouse.

Correct answer: true · Model said: unknown

The problem and its proof

The lion does not need the tiger. The mouse visits the tiger. The squirrel visits the mouse. The tiger eats the lion. If someone visits the mouse then they visit the lion. If someone visits the lion then the lion visits the mouse.

  1. The squirrel visits the mouse.If someone visits the mouse then they visit the lion.The squirrel visits the lion.
  2. If someone visits the lion then the lion visits the mouse.The lion visits the mouse.
wrong

Gary is not blue.

Correct answer: true · Model said: unknown (97.6% sure, probe)

The problem and its proof

Erin is cold. Fiona is cold. Fiona is not quiet. Fiona is rough. Gary is nice. Gary is not quiet. Gary is smart. Smart things are big. Big things are smart. If something is cold and not quiet then it is smart. All big, quiet things are smart. If something is nice and cold then it is not smart. All quiet, big things are not nice. Rough things are not nice. All big things are not blue.

  1. Gary is smart.Smart things are big.Gary is big.
  2. All big things are not blue.Gary is not blue.

Depth 3, open-world task

61.7%
right on 187 of 303 problems wrong; chance 33% (dashed line)
right

Harry is green.

Correct answer: unknown · Model said: unknown (100.0% sure, probe)

paraphrased
The problem and its proof

That guy Alan sure is nice. Bob is round but he is also nice and kind. People that know Charlie will tell you he is rough and cold. But he can also be kind and red despite being blue because he is so young. That guy Harry sure is nice. People who are very nice and easily flush red are often wearing green concealer. If you're truly nice, cold, and blue, you're kind, too. Big, kind folks are green ones. Round people who are kind tend to be nice. Young and red people look round. A nice and green man or woman is also red in color.

No proof: neither the statement nor its negation follows.

wrong

Eric is not green.

Correct answer: false · Model said: unknown (98.3% sure, probe)

paraphrased
The problem and its proof

Bob is cold to his ex-wife. She knows he is round and red. He is kind to everyone else. Eric is known for being big and rough, but few know he is also always red with cold. For being so cold, it's good Gary can remain nice. For being so cold, it's good Harry can remain nice. Round people who feel blue and are green in color are often young in age. Rough people who are red in color and big are usally round in shape. Kind red people are green on the inside. A rather big person is a red person. A nice but rough person will also be green and cold. A big and round individual is sure to be a kind individual, too. A person that is red, rough and kind often feels cold.

The proof that the statement's negation follows:

  1. Eric is red. + Eric is rough. + Eric is big.If someone is red and rough and big then they are round.Eric is round.
  2. + Eric is big.If someone is big and round then they are kind.Eric is kind.
  3. + Eric is red.If someone is kind and red then they are green.Eric is green.
wrong

The cow does not see the cat.

Correct answer: unknown · Model said: false (97.6% sure, probe)

The problem and its proof

The bear does not chase the cat. The bear eats the cat. The bear eats the cow. The bear is green. The bear is not nice. The bear sees the dog. The cat is nice. The cat is red. The cat sees the bear. The cow eats the dog. The dog is kind. If someone is kind and they see the dog then the dog chases the cow. If someone eats the bear and the bear chases the dog then the dog eats the cow. If someone sees the bear then they see the cow. If someone sees the cat and the cat sees the bear then they eat the dog. If someone is nice and they eat the cow then the cow does not chase the cat. If someone sees the cow then they see the cat. If someone is green then they see the bear. If someone eats the dog then they are red.

No proof: neither the statement nor its negation follows.

right

Charlie is not young.

Correct answer: true · Model said: true (99.9% sure, probe)

The problem and its proof

Charlie is smart. Fiona is not young. All white, round people are big. If someone is young then they are big. All young, round people are big. If Fiona is white then Fiona is big. All round people are not young. If Charlie is cold then Charlie is round. If someone is smart then they are cold. Smart people are blue.

  1. Charlie is smart.If someone is smart then they are cold.Charlie is cold.
  2. If Charlie is cold then Charlie is round.Charlie is round.
  3. All round people are not young.Charlie is not young.

Depth 4, open-world task

44.6%
right on 135 of 303 problems wrong; chance 33% (dashed line)
right

Anne is young.

Correct answer: true · Model said: true

The problem and its proof

Anne is big. Charlie is big. Charlie is blue. Charlie is cold. Charlie is green. Charlie is nice. Charlie is rough. Charlie is young. Fiona is big. Fiona is green. Gary is big. Gary is cold. Gary is green. Gary is nice. Gary is rough. Gary is young. All rough, young people are nice. All big, cold people are rough. If someone is rough and blue then they are young. Big people are cold. Rough people are blue.

  1. Anne is big.Big people are cold.Anne is cold.
  2. + Anne is big.All big, cold people are rough.Anne is rough.
  3. Rough people are blue.Anne is blue.
  4. If someone is rough and blue then they are young.Anne is young.
wrong

Charlie is not kind.

Correct answer: false · Model said: unknown (100.0% sure, probe)

paraphrased
The problem and its proof

Bob is a round and rough around the edges, and he is also big. Young Charlie was rough in demeanor. He was cold, which made his skin blue despite being big. Gary seems big and rough, but he's actually quite kind. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Folks who are rough and tumble yet gentle and kind are blue. Someone who has young, nice and cold qualities will possess a red quality, too. Anyone feeling cold is bound to act nice. A round person who is on the young side is also likely to treat people in a kind manner. Tom is a rough, young person to know, he is very green at his job but he is very round from his weight. Nice people who feel blue often look green when ill.

The proof that the statement's negation follows:

  1. Charlie is cold.If someone is cold then they are nice.Charlie is nice.
  2. + Charlie is blue.If someone is nice and blue then they are green.Charlie is green.
  3. + Charlie is rough. + Charlie is young.If someone is rough and young and green then they are round.Charlie is round.
  4. + Charlie is young.If someone is round and young then they are kind.Charlie is kind.
right

Charlie is not blue.

Correct answer: true · Model said: true

The problem and its proof

Bob is cold. Bob is quiet. Charlie is white. Dave is quiet. Dave is white. Gary is blue. Gary is quiet. All white, green things are cold. White things are cold. Red, smart things are not blue. All red things are green. Green, white things are quiet. Smart, white things are red. If Charlie is quiet then Charlie is red. If something is blue and cold then it is smart. All white, cold things are smart.

  1. Charlie is white.White things are cold.Charlie is cold.
  2. + Charlie is white.All white, cold things are smart.Charlie is smart.
  3. + Charlie is white.Smart, white things are red.Charlie is red.
  4. Red, smart things are not blue.Charlie is not blue.
wrong

Bob is nice.

Correct answer: true · Model said: unknown (100.0% sure, probe)

The problem and its proof

Anne is furry. Anne is red. Anne is smart. Bob is red. Dave is blue. Dave is kind. Erin is white. Furry people are kind. Furry, blue people are kind. All red people are furry. All nice, red people are smart. Kind, nice people are smart. All furry people are kind. White people are kind. If someone is white and red then they are nice. If Bob is kind then Bob is white.

  1. Bob is red.All red people are furry.Bob is furry.
  2. Furry people are kind.Bob is kind.
  3. If Bob is kind then Bob is white.Bob is white.
  4. + Bob is red.If someone is white and red then they are nice.Bob is nice.

Depth 5, open-world task

46.0%
right on 133 of 289 problems wrong; chance 33% (dashed line)
wrong

Anne is not big.

Correct answer: false · Model said: unknown (100.0% sure, probe)

The problem and its proof

Anne is blue. Anne is nice. Bob is kind. Bob is nice. Fiona is blue. Fiona is furry. Fiona is kind. Fiona is nice. Fiona is smart. Gary is blue. Gary is furry. Gary is smart. Big, furry things are green. Kind, nice things are big. All smart, green things are kind. All nice things are furry. If Anne is kind and Anne is big then Anne is green. If Fiona is blue then Fiona is green. Furry, green things are smart. Furry things are green.

The proof that the statement's negation follows:

  1. Anne is nice.All nice things are furry.Anne is furry.
  2. Furry things are green.Anne is green.
  3. Furry, green things are smart.Anne is smart.
  4. All smart, green things are kind.Anne is kind.
  5. + Anne is nice.Kind, nice things are big.Anne is big.
right

The cat does not see the cat.

Correct answer: false · Model said: false (95.2% sure, probe)

The problem and its proof

The cat does not chase the rabbit. The cat is cold. The cat is red. The cat sees the rabbit. The cat sees the tiger. The mouse is big. The mouse is red. The rabbit does not chase the cat. The rabbit likes the cat. The rabbit does not like the mouse. The tiger likes the mouse. If something likes the mouse then it chases the cat. If something sees the mouse then it chases the cat. If something chases the mouse then the mouse does not like the rabbit. If something chases the cat then it sees the cat. If the tiger sees the cat then the cat sees the mouse.

The proof that the statement's negation follows:

  1. The tiger likes the mouse.If something likes the mouse then it chases the cat.The tiger chases the cat.
  2. If something chases the cat then it sees the cat.The tiger sees the cat.
  3. If the tiger sees the cat then the cat sees the mouse.The cat sees the mouse.
  4. If something sees the mouse then it chases the cat.The cat chases the cat.
  5. If something chases the cat then it sees the cat.The cat sees the cat.
right

The cow is not big.

Correct answer: true · Model said: true

The problem and its proof

The cat is not round. The cow is blue. The cow is kind. The cow likes the cat. The cow sees the rabbit. The rabbit chases the cow. The rabbit is nice. The tiger does not chase the cow. The tiger does not like the rabbit. The tiger sees the rabbit. If something chases the cow then the cow chases the cat. If something chases the rabbit then it is not big. If something likes the cow then the cow is nice. If something is kind then it sees the rabbit. If something chases the cat then the cat likes the cow. If the rabbit chases the tiger then the rabbit does not see the cow. If something is nice and it sees the rabbit then it chases the rabbit. If something is big then it is round.

  1. The rabbit chases the cow.If something chases the cow then the cow chases the cat.The cow chases the cat.
  2. If something chases the cat then the cat likes the cow.The cat likes the cow.
  3. If something likes the cow then the cow is nice.The cow is nice.
  4. + The cow sees the rabbit.If something is nice and it sees the rabbit then it chases the rabbit.The cow chases the rabbit.
  5. If something chases the rabbit then it is not big.The cow is not big.
wrong

Anne is quiet.

Correct answer: false · Model said: unknown (100.0% sure, probe)

The problem and its proof

Anne is smart. Charlie is not blue. Charlie is quiet. Erin is not nice. Erin is round. Harry is green. Harry is young. Smart things are green. If Anne is round then Anne is young. If something is blue and green then it is not quiet. If something is green and smart then it is round. If something is young then it is blue. Smart things are nice. If Harry is quiet and Harry is not smart then Harry is young.

The proof that the statement's negation follows:

  1. Anne is smart.Smart things are green.Anne is green.
  2. + Anne is smart.If something is green and smart then it is round.Anne is round.
  3. If Anne is round then Anne is young.Anne is young.
  4. If something is young then it is blue.Anne is blue.
  5. If something is blue and green then it is not quiet.Anne is not quiet.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

98.0%
right on 294 of 300 problems wrong; chance 50% (dashed line)
right

Charlie is not red.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Charlie is red. If something is nice and not young then it is blue. If Charlie is red then Charlie is kind. All kind things are big.

Stated in the text: "Charlie is red."

right

The rabbit does not eat the bear.

Correct answer: false · Model said: false (99.9% sure, probe)

The problem and its proof

The bear is rough. The rabbit eats the bear. The squirrel visits the rabbit. If someone eats the bear then they need the bear. If someone needs the squirrel and they are not big then they need the bear. If someone eats the squirrel and they are not big then the squirrel is blue. If someone needs the rabbit then they visit the rabbit. If someone needs the bear then the bear visits the rabbit. If someone eats the squirrel then they are blue. If someone needs the rabbit then they eat the bear. If someone eats the squirrel and the squirrel is not rough then they are cold.

Stated in the text: "The rabbit eats the bear."

right

The rabbit is not cold.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

The dog needs the rabbit. The dog visits the rabbit. The rabbit eats the dog. If something visits the dog then it is young. Round things are young. If something needs the dog and it visits the dog then the dog eats the rabbit. If something is nice then it visits the dog. If something visits the rabbit and the rabbit is young then it eats the rabbit. If something is young then it eats the rabbit. If something is cold and it eats the dog then it needs the rabbit. If something visits the rabbit and the rabbit eats the dog then it is round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The squirrel chases the cow.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

The cow chases the lion. The cow likes the rabbit. The cow sees the lion. The cow sees the squirrel. The lion chases the squirrel. The lion likes the cow. The lion likes the squirrel. The rabbit chases the lion. The rabbit likes the lion. The squirrel is big. The squirrel is rough. The squirrel likes the lion. The squirrel sees the lion. The squirrel sees the rabbit. If something sees the rabbit and the rabbit sees the cow then the rabbit chases the cow. If something is rough then it likes the rabbit. If something is cold and it chases the lion then the lion likes the squirrel. If something likes the cow then it is rough. If the lion is big and the lion chases the rabbit then the rabbit chases the cow. If something chases the rabbit then it is rough. If the lion sees the rabbit and the lion chases the cow then the cow is cold. If the lion is rough and the lion likes the rabbit then the rabbit likes the cow. If something is red and it sees the cow then the cow likes the rabbit.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

76.7%
right on 230 of 300 problems wrong; chance 50% (dashed line)
right

The rabbit is not green.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

The rabbit is red. If the rabbit is not blue then the rabbit is green. Green things are round. Green, red things are cold.

  1. If the rabbit is not blue then the rabbit is green.The rabbit is green.
right

Fred is round.

Correct answer: false · Model said: false (100.0% sure, probe)

paraphrased
The problem and its proof

Charlie is a red, rough, green and blue man. For being so young, Dave is nice. He has rough, red hands from working all day and it makes him feel blue. Gary seems to be round. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Some cold people can be nice at times,and red at at other times. Anyone who is both red and green must be as cold as Christmas. I've noticed that nice people who have rough, green skin have a tendancy to be round. Most big and cold people are also rough. A big round young person is often blue. When a person is blue, rough and kind then it means they also are cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Harry is furry.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

Bob is nice. Bob is young. Charlie is kind. Charlie is smart. Dave is kind. Dave is white. Harry is nice. All young people are white. All young people are furry. All nice people are furry. If someone is nice then they are blue. Blue people are nice. If someone is smart and kind then they are nice. If someone is white then they are kind. All kind, furry people are smart. All kind, white people are young.

  1. Harry is nice.All nice people are furry.Harry is furry.
wrong

Eric is nice.

Correct answer: true · Model said: false (41.6% sure, probe)

paraphrased
The problem and its proof

Charlie can be rough and cold, but he is also green, an avid gardener. Eric is green, but often kind, even when he is blue and cold. The young person who is always feeling cold is named Fred. Gary is kind and nice and looks green. If a kind person is green and a bit blue then they are also nice. A blue colored person who is nice is a red person. A person who is kind and rough and blue is young. If someone is cold and blue, you will find that they are also quite young. A person who is green, blue, and red in color will be round shaped.

  1. Eric is green. + Eric is kind. + Eric is blue.If someone is green and kind and blue then they are nice.Eric is nice.

Depth 2, closed-world task

65.3%
right on 196 of 300 problems wrong; chance 50% (dashed line)
wrong

The lion does not see the bear.

Correct answer: false · Model said: true (92.2% sure, probe)

The problem and its proof

The bear is rough. The lion chases the bear. If someone chases the lion and the lion is not young then the lion chases the bear. If someone sees the bear then the bear sees the lion. If the bear chases the lion then the bear is blue. If someone chases the bear then they eat the bear. If someone eats the bear then they see the bear. If the bear is red then the bear chases the lion. If someone chases the bear and they are young then the bear chases the lion. If someone eats the bear then the bear is big.

  1. The lion chases the bear.If someone chases the bear then they eat the bear.The lion eats the bear.
  2. If someone eats the bear then they see the bear.The lion sees the bear.
right

The mouse does not chase the cow.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

The cow eats the lion. The lion visits the mouse. The mouse is big. If something chases the cow then the cow chases the mouse. If something chases the mouse then the mouse chases the cow. If something visits the mouse then it chases the mouse.

  1. The lion visits the mouse.If something visits the mouse then it chases the mouse.The lion chases the mouse.
  2. If something chases the mouse then the mouse chases the cow.The mouse chases the cow.
right

The cow is young.

Correct answer: true · Model said: true (99.7% sure, probe)

The problem and its proof

The bald eagle chases the tiger. The bald eagle needs the tiger. The bald eagle sees the cow. The bald eagle sees the tiger. The cow chases the tiger. The cow is kind. The cow is red. The cow sees the bald eagle. The cow sees the tiger. The tiger chases the bald eagle. The tiger chases the cow. The tiger is blue. The tiger is kind. The tiger is red. The tiger needs the bald eagle. Young, kind people are green. If someone needs the tiger and the tiger needs the cow then the cow is young. If someone chases the tiger and they chase the bald eagle then the bald eagle sees the tiger. If someone sees the tiger then the tiger needs the cow. If someone needs the bald eagle and the bald eagle chases the tiger then the bald eagle sees the cow. If someone is red and they need the tiger then they are blue.

  1. The bald eagle sees the tiger.If someone sees the tiger then the tiger needs the cow.The tiger needs the cow.
  2. + The bald eagle needs the tiger.If someone needs the tiger and the tiger needs the cow then the cow is young.The cow is young.
right

Eric is nice.

Correct answer: true · Model said: true (99.9% sure, probe)

paraphrased
The problem and its proof

For being so cold, it's good Alan can remain nice. Eric is young and green. This makes him cold to others. Fred is a kind person and he is also often cold. That guy Harry sure is nice. All those who are nice and big are red. If you come across a young, blue person you can count on them being nice to you. People who are said to be big and nice are round. People who are young are also blue. A kind person will certainly be young.

  1. Eric is young.If someone is young then they are blue.Eric is blue.
  2. + Eric is young.If someone is blue and young then they are nice.Eric is nice.

Depth 3, closed-world task

57.0%
right on 171 of 300 problems wrong; chance 50% (dashed line)
right

Fiona is blue.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Anne is red. Erin is young. Fiona is kind. All white things are red. If something is quiet and white then it is kind. If something is quiet then it is red. If something is red then it is rough. All rough things are blue. All young things are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Gary is not cold.

Correct answer: false · Model said: true (99.9% sure, probe)

paraphrased
The problem and its proof

Red Charlie is still young while also very blue and green. Dave seems to be round. Eric, who is relatively young, is also pretty big and tends to be cold. Gary who is young and nice is also quite big and feels green. Not a single young person is not also kind. When you meet someone who is nice and kind, they wont feel blue. All those who are nice and big are red. A young person who is big and rough and big is also usually round. Young and red people look round. If a person is blue and red and big, then that person is cold.

  1. Gary is young.If someone is young then they are kind.Gary is kind.
  2. + Gary is nice.If someone is kind and nice then they are blue.Gary is blue.
  3. + Gary is nice. + Gary is big.If someone is nice and big then they are red.Gary is red.
  4. + Gary is big.If someone is blue and big and red then they are cold.Gary is cold.
wrong

Eric is nice.

Correct answer: true · Model said: false (100.0% sure, probe)

paraphrased
The problem and its proof

Charlie is green and cold too. Eric is a big guy who is colored red from a sunburn. He also has a cold. No one knows Fred like me and he is a kind guy, very round in the belly and green as grass. People here that are both big and kind while being cold are always red. Red people are almost always rough. A person who is cold and blue is nice. If you're truly nice, cold, and blue, you're kind, too. A big, round person who is kind is also red. Rough, cold people are blue. Nice people with red and rough skin are green with envy.

  1. Eric is red.If someone is red then they are rough.Eric is rough.
  2. + Eric is cold.If someone is rough and cold then they are blue.Eric is blue.
  3. + Eric is cold.If someone is cold and blue then they are nice.Eric is nice.
right

Gary is not green.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Bob is quiet. Bob is smart. Dave is rough. Erin is big. Erin is rough. Gary is round. Gary is white. Smart people are green. All rough people are smart. If someone is round then they are rough.

  1. Gary is round.If someone is round then they are rough.Gary is rough.
  2. All rough people are smart.Gary is smart.
  3. Smart people are green.Gary is green.

Depth 4, closed-world task

48.3%
right on 145 of 300 problems wrong; chance 50% (dashed line)
right

Fiona is not smart.

Correct answer: true · Model said: true (100.0% sure, probe)

The problem and its proof

Bob is cold. Dave is big. Dave is furry. Dave is rough. Dave is round. Dave is smart. Fiona is round. If someone is smart then they are big. Furry people are smart. All furry people are smart. Smart people are furry. If someone is cold then they are furry. All smart people are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Bob is not nice.

Correct answer: true · Model said: false (99.0% sure, probe)

The problem and its proof

Bob is big. Charlie is blue. Gary is nice. Harry is kind. If someone is blue and big then they are cold. Quiet people are nice. Blue people are cold. All cold people are green. All cold, green people are nice. If someone is nice then they are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The lion is young.

Correct answer: true · Model said: false

The problem and its proof

The bear eats the cow. The bear likes the lion. The bear likes the mouse. The cow chases the bear. The cow chases the mouse. The cow eats the bear. The cow is nice. The lion chases the cow. The lion chases the mouse. The lion likes the bear. The mouse chases the cow. The mouse chases the lion. If someone is young then they are round. If someone eats the lion then they are young. If someone eats the bear then they eat the lion. If someone is nice and they chase the lion then they like the bear. If someone eats the lion then the lion eats the bear.

  1. The cow eats the bear.If someone eats the bear then they eat the lion.The cow eats the lion.
  2. If someone eats the lion then the lion eats the bear.The lion eats the bear.
  3. If someone eats the bear then they eat the lion.The lion eats the lion.
  4. If someone eats the lion then they are young.The lion is young.
right

Charlie is young.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Charlie is rough. Dave is white. Erin is blue. Harry is kind. If something is white then it is cold. Big things are white. If something is cold then it is young. If something is young then it is blue. If something is rough then it is blue. White, cold things are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 5, closed-world task

44.7%
right on 134 of 300 problems wrong; chance 50% (dashed line)
right

Erin is rough.

Correct answer: false · Model said: false (94.4% sure, probe)

The problem and its proof

Dave is blue. Dave is rough. Dave is young. Erin is blue. Erin is round. Erin is young. Fiona is rough. Gary is green. Gary is nice. Gary is young. Rough things are round. Green, nice things are quiet. Round, rough things are green. All quiet things are green. All green things are nice. All quiet things are rough. All rough, nice things are quiet. All quiet, green things are rough. If something is nice and quiet then it is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The rabbit likes the bear.

Correct answer: true · Model said: false (100.0% sure, probe)

The problem and its proof

The bear is red. The rabbit is red. The rabbit likes the tiger. The squirrel is cold. The tiger eats the bear. The tiger is big. The tiger likes the rabbit. All cold people are kind. If someone is kind then they like the bear. If someone chases the rabbit and they are kind then the rabbit is kind. If someone likes the bear then they chase the rabbit. If someone chases the tiger and the tiger is red then the tiger chases the bear. If the squirrel is kind and the squirrel likes the rabbit then the rabbit is green. If the tiger eats the bear and the bear chases the rabbit then the bear is green.

  1. The squirrel is cold.All cold people are kind.The squirrel is kind.
  2. If someone is kind then they like the bear.The squirrel likes the bear.
  3. If someone likes the bear then they chase the rabbit.The squirrel chases the rabbit.
  4. If someone chases the rabbit and they are kind then the rabbit is kind.The rabbit is kind.
  5. If someone is kind then they like the bear.The rabbit likes the bear.
wrong

The tiger does not see the tiger.

Correct answer: false · Model said: true (100.0% sure, probe)

The problem and its proof

The lion visits the mouse. The mouse sees the lion. The mouse sees the rabbit. The mouse visits the rabbit. The rabbit chases the lion. The rabbit chases the tiger. The rabbit is nice. The rabbit visits the lion. The tiger chases the lion. The tiger chases the mouse. If someone is green then they visit the tiger. If someone is nice then they visit the mouse. If someone sees the rabbit then they are kind. If someone sees the tiger then the tiger sees the rabbit. If someone is kind then they see the tiger. If the lion chases the mouse then the mouse sees the rabbit. If the rabbit does not chase the tiger then the tiger is cold.

  1. The mouse sees the rabbit.If someone sees the rabbit then they are kind.The mouse is kind.
  2. If someone is kind then they see the tiger.The mouse sees the tiger.
  3. If someone sees the tiger then the tiger sees the rabbit.The tiger sees the rabbit.
  4. If someone sees the rabbit then they are kind.The tiger is kind.
  5. If someone is kind then they see the tiger.The tiger sees the tiger.
right

Fiona is rough.

Correct answer: false · Model said: false (100.0% sure, probe)

The problem and its proof

Bob is cold. Dave is big. Dave is furry. Dave is rough. Dave is round. Dave is smart. Fiona is round. If someone is smart then they are big. Furry people are smart. All furry people are smart. Smart people are furry. If someone is cold then they are furry. All smart people are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

The OpenAI Decisions API, as we tested it

OpenAI's Decisions API is generally available (OpenAI's announcement of 22 October 2026). A request goes to POST /v1/decisions with the model gpt-6-luna, and each question is a predicate, a choice or a score. A choice answer returns the chosen option, a probability for every option and a separate confidence (OpenAI's Decisions API guide). We sent one choice question per problem, with the instructions and option descriptions every other model received.

Accuracy

It answered 54.0% of open-world problems and 56.3% of closed-world ones correctly. Jev answered 83.8% and 89.3%; +29.8 points on the open world task (95% interval +27.1 to +32.5) on the same items, and +33.0 points on the closed world task (95% interval +30.4 to +35.3) on the same items. GPT-6 Luna through the chat endpoint with reasoning off, the same model name, scored 64.1% and 65.0%, so the Decisions API scored 10.1 and 8.7 points lower on the same problems. We do not know why. We did not test other wordings of the instructions.

By depth on the open-world task: 90% when the answer is stated outright (depth 0), falling to 38% at five chained inferences. At depth 0 the request and the scoring work as intended; the loss is in the deeper problems.

On the open-world task it answered unknown for 50% of problems whose correct answer is true and 65% of those whose answer is false; Jev did so for 10% and 10%. On the closed-world task it answered true for 54% of problems whose answer is false (36% for GPT-6 Luna through chat, 11% for Jev).

Confidence

The response carries a confidence beside its probabilities. We checked whether it separates right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.57 on the open-world task and 0.62 on the closed-world task. Probabilities are rounded to two decimals, so the confidence took only 64 distinct values on the open-world task and 51 on the closed-world one.

Speed and price

OpenAI says the API "returns typed answers about 10x faster than the Responses API" (OpenAI's Decisions API guide); that is OpenAI's claim, and we did not compare against the Responses API. What we measured: a median of 219 ms per request and a 90th percentile of 379 ms over the network, in the scored run, with 8 requests at a time. That includes network time and is not like for like with models that ran on a laptop.

At OpenAI's list price of $0.10 per million input tokens (output tokens are not charged), the 3,600 problems cost $0.12, or $62 per million correct decisions. Cost per decision for every hosted model.

Where each one is better

Open world, Jev vs OpenAI Decisions API (GPT-6 Luna). Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. OpenAI Decisions API (GPT-6 Luna) is clearly ahead on unknown answers (+6.4 points).

Closed world, Jev vs OpenAI Decisions API (GPT-6 Luna). Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. OpenAI Decisions API (GPT-6 Luna) is not clearly ahead on any depth or answer class.

Open world, Jev vs GPT-6 Luna. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GPT-6 Luna is clearly ahead on unknown answers (+9.2 points).

Closed world, Jev vs GPT-6 Luna. Jev is clearly ahead on depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GPT-6 Luna is not clearly ahead on any depth or answer class. Too close to call: depth 0.

A slice counts as clearly ahead when its paired 95% interval excludes zero.

True, false and unknown, open world

By correct answer, Open world
Correct answerItemsJevOpenAI Decisions API (GPT-6 Luna)GPT-6 LunaBest constant
false60588.3% 85.5–90.932.4% 28.4–36.250.9% 46.8–54.9one answer
true60588.8% 86.1–91.149.6% 46.0–53.658.3% 54.0–62.1one answer
unknown59074.2% 70.7–77.680.7% 77.5–84.183.4% 80.5–86.6one answer

True, false and unknown, closed world

By correct answer, Closed world
Correct answerItemsJevOpenAI Decisions API (GPT-6 Luna)GPT-6 LunaBest constant
false90088.9% 86.9–90.945.9% 42.9–49.363.8% 60.6–66.9one answer
true90089.7% 87.7–91.666.7% 63.6–69.866.2% 63.1–69.2one answer

Paraphrased vs templated rules, open world

By wording, Open world
WordingItemsJevOpenAI Decisions API (GPT-6 Luna)GPT-6 LunaBest constant
templated1,29886.1% 84.2–87.956.8% 54.1–59.667.6% 64.8–70.234.5%
paraphrased (NatLang)50278.1% 74.5–82.146.8% 42.4–51.254.8% 50.4–59.435.9%

Paraphrased vs templated rules, closed world

By wording, Closed world
WordingItemsJevOpenAI Decisions API (GPT-6 Luna)GPT-6 LunaBest constant
templated1,30694.0% 92.8–95.360.6% 58.1–63.267.3% 64.8–69.950.2%
paraphrased (NatLang)49476.7% 73.1–80.444.7% 40.7–49.458.9% 54.7–63.450.6%

Answer by answer: Jev, OpenAI Decisions API (GPT-6 Luna) and GPT-6 Luna on the same problems

Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
OpenAI Decisions API (GPT-6 Luna), open-world task
Correct ↓ · said →truefalseunknown
true3002303
false18196391
unknown5559476
GPT-6 Luna (reasoning off), open-world task
Correct ↓ · said →truefalseunknown
true3533249
false7308290
unknown4553492

Jev said true on 37% of problems; the correct answer is true on 34%. OpenAI Decisions API (GPT-6 Luna) said unknown on 65% of problems; the correct answer is unknown on 33%. GPT-6 Luna (reasoning off) said unknown on 57% of problems; the correct answer is unknown on 33%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
OpenAI Decisions API (GPT-6 Luna), closed-world task
Correct ↓ · said →truefalse
true600300
false487413
GPT-6 Luna (reasoning off), closed-world task
Correct ↓ · said →truefalse
true596304
false326574

Jev said true on 50% of problems; the correct answer is true on 50%. OpenAI Decisions API (GPT-6 Luna) said true on 60% of problems; the correct answer is true on 50%. GPT-6 Luna (reasoning off) said true on 51% of problems; the correct answer is true on 50%.

Where OpenAI Decisions API (GPT-6 Luna) and GPT-6 Luna err

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

OpenAI Decisions API (GPT-6 Luna), open-world task

OpenAI Decisions API (GPT-6 Luna), open-world task
Correct ↓ · said →truefalseunknownRecall
true300230349.6%
false1819639132.4%
unknown555947680.7%

OpenAI Decisions API (GPT-6 Luna) answered false 14.3%, true 20.7%, unknown 65.0% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

OpenAI Decisions API (GPT-6 Luna), closed-world task

OpenAI Decisions API (GPT-6 Luna), closed-world task
Correct ↓ · said →truefalseRecall
true60030066.7%
false48741345.9%

OpenAI Decisions API (GPT-6 Luna) answered false 39.6%, true 60.4% of the time; the correct answers are false 50.0%, true 50.0%.

OpenAI Decisions API (GPT-6 Luna) at depth 0 and depth 5

OpenAI Decisions API (GPT-6 Luna), open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9703
false07327
unknown1099
OpenAI Decisions API (GPT-6 Luna), open-world task, depth 5
Correct ↓ · said →truefalseunknown
true30071
false11981
unknown101661

Open-world task. At depth 0, OpenAI Decisions API (GPT-6 Luna) was right on 90% of problems (the green diagonal); at depth 5, on 38%. The answer whose share moved most is unknown: 43% of its answers at depth 0, 74% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
OpenAI Decisions API (GPT-6 Luna), closed-world task, depth 0
Correct ↓ · said →truefalse
true1473
false5199
OpenAI Decisions API (GPT-6 Luna), closed-world task, depth 5
Correct ↓ · said →truefalse
true7179
false10743

Closed-world task. At depth 0, OpenAI Decisions API (GPT-6 Luna) was right on 82% of problems (the green diagonal); at depth 5, on 38%. The answer whose share moved most is true: 66% of its answers at depth 0, 59% at depth 5, against 50% and 50% of the correct answers.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

GPT-6 Luna, open-world task

GPT-6 Luna (reasoning off), open-world task
Correct ↓ · said →truefalseunknownRecall
true353324958.3%
false730829050.9%
unknown455349283.4%

GPT-6 Luna answered false 20.2%, true 22.5%, unknown 57.3% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

GPT-6 Luna, closed-world task

GPT-6 Luna (reasoning off), closed-world task
Correct ↓ · said →truefalseRecall
true59630466.2%
false32657463.8%

GPT-6 Luna answered false 48.8%, true 51.2% of the time; the correct answers are false 50.0%, true 50.0%.

GPT-6 Luna at depth 0 and depth 5

GPT-6 Luna (reasoning off), open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9604
false28513
unknown1099
GPT-6 Luna (reasoning off), open-world task, depth 5
Correct ↓ · said →truefalseunknown
true36065
false03566
unknown131262

Open-world task. At depth 0, GPT-6 Luna was right on 93% of problems (the green diagonal); at depth 5, on 46%. The answer whose share moved most is unknown: 39% of its answers at depth 0, 67% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
GPT-6 Luna (reasoning off), closed-world task, depth 0
Correct ↓ · said →truefalse
true1482
false4146
GPT-6 Luna (reasoning off), closed-world task, depth 5
Correct ↓ · said →truefalse
true7377
false8961

Closed-world task. At depth 0, GPT-6 Luna was right on 98% of problems (the green diagonal); at depth 5, on 45%. The answer whose share moved most is true: 51% of its answers at depth 0, 54% at depth 5, against 50% and 50% of the correct answers.

OpenAI Decisions API (GPT-6 Luna) and GPT-6 Luna across every breakdown

For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownOpenAI Decisions API (GPT-6 Luna)GPT-6 Luna
Proof depthdepth 4 35.3% · depth 0 89.7% (54.4 points)depth 4 44.6% · depth 0 93.3% (48.8 points)
Gold answerfalse 32.4% · unknown 80.7% (48.3 points)false 50.9% · unknown 83.4% (32.5 points)
Theory kindattribute 51.3% · relation 57.9% (6.6 points)attribute 62.5% · relation 66.3% (3.8 points)
Negation in the theorywithout negation 54.0% · with negation 54.0% (0.1 points)without negation 61.2% · with negation 67.1% (5.9 points)
Negated statementplain statement 50.5% · negated statement 57.5% (7.0 points)plain statement 62.3% · negated statement 65.8% (3.5 points)
Question strategyinv-proof 32.4% · inv-random 100.0% (67.6 points)inv-proof 50.9% · inv-random 100.0% (49.1 points)
Paraphrased rulesparaphrased (NatLang) 46.8% · templated 56.8% (10.0 points)paraphrased (NatLang) 54.8% · templated 67.6% (12.9 points)
Theory's deepest proof4 38.9% · 0 84.5% (45.6 points)4 40.7% · 0 87.3% (46.6 points)
Theory length110+ 44.5% · 0-49 80.2% (35.7 points)110+ 52.6% · 0-49 91.5% (38.9 points)
Number of rules6-7 50.4% · 0-2 77.5% (27.1 points)3-5 59.1% · 0-2 90.2% (31.1 points)
Number of facts13+ 46.5% · 0-3 70.4% (23.8 points)13+ 56.6% · 0-3 81.4% (24.8 points)
Proof size5+ 21.9% · 0-1 81.8% (59.9 points)5+ 33.6% · 0-1 85.2% (51.6 points)

Exactly what OpenAI Decisions API (GPT-6 Luna) and GPT-6 Luna were asked

One open-world problem at depth 0, as each model received it. The correct answer is unknown.

The bear is green. Green things are not young.

Statement: The bear is not big.
Jevright answered unknown (100.0%); correct answer unknown

POST https://api.typesafe.ai/v1/systemone

Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%. Its separate confidence: 100.0%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "jev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "jev-1.13.0",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 1,
      "probabilities": {
        "false": 0,
        "true": 0,
        "unknown": 1
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 460,
    "output_tokens": 38
  }
}
OpenAI Decisions API (GPT-6 Luna)right answered unknown (100.0%); correct answer unknown

POST https://api.openai.com/v1/decisions

OpenAI's Decisions API, model gpt-6-luna. The problem is the input; the question is one choice question whose instructions are the question's and whose choices are the options, each with its meaning. The answer carries the chosen option, a probability for each option and a separate confidence.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%. Its separate confidence: 100.0%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "gpt-6-luna",
  "input": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.",
  "questions": [
    {
      "type": "choice",
      "name": "Decision",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "choices": [
        {
          "value": "true",
          "description": "The statement follows from the facts and rules."
        },
        {
          "value": "false",
          "description": "The negation of the statement follows from the facts and rules."
        },
        {
          "value": "unknown",
          "description": "Neither the statement nor its negation follows from the facts and rules."
        }
      ]
    }
  ]
}

Response (the saved record)

{
  "model": "gpt-6-luna",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 1,
      "probabilities": {
        "false": 0,
        "true": 0,
        "unknown": 1
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 249,
    "input_tokens_details": {
      "cache_write_tokens": 0,
      "cached_tokens": 0
    },
    "output_tokens": 0,
    "output_tokens_details": {
      "reasoning_tokens": 0
    },
    "total_tokens": 249
  }
}
GPT-6 Lunaright answered unknown; correct answer unknown

POST https://api.openai.com/v1/chat/completions

GPT-6 Luna with reasoning off (reasoning_effort none, its lowest setting), as a direct classifier. One chat message carries the problem, the question and every option with its meaning; the strict JSON schema only allows one of the options as the answer.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

Options:
- true: The statement follows from the facts and rules.
- false: The negation of the statement follows from the facts and rules.
- unknown: Neither the statement nor its negation follows from the facts and rules.

Reply with a JSON object {"answer": <option>} where <option> is exactly one of: true, false, unknown.

The answers it can give

true
false
unknown
The full request and response, exactly as sent and recorded

Request

{
  "model": "gpt-6-luna",
  "max_completion_tokens": 1024,
  "reasoning_effort": "none",
  "messages": [
    {
      "role": "user",
      "content": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.\n\nReply with a JSON object {\"answer\": <option>} where <option> is exactly one of: true, false, unknown."
    }
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "decision",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "answer": {
            "type": "string",
            "enum": [
              "true",
              "false",
              "unknown"
            ]
          }
        },
        "required": [
          "answer"
        ],
        "additionalProperties": false
      }
    }
  }
}

Response (the saved record)

{
  "model": "gpt-6-luna",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "raw": null,
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 221,
    "output_tokens": 11,
    "stop_reason": "stop"
  }
}

Time and money

OpenAI Decisions API (GPT-6 Luna)'s median open-world decision took 219 ms against Jev's 189 ms. At list price OpenAI Decisions API (GPT-6 Luna) cost $34 per million decisions against Jev's $23: 1.5 times as much per decision and 2.3 times as much per correct decision.GPT-6 Luna's median open-world decision took 825 ms against Jev's 189 ms, and 1 of its 3,600 decisions took ten seconds or more (the slowest 10 s). At list price GPT-6 Luna cost $37 per million decisions against Jev's $23: 1.6 times as much per decision and 2.1 times as much per correct decision.

Every decision's time

  • Jev
  • OpenAI Decisions API (GPT-6 Luna)
  • GPT-6 Luna(reasoning off)
05001,0001,5002,0000.1 s0.3 s1 s3 s10 s30 s100 stime per decision (log scale)1 GPT-6 Luna decisions ≥ 10 sJev median 179 msOpenAI Decisions API (GPT-6 Luna) median 215 msGPT-6 Luna median 875 ms

Average time per decision, by proof depth

Open world: true, false or unknown

0 s1 s2 s012345proof depth (inference steps)207ms202ms209ms203ms207ms209ms255ms249ms264ms288ms278ms258ms894ms890ms902ms920ms876ms894ms

Closed world: true or false

0 s1 s2 s012345proof depth (inference steps)177ms177ms175ms174ms174ms178ms247ms246ms248ms248ms244ms237ms993ms1.0s974ms1.0s990ms981ms
ModelTaskDepth 0Depth 1Depth 2Depth 3Depth 4Depth 5
JevOpen world207 ms202 ms209 ms203 ms207 ms209 ms
JevClosed world177 ms177 ms175 ms174 ms174 ms178 ms
OpenAI Decisions API (GPT-6 Luna)Open world255 ms249 ms264 ms288 ms278 ms258 ms
OpenAI Decisions API (GPT-6 Luna)Closed world247 ms246 ms248 ms248 ms244 ms237 ms
GPT-6 LunaOpen world894 ms890 ms902 ms920 ms876 ms894 ms
GPT-6 LunaClosed world993 ms1.0 s974 ms1.0 s990 ms981 ms

List-price cost per million decisions, by proof depth

Open world: true, false or unknown

$0$10$20$30$40012345proof depth (inference steps)$23$23$23$24$24$24$34$34$34$35$36$36$36$37$37$38$39$39

Closed world: true or false

$0$10$20$30$40012345proof depth (inference steps)$22$22$22$23$23$23$32$32$32$33$34$35$34$35$35$36$37$37
ModelTaskDepth 0Depth 1Depth 2Depth 3Depth 4Depth 5
JevOpen world$23$23$23$24$24$24
JevClosed world$22$22$22$23$23$23
OpenAI Decisions API (GPT-6 Luna)Open world$34$34$34$35$36$36
OpenAI Decisions API (GPT-6 Luna)Closed world$32$32$32$33$34$35
GPT-6 LunaOpen world$36$37$37$38$39$39
GPT-6 LunaClosed world$34$35$35$36$37$37

Times are wall time per request; OpenAI Decisions API (GPT-6 Luna) timed in its scored run with 8 requests in flight, the others one request at a time. Costs are list prices times the tokens each API reported. Cost per decision for every hosted model · Speed, size and memory.

Consistency and calibration

OpenAI Decisions API (GPT-6 Luna)

OpenAI Decisions API (GPT-6 Luna) was not asked a second time, so its consistency is not measured.

Calibration on the open world task: OpenAI Decisions API (GPT-6 Luna)'s probability for its own answer averaged 88.2% against 54.0% accuracy; expected calibration error 34.7 points.

Calibration on the closed world task: OpenAI Decisions API (GPT-6 Luna)'s probability for its own answer averaged 87.4% against 56.3% accuracy; expected calibration error 31.2 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.57 on the open world task against Jev's 0.85; 0.62 on the closed world task against Jev's 0.86.

GPT-6 Luna

Asked every open world question a second time, GPT-6 Luna gave the same answer on 89.7% of 1,800 items (185 changed; Gwet's AC1 0.855, interval 0.837 to 0.876). Accuracy was 64.1% the first time and 62.8% the second.

Asked every closed world question a second time, GPT-6 Luna gave the same answer on 90.6% of 1,800 items (169 changed; Gwet's AC1 0.812, interval 0.784 to 0.838). Accuracy was 65.0% the first time and 64.8% the second.

We asked GPT-6 Luna, like every model, for its answer only, so no confidence was recorded and its calibration is not measured here. A separate probe read its log-probabilities: see its calibration in the log-probability probe, on the OpenAI Decisions API page.

Size, hardware and speed

ModelKindParametersRan onMedian time per decisionMemory peak
Jevhosted decision modelundisclosedvendor's servers189 msnot applicable (hosted)
OpenAI Decisions API (GPT-6 Luna)hosted decision modelundisclosedvendor's servers219 msnot applicable (hosted)
GPT-6 LunaLLM classifierundisclosedvendor's servers825 msnot applicable (hosted)

Times are open-world, one request at a time except OpenAI Decisions API (GPT-6 Luna), timed in its scored run with 8 requests in flight. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.

OpenAI Decisions API (GPT-6 Luna)

Parameters
undisclosed
Architecture
undisclosed (GPT-6 Luna through OpenAI's dedicated Decisions endpoint)
Weights
not available (hosted)
Where it ran
OpenAI API, POST /v1/decisions, model gpt-6-luna, one choice question per request; generally available 2026-10-22
Maker's stated hardware
not applicable (hosted)
Price
$0.10 per million input tokens; cached input $0.01; output free
Time per decision, open world
median 219 ms, 90th percentile 379 ms, over the network
Time per decision, closed world
median 210 ms, 90th percentile 351 ms, over the network
Source
developers.openai.com/api/docs/guides/decisions

GPT-6 Luna

Parameters
undisclosed
Architecture
undisclosed
Weights
not available (hosted)
Where it ran
OpenAI Chat Completions API, reasoning_effort none, strict JSON schema
Maker's stated hardware
not applicable (hosted)
Price
$0.10 input / $0.50 output per million tokens
Time per decision, open world
median 825 ms, 90th percentile 1,165 ms, over the network
Time per decision, closed world
median 929 ms, 90th percentile 1,268 ms, over the network
Source
developers.openai.com/api/docs/models/gpt-6-luna

The models

Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

OpenAI Decisions API (GPT-6 Luna) The OpenAI Decisions API is OpenAI's dedicated endpoint for typed answers (POST /v1/decisions, model gpt-6-luna). We sent one choice question per item, with the problem as the input, the question's instructions and each option with its description, the same wording every other model got. It returns the chosen option, a probability for each option and a separate confidence. This is a different entry from GPT-6 Luna through the chat endpoint. Its size and architecture are not published.

GPT-6 Luna GPT-6 Luna through the chat endpoint is a hosted large language model made by OpenAI. We used it as a classifier: one request per item through the Chat Completions API, with reasoning effort set to none (its lowest setting) and a strict JSON schema that allows only the task's options. Its size and architecture are not published.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-16, from the harness's working tree at commit b324cd3 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.