Jev vs GPT-6 Luna: a decision model against an LLM classifier
Jev is built to make one decision per request. GPT-6 Luna is a general large language model, used here as a classifier with reasoning off. Both answered the same 3,600 problems.
GPT-6 Luna ran with reasoning effort set to none, its lowest setting, under the same one-request protocol as every model here: the same question, the same options, one answer per item. A different reasoning setting would be a different model, scored separately.
OpenAI's new Decisions API is built on a version of GPT-6 Luna, so these results preview the accuracy of a Luna-based decision model on multi-step reasoning. They are not a test of the API itself. What this means for the OpenAI Decisions API.
Jev minus GPT-6 Luna: +24.3 points on the same 1,800 items, 95% interval +21.9 to +26.6: Jev is clearly ahead.
Jev vs GPT-6 Luna by proof depth
Each point is about 300 problems that need that many inference steps, with its 95% interval.
Open world: true, false or unknown
Jev
GPT-6 Luna(reasoning off)
Closed world: true or false
Jev
GPT-6 Luna(reasoning off)
Jev minus GPT-6 Luna by depth, open world
−100+43
depth 0300 items
+4.3
depth 1302 items
+9.9
depth 2303 items
+24.4
depth 3303 items
+18.2
depth 4303 items
+27.4
depth 5289 items
+34.9
Jev minus GPT-6 Luna by depth, closed world
−100+51
depth 0300 items
+1.3
depth 1300 items
+17.0
depth 2300 items
+26.7
depth 3300 items
+28.0
depth 4300 items
+28.0
depth 5300 items
+44.7
Paired differences: positive means Jev was more accurate on the same items.
Accuracy by depth, open world
By depth, Open world
Proof depth
Items
Jev
GPT-6 Luna
Best constant
depth 0
300
97.7%96.0–99.3
93.3%90.3–96.0
33.3%
depth 1
302
87.7%83.8–91.4
77.8%73.2–82.8
33.4%
depth 2
303
84.8%80.2–88.4
60.4%54.5–66.0
33.3%
depth 3
303
79.9%75.2–84.5
61.7%56.8–67.7
33.3%
depth 4
303
71.9%66.7–77.2
44.6%38.6–49.5
33.3%
depth 5
289
81.0%76.5–85.5
46.0%40.1–51.6
34.9%
Accuracy by depth, closed world
By depth, Closed world
Proof depth
Items
Jev
GPT-6 Luna
Best constant
depth 0
300
99.3%98.3–100.0
98.0%96.3–99.3
50.0%
depth 1
300
93.7%91.0–96.3
76.7%72.3–81.0
50.0%
depth 2
300
92.0%88.7–94.7
65.3%60.0–70.7
50.0%
depth 3
300
85.0%81.3–89.3
57.0%51.7–63.0
50.0%
depth 4
300
76.3%71.7–81.0
48.3%43.0–53.7
50.0%
depth 5
300
89.3%85.7–92.3
44.7%38.7–50.3
50.0%
What GPT-6 Luna got right and wrong, depth by depth
At each proof depth, GPT-6 Luna's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.
How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)
Depth 0, open-world task
93.3%
right on 280 of 300 problems wrong; chance 33% (dashed line)
Alan is not young.
Correct answer: false · Model said: false (100.0% sure, probe)
paraphrasedThe problem and its proof
Alan is young and green. This makes him cold to others. Eric seems to be round. Even though Gary is big and rough he is sometimes round and kind. Harry is a young, round shaped young man who is also very cold. Truth be told, every big, rough person out there that is green, will also be colored blue. Big people who are red in color are almost always young. If you come across a young, blue person you can count on them being nice to you. People turn green when they are feeling blue. Nice people that are very green and even round shaped will be very young. Any person who can be red and blue at the same time is cold. If you ever meet someone who is both green and blue while being nice, then they'll be big, too.
Stated in the text: "Alan is young."
The lion does not visit the bear.
Correct answer: true · Model said: true (100.0% sure, probe)
The problem and its proof
The bald eagle visits the tiger. The bear visits the bald eagle. The lion does not visit the bear. The tiger needs the lion. If someone needs the lion then they do not need the tiger. If someone sees the bear and they see the lion then they do not need the bear.
Stated in the text: "The lion does not visit the bear."
Bob is blue.
Correct answer: unknown · Model said: unknown (100.0% sure, probe)
paraphrasedThe problem and its proof
Alan woke up one day to find he had become round. He was still rough, red, and green, but now he was also big. Bob seems to be round. The young person who is always feeling cold is named Charlie. Dave who is young and nice is also quite big and feels green. Not a single young person is not also kind. Even though young people are rough, they are still very red. Watch out for young people that are kind for they are rough. Nice people with red and rough skin are green with envy. A young person who is big and rough and big is also usually round. When someone is blue, nice and young they will also be as big as the sun.
No proof: neither the statement nor its negation follows.
Charlie is round.
Correct answer: true · Model said: true (100.0% sure, probe)
paraphrasedThe problem and its proof
Charlie is a round and rough around the edges, and he is also big. Eric is young and green. Even though he is round he is cold outside. Fred is a young, round shaped young man who is also very cold. Gary may be round, but he is also kind. Do not fear big cold people for they are often kind. Someone who is a young age and looks like they are round are also red. Any human who is rough, green, and kind, has a blue personality. Any red sort of person is a nice person. Nice and green people are often found to be big as well.
Stated in the text: "Charlie is round."
Depth 1, open-world task
77.8%
right on 235 of 302 problems wrong; chance 33% (dashed line)
The tiger is not kind.
Correct answer: true · Model said: true (100.0% sure, probe)
The problem and its proof
The tiger is nice. The tiger is red. The tiger is round. If someone is round then they are not kind.
The tiger is round.→If someone is round then they are not kind.→The tiger is not kind.
Gary is rough.
Correct answer: unknown · Model said: unknown (100.0% sure, probe)
The problem and its proof
Bob is red. Erin is not green. Gary is young. Rough people are green. If someone is young then they are green. All furry people are big. Cold, furry people are young. If someone is red then they are furry. All furry, red people are rough.
No proof: neither the statement nor its negation follows.
Eric is not round.
Correct answer: false · Model said: unknown (98.9% sure, probe)
paraphrasedThe problem and its proof
Alan, who is relatively young, is also pretty big and tends to be cold. No one knows Charlie like me and he is a kind guy, very round in the belly and green as grass. Eric has big, red cheeks with a rough texture. He always wear green sweaters because he is easily cold. That guy Gary sure is nice. Rough people who are red in color and big are usally round in shape. A rather big person is a red person. Young people that are turning blue from being cold will be rough looking. When someone is green and round, they can be rough but kind. People with big smiles and round eyes will have red hair. Is a kind person if feeling cold and looking red then they are usually young. Red people, who are also round and young, are often quite blue as well.
The proof that the statement's negation follows:
Eric is red. + Eric is rough. + Eric is big.→If someone is red and rough and big then they are round.→Eric is round.
Harry is white.
Correct answer: unknown · Model said: unknown (100.0% sure, probe)
The problem and its proof
Bob is not smart. Bob is white. Harry is green. Round people are white. White people are blue. If Bob is white and Bob is blue then Bob is round. Round, white people are not quiet. If someone is blue and not white then they are smart. If someone is white then they are not smart. If someone is smart and not quiet then they are green. All smart, round people are not green.
No proof: neither the statement nor its negation follows.
Depth 2, open-world task
60.4%
right on 183 of 303 problems wrong; chance 33% (dashed line)
The squirrel is young.
Correct answer: false · Model said: false (84.2% sure, probe)
The problem and its proof
The dog sees the squirrel. The dog visits the rabbit. The rabbit likes the squirrel. The rabbit sees the dog. The squirrel likes the dog. The squirrel sees the dog. The squirrel does not visit the rabbit. If someone sees the dog and the dog visits the rabbit then they visit the dog. If someone visits the dog then they are not young.
The proof that the statement's negation follows:
The squirrel sees the dog. + The dog visits the rabbit.→If someone sees the dog and the dog visits the rabbit then they visit the dog.→The squirrel visits the dog.
→If someone visits the dog then they are not young.→The squirrel is not young.
The bear eats the mouse.
Correct answer: true · Model said: true (100.0% sure, probe)
The problem and its proof
The bear sees the mouse. The mouse chases the bear. If something is big then it eats the mouse. If something sees the mouse then it is big.
The bear sees the mouse.→If something sees the mouse then it is big.→The bear is big.
→If something is big then it eats the mouse.→The bear eats the mouse.
The lion visits the mouse.
Correct answer: true · Model said: unknown
The problem and its proof
The lion does not need the tiger. The mouse visits the tiger. The squirrel visits the mouse. The tiger eats the lion. If someone visits the mouse then they visit the lion. If someone visits the lion then the lion visits the mouse.
The squirrel visits the mouse.→If someone visits the mouse then they visit the lion.→The squirrel visits the lion.
→If someone visits the lion then the lion visits the mouse.→The lion visits the mouse.
Gary is not blue.
Correct answer: true · Model said: unknown (97.6% sure, probe)
The problem and its proof
Erin is cold. Fiona is cold. Fiona is not quiet. Fiona is rough. Gary is nice. Gary is not quiet. Gary is smart. Smart things are big. Big things are smart. If something is cold and not quiet then it is smart. All big, quiet things are smart. If something is nice and cold then it is not smart. All quiet, big things are not nice. Rough things are not nice. All big things are not blue.
Gary is smart.→Smart things are big.→Gary is big.
→All big things are not blue.→Gary is not blue.
Depth 3, open-world task
61.7%
right on 187 of 303 problems wrong; chance 33% (dashed line)
Harry is green.
Correct answer: unknown · Model said: unknown (100.0% sure, probe)
paraphrasedThe problem and its proof
That guy Alan sure is nice. Bob is round but he is also nice and kind. People that know Charlie will tell you he is rough and cold. But he can also be kind and red despite being blue because he is so young. That guy Harry sure is nice. People who are very nice and easily flush red are often wearing green concealer. If you're truly nice, cold, and blue, you're kind, too. Big, kind folks are green ones. Round people who are kind tend to be nice. Young and red people look round. A nice and green man or woman is also red in color.
No proof: neither the statement nor its negation follows.
Eric is not green.
Correct answer: false · Model said: unknown (98.3% sure, probe)
paraphrasedThe problem and its proof
Bob is cold to his ex-wife. She knows he is round and red. He is kind to everyone else. Eric is known for being big and rough, but few know he is also always red with cold. For being so cold, it's good Gary can remain nice. For being so cold, it's good Harry can remain nice. Round people who feel blue and are green in color are often young in age. Rough people who are red in color and big are usally round in shape. Kind red people are green on the inside. A rather big person is a red person. A nice but rough person will also be green and cold. A big and round individual is sure to be a kind individual, too. A person that is red, rough and kind often feels cold.
The proof that the statement's negation follows:
Eric is red. + Eric is rough. + Eric is big.→If someone is red and rough and big then they are round.→Eric is round.
+ Eric is big.→If someone is big and round then they are kind.→Eric is kind.
+ Eric is red.→If someone is kind and red then they are green.→Eric is green.
The cow does not see the cat.
Correct answer: unknown · Model said: false (97.6% sure, probe)
The problem and its proof
The bear does not chase the cat. The bear eats the cat. The bear eats the cow. The bear is green. The bear is not nice. The bear sees the dog. The cat is nice. The cat is red. The cat sees the bear. The cow eats the dog. The dog is kind. If someone is kind and they see the dog then the dog chases the cow. If someone eats the bear and the bear chases the dog then the dog eats the cow. If someone sees the bear then they see the cow. If someone sees the cat and the cat sees the bear then they eat the dog. If someone is nice and they eat the cow then the cow does not chase the cat. If someone sees the cow then they see the cat. If someone is green then they see the bear. If someone eats the dog then they are red.
No proof: neither the statement nor its negation follows.
Charlie is not young.
Correct answer: true · Model said: true (99.9% sure, probe)
The problem and its proof
Charlie is smart. Fiona is not young. All white, round people are big. If someone is young then they are big. All young, round people are big. If Fiona is white then Fiona is big. All round people are not young. If Charlie is cold then Charlie is round. If someone is smart then they are cold. Smart people are blue.
Charlie is smart.→If someone is smart then they are cold.→Charlie is cold.
→If Charlie is cold then Charlie is round.→Charlie is round.
→All round people are not young.→Charlie is not young.
Depth 4, open-world task
44.6%
right on 135 of 303 problems wrong; chance 33% (dashed line)
Anne is young.
Correct answer: true · Model said: true
The problem and its proof
Anne is big. Charlie is big. Charlie is blue. Charlie is cold. Charlie is green. Charlie is nice. Charlie is rough. Charlie is young. Fiona is big. Fiona is green. Gary is big. Gary is cold. Gary is green. Gary is nice. Gary is rough. Gary is young. All rough, young people are nice. All big, cold people are rough. If someone is rough and blue then they are young. Big people are cold. Rough people are blue.
Anne is big.→Big people are cold.→Anne is cold.
+ Anne is big.→All big, cold people are rough.→Anne is rough.
→Rough people are blue.→Anne is blue.
→If someone is rough and blue then they are young.→Anne is young.
Charlie is not kind.
Correct answer: false · Model said: unknown (100.0% sure, probe)
paraphrasedThe problem and its proof
Bob is a round and rough around the edges, and he is also big. Young Charlie was rough in demeanor. He was cold, which made his skin blue despite being big. Gary seems big and rough, but he's actually quite kind. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Folks who are rough and tumble yet gentle and kind are blue. Someone who has young, nice and cold qualities will possess a red quality, too. Anyone feeling cold is bound to act nice. A round person who is on the young side is also likely to treat people in a kind manner. Tom is a rough, young person to know, he is very green at his job but he is very round from his weight. Nice people who feel blue often look green when ill.
The proof that the statement's negation follows:
Charlie is cold.→If someone is cold then they are nice.→Charlie is nice.
+ Charlie is blue.→If someone is nice and blue then they are green.→Charlie is green.
+ Charlie is rough. + Charlie is young.→If someone is rough and young and green then they are round.→Charlie is round.
+ Charlie is young.→If someone is round and young then they are kind.→Charlie is kind.
Charlie is not blue.
Correct answer: true · Model said: true
The problem and its proof
Bob is cold. Bob is quiet. Charlie is white. Dave is quiet. Dave is white. Gary is blue. Gary is quiet. All white, green things are cold. White things are cold. Red, smart things are not blue. All red things are green. Green, white things are quiet. Smart, white things are red. If Charlie is quiet then Charlie is red. If something is blue and cold then it is smart. All white, cold things are smart.
Charlie is white.→White things are cold.→Charlie is cold.
+ Charlie is white.→All white, cold things are smart.→Charlie is smart.
+ Charlie is white.→Smart, white things are red.→Charlie is red.
→Red, smart things are not blue.→Charlie is not blue.
Bob is nice.
Correct answer: true · Model said: unknown (100.0% sure, probe)
The problem and its proof
Anne is furry. Anne is red. Anne is smart. Bob is red. Dave is blue. Dave is kind. Erin is white. Furry people are kind. Furry, blue people are kind. All red people are furry. All nice, red people are smart. Kind, nice people are smart. All furry people are kind. White people are kind. If someone is white and red then they are nice. If Bob is kind then Bob is white.
Bob is red.→All red people are furry.→Bob is furry.
→Furry people are kind.→Bob is kind.
→If Bob is kind then Bob is white.→Bob is white.
+ Bob is red.→If someone is white and red then they are nice.→Bob is nice.
Depth 5, open-world task
46.0%
right on 133 of 289 problems wrong; chance 33% (dashed line)
Anne is not big.
Correct answer: false · Model said: unknown (100.0% sure, probe)
The problem and its proof
Anne is blue. Anne is nice. Bob is kind. Bob is nice. Fiona is blue. Fiona is furry. Fiona is kind. Fiona is nice. Fiona is smart. Gary is blue. Gary is furry. Gary is smart. Big, furry things are green. Kind, nice things are big. All smart, green things are kind. All nice things are furry. If Anne is kind and Anne is big then Anne is green. If Fiona is blue then Fiona is green. Furry, green things are smart. Furry things are green.
The proof that the statement's negation follows:
Anne is nice.→All nice things are furry.→Anne is furry.
→Furry things are green.→Anne is green.
→Furry, green things are smart.→Anne is smart.
→All smart, green things are kind.→Anne is kind.
+ Anne is nice.→Kind, nice things are big.→Anne is big.
The cat does not see the cat.
Correct answer: false · Model said: false (95.2% sure, probe)
The problem and its proof
The cat does not chase the rabbit. The cat is cold. The cat is red. The cat sees the rabbit. The cat sees the tiger. The mouse is big. The mouse is red. The rabbit does not chase the cat. The rabbit likes the cat. The rabbit does not like the mouse. The tiger likes the mouse. If something likes the mouse then it chases the cat. If something sees the mouse then it chases the cat. If something chases the mouse then the mouse does not like the rabbit. If something chases the cat then it sees the cat. If the tiger sees the cat then the cat sees the mouse.
The proof that the statement's negation follows:
The tiger likes the mouse.→If something likes the mouse then it chases the cat.→The tiger chases the cat.
→If something chases the cat then it sees the cat.→The tiger sees the cat.
→If the tiger sees the cat then the cat sees the mouse.→The cat sees the mouse.
→If something sees the mouse then it chases the cat.→The cat chases the cat.
→If something chases the cat then it sees the cat.→The cat sees the cat.
The cow is not big.
Correct answer: true · Model said: true
The problem and its proof
The cat is not round. The cow is blue. The cow is kind. The cow likes the cat. The cow sees the rabbit. The rabbit chases the cow. The rabbit is nice. The tiger does not chase the cow. The tiger does not like the rabbit. The tiger sees the rabbit. If something chases the cow then the cow chases the cat. If something chases the rabbit then it is not big. If something likes the cow then the cow is nice. If something is kind then it sees the rabbit. If something chases the cat then the cat likes the cow. If the rabbit chases the tiger then the rabbit does not see the cow. If something is nice and it sees the rabbit then it chases the rabbit. If something is big then it is round.
The rabbit chases the cow.→If something chases the cow then the cow chases the cat.→The cow chases the cat.
→If something chases the cat then the cat likes the cow.→The cat likes the cow.
→If something likes the cow then the cow is nice.→The cow is nice.
+ The cow sees the rabbit.→If something is nice and it sees the rabbit then it chases the rabbit.→The cow chases the rabbit.
→If something chases the rabbit then it is not big.→The cow is not big.
Anne is quiet.
Correct answer: false · Model said: unknown (100.0% sure, probe)
The problem and its proof
Anne is smart. Charlie is not blue. Charlie is quiet. Erin is not nice. Erin is round. Harry is green. Harry is young. Smart things are green. If Anne is round then Anne is young. If something is blue and green then it is not quiet. If something is green and smart then it is round. If something is young then it is blue. Smart things are nice. If Harry is quiet and Harry is not smart then Harry is young.
The proof that the statement's negation follows:
Anne is smart.→Smart things are green.→Anne is green.
+ Anne is smart.→If something is green and smart then it is round.→Anne is round.
→If Anne is round then Anne is young.→Anne is young.
→If something is young then it is blue.→Anne is blue.
→If something is blue and green then it is not quiet.→Anne is not quiet.
The closed-world task (true or false), depth by depth
Depth 0, closed-world task
98.0%
right on 294 of 300 problems wrong; chance 50% (dashed line)
Charlie is not red.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
Charlie is red. If something is nice and not young then it is blue. If Charlie is red then Charlie is kind. All kind things are big.
Stated in the text: "Charlie is red."
The rabbit does not eat the bear.
Correct answer: false · Model said: false (99.9% sure, probe)
The problem and its proof
The bear is rough. The rabbit eats the bear. The squirrel visits the rabbit. If someone eats the bear then they need the bear. If someone needs the squirrel and they are not big then they need the bear. If someone eats the squirrel and they are not big then the squirrel is blue. If someone needs the rabbit then they visit the rabbit. If someone needs the bear then the bear visits the rabbit. If someone eats the squirrel then they are blue. If someone needs the rabbit then they eat the bear. If someone eats the squirrel and the squirrel is not rough then they are cold.
Stated in the text: "The rabbit eats the bear."
The rabbit is not cold.
Correct answer: true · Model said: true (100.0% sure, probe)
The problem and its proof
The dog needs the rabbit. The dog visits the rabbit. The rabbit eats the dog. If something visits the dog then it is young. Round things are young. If something needs the dog and it visits the dog then the dog eats the rabbit. If something is nice then it visits the dog. If something visits the rabbit and the rabbit is young then it eats the rabbit. If something is young then it eats the rabbit. If something is cold and it eats the dog then it needs the rabbit. If something visits the rabbit and the rabbit eats the dog then it is round.
No proof to show: under the closed world this answer holds because something cannot be derived.
The squirrel chases the cow.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
The cow chases the lion. The cow likes the rabbit. The cow sees the lion. The cow sees the squirrel. The lion chases the squirrel. The lion likes the cow. The lion likes the squirrel. The rabbit chases the lion. The rabbit likes the lion. The squirrel is big. The squirrel is rough. The squirrel likes the lion. The squirrel sees the lion. The squirrel sees the rabbit. If something sees the rabbit and the rabbit sees the cow then the rabbit chases the cow. If something is rough then it likes the rabbit. If something is cold and it chases the lion then the lion likes the squirrel. If something likes the cow then it is rough. If the lion is big and the lion chases the rabbit then the rabbit chases the cow. If something chases the rabbit then it is rough. If the lion sees the rabbit and the lion chases the cow then the cow is cold. If the lion is rough and the lion likes the rabbit then the rabbit likes the cow. If something is red and it sees the cow then the cow likes the rabbit.
No proof to show: under the closed world this answer holds because something cannot be derived.
Depth 1, closed-world task
76.7%
right on 230 of 300 problems wrong; chance 50% (dashed line)
The rabbit is not green.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
The rabbit is red. If the rabbit is not blue then the rabbit is green. Green things are round. Green, red things are cold.
→If the rabbit is not blue then the rabbit is green.→The rabbit is green.
Fred is round.
Correct answer: false · Model said: false (100.0% sure, probe)
paraphrasedThe problem and its proof
Charlie is a red, rough, green and blue man. For being so young, Dave is nice. He has rough, red hands from working all day and it makes him feel blue. Gary seems to be round. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Some cold people can be nice at times,and red at at other times. Anyone who is both red and green must be as cold as Christmas. I've noticed that nice people who have rough, green skin have a tendancy to be round. Most big and cold people are also rough. A big round young person is often blue. When a person is blue, rough and kind then it means they also are cold.
No proof to show: under the closed world this answer holds because something cannot be derived.
Harry is furry.
Correct answer: true · Model said: true (100.0% sure, probe)
The problem and its proof
Bob is nice. Bob is young. Charlie is kind. Charlie is smart. Dave is kind. Dave is white. Harry is nice. All young people are white. All young people are furry. All nice people are furry. If someone is nice then they are blue. Blue people are nice. If someone is smart and kind then they are nice. If someone is white then they are kind. All kind, furry people are smart. All kind, white people are young.
Harry is nice.→All nice people are furry.→Harry is furry.
Eric is nice.
Correct answer: true · Model said: false (41.6% sure, probe)
paraphrasedThe problem and its proof
Charlie can be rough and cold, but he is also green, an avid gardener. Eric is green, but often kind, even when he is blue and cold. The young person who is always feeling cold is named Fred. Gary is kind and nice and looks green. If a kind person is green and a bit blue then they are also nice. A blue colored person who is nice is a red person. A person who is kind and rough and blue is young. If someone is cold and blue, you will find that they are also quite young. A person who is green, blue, and red in color will be round shaped.
Eric is green. + Eric is kind. + Eric is blue.→If someone is green and kind and blue then they are nice.→Eric is nice.
Depth 2, closed-world task
65.3%
right on 196 of 300 problems wrong; chance 50% (dashed line)
The lion does not see the bear.
Correct answer: false · Model said: true (92.2% sure, probe)
The problem and its proof
The bear is rough. The lion chases the bear. If someone chases the lion and the lion is not young then the lion chases the bear. If someone sees the bear then the bear sees the lion. If the bear chases the lion then the bear is blue. If someone chases the bear then they eat the bear. If someone eats the bear then they see the bear. If the bear is red then the bear chases the lion. If someone chases the bear and they are young then the bear chases the lion. If someone eats the bear then the bear is big.
The lion chases the bear.→If someone chases the bear then they eat the bear.→The lion eats the bear.
→If someone eats the bear then they see the bear.→The lion sees the bear.
The mouse does not chase the cow.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
The cow eats the lion. The lion visits the mouse. The mouse is big. If something chases the cow then the cow chases the mouse. If something chases the mouse then the mouse chases the cow. If something visits the mouse then it chases the mouse.
The lion visits the mouse.→If something visits the mouse then it chases the mouse.→The lion chases the mouse.
→If something chases the mouse then the mouse chases the cow.→The mouse chases the cow.
The cow is young.
Correct answer: true · Model said: true (99.7% sure, probe)
The problem and its proof
The bald eagle chases the tiger. The bald eagle needs the tiger. The bald eagle sees the cow. The bald eagle sees the tiger. The cow chases the tiger. The cow is kind. The cow is red. The cow sees the bald eagle. The cow sees the tiger. The tiger chases the bald eagle. The tiger chases the cow. The tiger is blue. The tiger is kind. The tiger is red. The tiger needs the bald eagle. Young, kind people are green. If someone needs the tiger and the tiger needs the cow then the cow is young. If someone chases the tiger and they chase the bald eagle then the bald eagle sees the tiger. If someone sees the tiger then the tiger needs the cow. If someone needs the bald eagle and the bald eagle chases the tiger then the bald eagle sees the cow. If someone is red and they need the tiger then they are blue.
The bald eagle sees the tiger.→If someone sees the tiger then the tiger needs the cow.→The tiger needs the cow.
+ The bald eagle needs the tiger.→If someone needs the tiger and the tiger needs the cow then the cow is young.→The cow is young.
Eric is nice.
Correct answer: true · Model said: true (99.9% sure, probe)
paraphrasedThe problem and its proof
For being so cold, it's good Alan can remain nice. Eric is young and green. This makes him cold to others. Fred is a kind person and he is also often cold. That guy Harry sure is nice. All those who are nice and big are red. If you come across a young, blue person you can count on them being nice to you. People who are said to be big and nice are round. People who are young are also blue. A kind person will certainly be young.
Eric is young.→If someone is young then they are blue.→Eric is blue.
+ Eric is young.→If someone is blue and young then they are nice.→Eric is nice.
Depth 3, closed-world task
57.0%
right on 171 of 300 problems wrong; chance 50% (dashed line)
Fiona is blue.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
Anne is red. Erin is young. Fiona is kind. All white things are red. If something is quiet and white then it is kind. If something is quiet then it is red. If something is red then it is rough. All rough things are blue. All young things are blue.
No proof to show: under the closed world this answer holds because something cannot be derived.
Gary is not cold.
Correct answer: false · Model said: true (99.9% sure, probe)
paraphrasedThe problem and its proof
Red Charlie is still young while also very blue and green. Dave seems to be round. Eric, who is relatively young, is also pretty big and tends to be cold. Gary who is young and nice is also quite big and feels green. Not a single young person is not also kind. When you meet someone who is nice and kind, they wont feel blue. All those who are nice and big are red. A young person who is big and rough and big is also usually round. Young and red people look round. If a person is blue and red and big, then that person is cold.
Gary is young.→If someone is young then they are kind.→Gary is kind.
+ Gary is nice.→If someone is kind and nice then they are blue.→Gary is blue.
+ Gary is nice. + Gary is big.→If someone is nice and big then they are red.→Gary is red.
+ Gary is big.→If someone is blue and big and red then they are cold.→Gary is cold.
Eric is nice.
Correct answer: true · Model said: false (100.0% sure, probe)
paraphrasedThe problem and its proof
Charlie is green and cold too. Eric is a big guy who is colored red from a sunburn. He also has a cold. No one knows Fred like me and he is a kind guy, very round in the belly and green as grass. People here that are both big and kind while being cold are always red. Red people are almost always rough. A person who is cold and blue is nice. If you're truly nice, cold, and blue, you're kind, too. A big, round person who is kind is also red. Rough, cold people are blue. Nice people with red and rough skin are green with envy.
Eric is red.→If someone is red then they are rough.→Eric is rough.
+ Eric is cold.→If someone is rough and cold then they are blue.→Eric is blue.
+ Eric is cold.→If someone is cold and blue then they are nice.→Eric is nice.
Gary is not green.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
Bob is quiet. Bob is smart. Dave is rough. Erin is big. Erin is rough. Gary is round. Gary is white. Smart people are green. All rough people are smart. If someone is round then they are rough.
Gary is round.→If someone is round then they are rough.→Gary is rough.
→All rough people are smart.→Gary is smart.
→Smart people are green.→Gary is green.
Depth 4, closed-world task
48.3%
right on 145 of 300 problems wrong; chance 50% (dashed line)
Fiona is not smart.
Correct answer: true · Model said: true (100.0% sure, probe)
The problem and its proof
Bob is cold. Dave is big. Dave is furry. Dave is rough. Dave is round. Dave is smart. Fiona is round. If someone is smart then they are big. Furry people are smart. All furry people are smart. Smart people are furry. If someone is cold then they are furry. All smart people are rough.
No proof to show: under the closed world this answer holds because something cannot be derived.
Bob is not nice.
Correct answer: true · Model said: false (99.0% sure, probe)
The problem and its proof
Bob is big. Charlie is blue. Gary is nice. Harry is kind. If someone is blue and big then they are cold. Quiet people are nice. Blue people are cold. All cold people are green. All cold, green people are nice. If someone is nice then they are blue.
No proof to show: under the closed world this answer holds because something cannot be derived.
The lion is young.
Correct answer: true · Model said: false
The problem and its proof
The bear eats the cow. The bear likes the lion. The bear likes the mouse. The cow chases the bear. The cow chases the mouse. The cow eats the bear. The cow is nice. The lion chases the cow. The lion chases the mouse. The lion likes the bear. The mouse chases the cow. The mouse chases the lion. If someone is young then they are round. If someone eats the lion then they are young. If someone eats the bear then they eat the lion. If someone is nice and they chase the lion then they like the bear. If someone eats the lion then the lion eats the bear.
The cow eats the bear.→If someone eats the bear then they eat the lion.→The cow eats the lion.
→If someone eats the lion then the lion eats the bear.→The lion eats the bear.
→If someone eats the bear then they eat the lion.→The lion eats the lion.
→If someone eats the lion then they are young.→The lion is young.
Charlie is young.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
Charlie is rough. Dave is white. Erin is blue. Harry is kind. If something is white then it is cold. Big things are white. If something is cold then it is young. If something is young then it is blue. If something is rough then it is blue. White, cold things are big.
No proof to show: under the closed world this answer holds because something cannot be derived.
Depth 5, closed-world task
44.7%
right on 134 of 300 problems wrong; chance 50% (dashed line)
Erin is rough.
Correct answer: false · Model said: false (94.4% sure, probe)
The problem and its proof
Dave is blue. Dave is rough. Dave is young. Erin is blue. Erin is round. Erin is young. Fiona is rough. Gary is green. Gary is nice. Gary is young. Rough things are round. Green, nice things are quiet. Round, rough things are green. All quiet things are green. All green things are nice. All quiet things are rough. All rough, nice things are quiet. All quiet, green things are rough. If something is nice and quiet then it is blue.
No proof to show: under the closed world this answer holds because something cannot be derived.
The rabbit likes the bear.
Correct answer: true · Model said: false (100.0% sure, probe)
The problem and its proof
The bear is red. The rabbit is red. The rabbit likes the tiger. The squirrel is cold. The tiger eats the bear. The tiger is big. The tiger likes the rabbit. All cold people are kind. If someone is kind then they like the bear. If someone chases the rabbit and they are kind then the rabbit is kind. If someone likes the bear then they chase the rabbit. If someone chases the tiger and the tiger is red then the tiger chases the bear. If the squirrel is kind and the squirrel likes the rabbit then the rabbit is green. If the tiger eats the bear and the bear chases the rabbit then the bear is green.
The squirrel is cold.→All cold people are kind.→The squirrel is kind.
→If someone is kind then they like the bear.→The squirrel likes the bear.
→If someone likes the bear then they chase the rabbit.→The squirrel chases the rabbit.
→If someone chases the rabbit and they are kind then the rabbit is kind.→The rabbit is kind.
→If someone is kind then they like the bear.→The rabbit likes the bear.
The tiger does not see the tiger.
Correct answer: false · Model said: true (100.0% sure, probe)
The problem and its proof
The lion visits the mouse. The mouse sees the lion. The mouse sees the rabbit. The mouse visits the rabbit. The rabbit chases the lion. The rabbit chases the tiger. The rabbit is nice. The rabbit visits the lion. The tiger chases the lion. The tiger chases the mouse. If someone is green then they visit the tiger. If someone is nice then they visit the mouse. If someone sees the rabbit then they are kind. If someone sees the tiger then the tiger sees the rabbit. If someone is kind then they see the tiger. If the lion chases the mouse then the mouse sees the rabbit. If the rabbit does not chase the tiger then the tiger is cold.
The mouse sees the rabbit.→If someone sees the rabbit then they are kind.→The mouse is kind.
→If someone is kind then they see the tiger.→The mouse sees the tiger.
→If someone sees the tiger then the tiger sees the rabbit.→The tiger sees the rabbit.
→If someone sees the rabbit then they are kind.→The tiger is kind.
→If someone is kind then they see the tiger.→The tiger sees the tiger.
Fiona is rough.
Correct answer: false · Model said: false (100.0% sure, probe)
The problem and its proof
Bob is cold. Dave is big. Dave is furry. Dave is rough. Dave is round. Dave is smart. Fiona is round. If someone is smart then they are big. Furry people are smart. All furry people are smart. Smart people are furry. If someone is cold then they are furry. All smart people are rough.
No proof to show: under the closed world this answer holds because something cannot be derived.
What this means for the OpenAI Decisions API
At OpenAI DevDay on 29 September 2026, OpenAI announced the Decisions API, in limited preview with broader availability "in the coming days". OpenAI says it "uses a version of GPT-6 Luna": the model picks among answers the developer defines in advance, to classify, route or choose an agent's next action (The Decoder, Axios).
That is the shape of task we ran: GPT-6 Luna with reasoning off, one request per problem, and a fixed list of answers enforced with a strict JSON schema. So these results preview the accuracy a Luna-based decision model gets on multi-step reasoning. GPT-6 Luna answered 64.1% of open-world problems and 65.0% of closed-world ones correctly. On the open-world task it scored 93.3% when the answer is stated outright (depth 0) and 46.0% when it takes five chained inferences, where Jev scored 81.0% (+34.9 points on the same items). If your decisions need several steps of reasoning, the depth results above are the ones to read.
These are not results for the Decisions API itself. OpenAI uses a specialized version of Luna, and its accuracy may differ. As of 1 October 2026 OpenAI had published no Decisions API documentation. We will benchmark the API itself when we can get access.
On speed, OpenAI's own chart shows 150 ms per decision for the Decisions API, against 1.6 s for Luna through the regular API (The Decoder). Those are OpenAI's figures, not our measurements.
Confidence and log-probabilities: things to know
The scored benchmark recorded no confidence. We asked every model for its answer only. A separate probe then read GPT-6 Luna's log-probabilities on all 3,600 items: when Luna stated 99% or more, it was right 68.7% of the time on open-world problems. The full preview, with a verbatim request and response.
With reasoning off, Luna exposes its log-probabilities. It returns the chosen answer's log-probability and the most likely alternatives, which cover 99.8% and 99.9% of the probability on average (open world, closed world). With reasoning on, the request is refused: "'logprobs' is not supported with this model". OpenAI's guide says to remove logprobs and top_logprobs when reasoning effort is not none (OpenAI's latest-model guide; GPT-6 Luna's model page).
Overconfidence. OpenAI's GPT-4 Technical Report found the pre-trained model well calibrated, and that post-training reduced calibration markedly (its Figure 8). In our experience, where OpenAI models do expose log-probabilities, they tend to be too overconfident to calibrate into a useful confidence.
Why log-probabilities are limited with reasoning on: our suspicion, with no proof. Researchers recovered the final layer of production OpenAI models from API log-probabilities and logit bias, and OpenAI changed its API in response (Carlini et al., 2024; see also Finlayson et al., 2024). OpenAI also chose not to show o1's raw chain of thought, citing user experience, competitive advantage and chain-of-thought monitoring (OpenAI on o1's chain of thought). None of this shows why Luna's log-probabilities are limited.
Where each one is better
Open world, Jev vs GPT-6 Luna. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GPT-6 Luna is clearly ahead on unknown answers (+9.2 points).
Closed world, Jev vs GPT-6 Luna. Jev is clearly ahead on depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GPT-6 Luna is not clearly ahead on any depth or answer class. Too close to call: depth 0.
A slice counts as clearly ahead when its paired 95% interval excludes zero.
True, false and unknown, open world
By correct answer, Open world
Correct answer
Items
Jev
GPT-6 Luna
Best constant
false
605
88.3%85.5–90.9
50.9%46.8–54.9
one answer
true
605
88.8%86.1–91.1
58.3%54.0–62.1
one answer
unknown
590
74.2%70.7–77.6
83.4%80.5–86.6
one answer
True, false and unknown, closed world
By correct answer, Closed world
Correct answer
Items
Jev
GPT-6 Luna
Best constant
false
900
88.9%86.9–90.9
63.8%60.6–66.9
one answer
true
900
89.7%87.7–91.6
66.2%63.1–69.2
one answer
Paraphrased vs templated rules, open world
By wording, Open world
Wording
Items
Jev
GPT-6 Luna
Best constant
templated
1,298
86.1%84.2–87.9
67.6%64.8–70.2
34.5%
paraphrased (NatLang)
502
78.1%74.5–82.1
54.8%50.4–59.4
35.9%
Paraphrased vs templated rules, closed world
By wording, Closed world
Wording
Items
Jev
GPT-6 Luna
Best constant
templated
1,306
94.0%92.8–95.3
67.3%64.8–69.9
50.2%
paraphrased (NatLang)
494
76.7%73.1–80.4
58.9%54.7–63.4
50.6%
Answer by answer: Jev and GPT-6 Luna on the same problems
Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.
right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.
Open world: true, false or unknown
Jev, open-world task
Correct ↓ · said →
true
false
unknown
true
537
5
63
false
10
534
61
unknown
124
28
438
GPT-6 Luna (reasoning off), open-world task
Correct ↓ · said →
true
false
unknown
true
353
3
249
false
7
308
290
unknown
45
53
492
Jev said true on 37% of problems; the correct answer is true on 34%. GPT-6 Luna (reasoning off) said unknown on 57% of problems; the correct answer is unknown on 33%.
Closed world: true or false
Jev, closed-world task
Correct ↓ · said →
true
false
true
807
93
false
100
800
GPT-6 Luna (reasoning off), closed-world task
Correct ↓ · said →
true
false
true
596
304
false
326
574
Jev said true on 50% of problems; the correct answer is true on 50%. GPT-6 Luna (reasoning off) said true on 51% of problems; the correct answer is true on 50%.
Where GPT-6 Luna errs
Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.
right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.
GPT-6 Luna, open-world task
GPT-6 Luna (reasoning off), open-world task
Correct ↓ · said →
true
false
unknown
Recall
true
353
3
249
58.3%
false
7
308
290
50.9%
unknown
45
53
492
83.4%
GPT-6 Luna answered false 20.2%, true 22.5%, unknown 57.3% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.
GPT-6 Luna, closed-world task
GPT-6 Luna (reasoning off), closed-world task
Correct ↓ · said →
true
false
Recall
true
596
304
66.2%
false
326
574
63.8%
GPT-6 Luna answered false 48.8%, true 51.2% of the time; the correct answers are false 50.0%, true 50.0%.
GPT-6 Luna at depth 0 and depth 5
GPT-6 Luna (reasoning off), open-world task, depth 0
Correct ↓ · said →
true
false
unknown
true
96
0
4
false
2
85
13
unknown
1
0
99
GPT-6 Luna (reasoning off), open-world task, depth 5
Correct ↓ · said →
true
false
unknown
true
36
0
65
false
0
35
66
unknown
13
12
62
Open-world task. At depth 0, GPT-6 Luna was right on 93% of problems (the green diagonal); at depth 5, on 46%. The answer whose share moved most is unknown: 39% of its answers at depth 0, 67% at depth 5, against 33% and 30% of the correct answers.
The closed-world task at depth 0 and depth 5
GPT-6 Luna (reasoning off), closed-world task, depth 0
Correct ↓ · said →
true
false
true
148
2
false
4
146
GPT-6 Luna (reasoning off), closed-world task, depth 5
Correct ↓ · said →
true
false
true
73
77
false
89
61
Closed-world task. At depth 0, GPT-6 Luna was right on 98% of problems (the green diagonal); at depth 5, on 45%. The answer whose share moved most is true: 51% of its answers at depth 0, 54% at depth 5, against 50% and 50% of the correct answers.
GPT-6 Luna across every breakdown
For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.
Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.
The text the model reads
The bear is green. Green things are not young.
Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%.
The full request and response, exactly as sent and recorded
Request
{
"model": "jev-latest",
"state": {
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}
GPT-6 Luna answered unknown; correct answer unknown
POST https://api.openai.com/v1/chat/completions
GPT-6 Luna with reasoning off (reasoning_effort none, its lowest setting), as a direct classifier. One chat message carries the problem, the question and every option with its meaning; the strict JSON schema only allows one of the options as the answer.
The text the model reads
The bear is green. Green things are not young.
Statement: The bear is not big.
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
Options:
- true: The statement follows from the facts and rules.
- false: The negation of the statement follows from the facts and rules.
- unknown: Neither the statement nor its negation follows from the facts and rules.
Reply with a JSON object {"answer": <option>} where <option> is exactly one of: true, false, unknown.
The answers it can give
true
false
unknown
The full request and response, exactly as sent and recorded
Request
{
"model": "gpt-6-luna",
"max_completion_tokens": 1024,
"reasoning_effort": "none",
"messages": [
{
"role": "user",
"content": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.\n\nReply with a JSON object {\"answer\": <option>} where <option> is exactly one of: true, false, unknown."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "decision",
"strict": true,
"schema": {
"type": "object",
"properties": {
"answer": {
"type": "string",
"enum": [
"true",
"false",
"unknown"
]
}
},
"required": [
"answer"
],
"additionalProperties": false
}
}
}
}
GPT-6 Luna's median open-world decision took 825 ms against Jev's 189 ms, and 1 of its 3,600 decisions took ten seconds or more (the slowest 10 s). At list price GPT-6 Luna cost $37 per million decisions against Jev's $23: 1.6 times as much per decision and 2.1 times as much per correct decision.
Every decision's time
Jev
GPT-6 Luna(reasoning off)
Average time per decision, by proof depth
Open world: true, false or unknown
Closed world: true or false
Model
Task
Depth 0
Depth 1
Depth 2
Depth 3
Depth 4
Depth 5
Jev
Open world
207 ms
202 ms
209 ms
203 ms
207 ms
209 ms
Jev
Closed world
177 ms
177 ms
175 ms
174 ms
174 ms
178 ms
GPT-6 Luna
Open world
894 ms
890 ms
902 ms
920 ms
876 ms
894 ms
GPT-6 Luna
Closed world
993 ms
1.0 s
974 ms
1.0 s
990 ms
981 ms
List-price cost per million decisions, by proof depth
Asked every open world question a second time, GPT-6 Luna gave the same answer on 89.7% of 1,800 items (185 changed; Gwet's AC1 0.855, interval 0.837 to 0.876). Accuracy was 64.1% the first time and 62.8% the second.
Asked every closed world question a second time, GPT-6 Luna gave the same answer on 90.6% of 1,800 items (169 changed; Gwet's AC1 0.812, interval 0.784 to 0.838). Accuracy was 65.0% the first time and 64.8% the second.
Times are open-world, one request at a time. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.
Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.
GPT-6 Luna GPT-6 Luna is a hosted large language model made by OpenAI. We used it as a classifier: one request per item through the Chat Completions API, with reasoning effort set to none (its lowest setting) and a strict JSON schema that allows only the task's options. Its size and architecture are not published.
Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.
Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.