Skip to main content
HardDecisions

Comparison · Kev vs Jev

Kev vs Jev: is an open-source decision model a Jev alternative?

Kev is an open decision model you can run on your own hardware; Jev is hosted. We ran every Kev size that fits a 32 GB laptop against Jev on the same problems, and asked whether a bigger Kev helps.

Overall: Jev, Kev-9B, Kev-4B and Kev-0.8B

Open world: true, false or unknown

  • Jev82.2 to 85.5 (95%), 1,800 items83.8%
  • Kev-9B56.3 to 60.9 (95%), 1,800 items58.6%
  • Kev-4B51.4 to 56.0 (95%), 1,800 items53.6%
  • Kev-0.8B51.4 to 55.9 (95%), 1,800 items53.6%

Jev minus Kev-9B: +25.3 points on the same 1,800 items, 95% interval +22.7 to +27.8: Jev is clearly ahead.

Jev minus Kev-4B: +30.3 points on the same 1,800 items, 95% interval +27.4 to +32.9: Jev is clearly ahead.

Jev minus Kev-0.8B: +30.3 points on the same 1,800 items, 95% interval +27.7 to +32.7: Jev is clearly ahead.

Closed world: true or false

  • Jev87.8 to 90.7 (95%), 1,800 items89.3%
  • Kev-9B61.5 to 66.1 (95%), 1,800 items63.8%
  • Kev-4B56.6 to 61.1 (95%), 1,800 items58.8%
  • Kev-0.8B53.5 to 57.9 (95%), 1,800 items55.8%

Jev minus Kev-9B: +25.5 points on the same 1,800 items, 95% interval +23.0 to +27.8: Jev is clearly ahead.

Jev minus Kev-4B: +30.5 points on the same 1,800 items, 95% interval +27.9 to +33.1: Jev is clearly ahead.

Jev minus Kev-0.8B: +33.5 points on the same 1,800 items, 95% interval +30.7 to +36.3: Jev is clearly ahead.

Jev vs Kev-9B, Kev-4B and Kev-0.8B by proof depth

Each point is about 300 problems that need that many inference steps, with its 95% interval.

Open world: true, false or unknown
  • Jev
  • Kev-9B
  • Kev-4B
  • Kev-0.8B
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
  • Kev-9B
  • Kev-4B
  • Kev-0.8B
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Jev minus Kev-9B by depth, open world

depth 0300 items
+8.3
depth 1302 items
+13.9
depth 2303 items
+26.4
depth 3303 items
+30.4
depth 4303 items
+33.3
depth 5289 items
+39.8

Jev minus Kev-9B by depth, closed world

depth 0300 items
+9.7
depth 1300 items
+16.7
depth 2300 items
+28.0
depth 3300 items
+26.3
depth 4300 items
+26.0
depth 5300 items
+46.3

Jev minus Kev-4B by depth, open world

depth 0300 items
+10.0
depth 1302 items
+24.8
depth 2303 items
+32.7
depth 3303 items
+35.3
depth 4303 items
+34.7
depth 5289 items
+44.6

Jev minus Kev-4B by depth, closed world

depth 0300 items
+13.7
depth 1300 items
+25.3
depth 2300 items
+37.7
depth 3300 items
+29.0
depth 4300 items
+31.3
depth 5300 items
+46.0

Jev minus Kev-0.8B by depth, open world

depth 0300 items
+27.3
depth 1302 items
+33.8
depth 2303 items
+38.0
depth 3303 items
+31.0
depth 4303 items
+21.5
depth 5289 items
+30.1

Jev minus Kev-0.8B by depth, closed world

depth 0300 items
+31.3
depth 1300 items
+34.0
depth 2300 items
+40.7
depth 3300 items
+32.0
depth 4300 items
+25.3
depth 5300 items
+37.7

Paired differences: positive means Jev was more accurate on the same items.

Accuracy by depth, open world

By depth, Open world
Proof depthItemsJevKev-9BKev-4BKev-0.8BBest constant
depth 030097.7% 96.0–99.389.3% 85.7–92.787.7% 84.0–91.370.3% 65.3–75.733.3%
depth 130287.7% 83.8–91.473.8% 68.9–78.862.9% 57.3–68.254.0% 48.3–59.633.4%
depth 230384.8% 80.2–88.458.4% 53.1–63.752.1% 46.5–57.846.9% 41.3–52.833.3%
depth 330379.9% 75.2–84.549.5% 44.2–55.444.6% 39.3–49.848.8% 43.6–54.533.3%
depth 430371.9% 66.7–77.238.6% 33.3–43.637.3% 31.7–42.250.5% 44.9–56.133.3%
depth 528981.0% 76.5–85.541.2% 35.3–46.736.3% 31.1–41.950.9% 45.3–56.734.9%

Accuracy by depth, closed world

By depth, Closed world
Proof depthItemsJevKev-9BKev-4BKev-0.8BBest constant
depth 030099.3% 98.3–100.089.7% 86.0–93.085.7% 81.7–89.768.0% 62.7–73.350.0%
depth 130093.7% 91.0–96.377.0% 72.7–81.768.3% 63.0–73.759.7% 54.0–65.750.0%
depth 230092.0% 88.7–94.764.0% 59.0–69.354.3% 49.0–59.751.3% 45.7–57.050.0%
depth 330085.0% 81.3–89.358.7% 53.0–64.356.0% 50.7–61.353.0% 47.7–58.750.0%
depth 430076.3% 71.7–81.050.3% 44.7–55.745.0% 39.0–50.351.0% 45.7–57.050.0%
depth 530089.3% 85.7–92.343.0% 37.3–48.743.3% 38.3–49.051.7% 46.0–57.350.0%

What Kev-9B got right and wrong, depth by depth

At each proof depth, Kev-9B's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

89.3%
right on 268 of 300 problems wrong; chance 33% (dashed line)
right

The rabbit is not green.

Correct answer: true · Model said: true (98.4% sure)

The problem and its proof

The bald eagle does not eat the rabbit. The bald eagle eats the tiger. The bald eagle is round. The bald eagle does not see the rabbit. The bald eagle does not visit the tiger. The rabbit eats the bald eagle. The rabbit is not blue. The rabbit is not green. The rabbit is round. The rabbit sees the tiger. The rabbit does not visit the bald eagle. The tiger does not eat the bald eagle. The tiger eats the rabbit. The tiger is not nice. The tiger sees the bald eagle. The tiger visits the bald eagle. If someone is nice then they visit the bald eagle. If someone sees the tiger and the tiger is nice then they visit the bald eagle. If someone visits the bald eagle and they are not green then they visit the rabbit.

Stated in the text: "The rabbit is not green."

right

Alan is young.

Correct answer: unknown · Model said: unknown (97.6% sure)

paraphrased
The problem and its proof

Alan is a cold and round man who has red and green skin. Dave is green to the job, which is why he turned blue from being so cold. Fred seems to be round. A person who is cold and blue is nice. It seems that a cold person who is round and nice will be green. A nice young person who is big can be considered round. You'll always find rough, cold, green people to also be red people. People that are green and big while also being cold are always nice. Count on anyone you meet who is big, round, and red also being kind. A nice person with cold skin is going to be rough.

No proof: neither the statement nor its negation follows.

right

The tiger is red.

Correct answer: unknown · Model said: unknown (99.1% sure)

The problem and its proof

The squirrel needs the tiger. The tiger does not like the squirrel. The tiger sees the squirrel. If something likes the squirrel and it is green then it is rough. If something needs the tiger then it is nice. If something sees the squirrel and the squirrel is nice then it needs the tiger.

No proof: neither the statement nor its negation follows.

right

The rabbit is green.

Correct answer: unknown · Model said: unknown (57.0% sure)

The problem and its proof

The cat sees the rabbit. The rabbit is rough. If someone needs the cat and they are red then they chase the cat. If the cat sees the rabbit then the cat is red. If someone chases the cat then the cat is nice. If the cat needs the rabbit and the cat is not rough then the cat is green. If the cat is red then the cat is green. If someone is rough then they need the cat. If someone sees the cat then they chase the rabbit. If someone needs the cat then the cat sees the rabbit.

No proof: neither the statement nor its negation follows.

Depth 1, open-world task

73.8%
right on 223 of 302 problems wrong; chance 33% (dashed line)
wrong

Harry is rough.

Correct answer: true · Model said: unknown (72.4% sure)

paraphrased
The problem and its proof

Green, big and red are qualities which all describe Alan. Bob seems to be round. Eric is cold but nice with red hair and dresses green. Harry may be round, but he is also kind. People,though kind and rough are also really nice. A kind person will certainly be rough as well. A person that is both nice and rough is someone who is also big. Anyone rough in texture and young in age is a cold person. I bet anyone who is round and kind is also pretty young.

  1. Harry is kind.If someone is kind then they are rough.Harry is rough.
right

Fiona is not red.

Correct answer: true · Model said: true (95.7% sure)

The problem and its proof

Anne is quiet. Anne is not red. Anne is rough. Anne is smart. Bob is not blue. Bob is not furry. Bob is green. Bob is quiet. Bob is red. Bob is rough. Erin is blue. Fiona is furry. Fiona is not quiet. Fiona is rough. Fiona is not smart. If Fiona is furry then Fiona is not red. If someone is furry and not red then they are blue.

  1. Fiona is furry.If Fiona is furry then Fiona is not red.Fiona is not red.
right

The cow chases the cow.

Correct answer: false · Model said: false (60.9% sure)

The problem and its proof

The cow does not chase the rabbit. The cow is big. The cow is not green. The cow is rough. The cow is round. The cow is young. The cow likes the rabbit. The cow visits the rabbit. The rabbit does not chase the cow. The rabbit is not big. The rabbit is green. The rabbit is rough. The rabbit is not round. The rabbit is young. The rabbit likes the cow. The rabbit does not visit the cow. If the rabbit chases the cow then the cow chases the rabbit. If someone chases the cow then the cow visits the rabbit. If someone is round then they do not chase the cow. If the rabbit chases the cow then the cow likes the rabbit. If someone visits the cow then they are young. If the rabbit chases the cow and the rabbit does not visit the cow then the cow is rough. If someone chases the cow and they do not chase the rabbit then the rabbit likes the cow. If someone visits the rabbit and they do not visit the cow then the rabbit likes the cow.

The proof that the statement's negation follows:

  1. The cow is round.If someone is round then they do not chase the cow.The cow does not chase the cow.
right

The dog is not nice.

Correct answer: true · Model said: true (94.4% sure)

The problem and its proof

The dog is kind. If the dog is nice then the dog is not red. If the dog is nice and the dog is not cold then the dog is not red. If something is kind then it is not nice. Red things are round. If something is red then it is not kind. If the dog is not nice then the dog is round.

  1. The dog is kind.If something is kind then it is not nice.The dog is not nice.

Depth 2, open-world task

58.4%
right on 177 of 303 problems wrong; chance 33% (dashed line)
wrong

The dog likes the dog.

Correct answer: false · Model said: unknown (84.3% sure)

The problem and its proof

The cat chases the dog. The cat is cold. The cat is rough. The cat is young. The cat likes the dog. The cat needs the dog. The dog does not chase the cat. The dog is not nice. The dog is young. The dog needs the cat. If something chases the cat then the cat is rough. If the cat is not red then the cat likes the dog. If the dog is young and the dog is not cold then the dog needs the cat. If the dog chases the cat then the cat chases the dog. If the cat is young then the cat chases the dog. If something likes the cat then it does not like the dog. If something is rough then it needs the dog. If something is young and it needs the dog then the dog likes the cat.

The proof that the statement's negation follows:

  1. The cat is young. + The cat needs the dog.If something is young and it needs the dog then the dog likes the cat.The dog likes the cat.
  2. If something likes the cat then it does not like the dog.The dog does not like the dog.
right

The bear does not eat the mouse.

Correct answer: false · Model said: false (69.6% sure)

The problem and its proof

The bear sees the mouse. The mouse chases the bear. If something is big then it eats the mouse. If something sees the mouse then it is big.

The proof that the statement's negation follows:

  1. The bear sees the mouse.If something sees the mouse then it is big.The bear is big.
  2. If something is big then it eats the mouse.The bear eats the mouse.
wrong

The cat likes the cat.

Correct answer: unknown · Model said: true (78.2% sure)

The problem and its proof

The bald eagle is blue. The bald eagle likes the cat. The cat chases the dog. The dog visits the cat. The squirrel is not big. The squirrel is cold. The squirrel is red. If something visits the cat then it chases the cat. If something chases the cat then it likes the cat.

No proof: neither the statement nor its negation follows.

right

Dave is not red.

Correct answer: false · Model said: false (77.8% sure)

paraphrased
The problem and its proof

Young Dave has a round form, believes in green and is rough. Fred is cold but nice with red hair and dresses green. For being so cold, it's good Gary can remain nice. Harry seems to be round. A person who is feeling rough, has red cheeks and feeling green will also tend to feel blue. You must be aware of big, red, young people being rough. Interesting that all round people are cold. People who are round and green while being cold are also red.

The proof that the statement's negation follows:

  1. Dave is round.If someone is round then they are cold.Dave is cold.
  2. + Dave is round. + Dave is green.If someone is round and green and cold then they are red.Dave is red.

Depth 3, open-world task

49.5%
right on 150 of 303 problems wrong; chance 33% (dashed line)
wrong

The mouse does not like the tiger.

Correct answer: unknown · Model said: false (76.0% sure)

The problem and its proof

The mouse is young. The mouse visits the tiger. The tiger does not eat the mouse. The tiger is rough. The tiger is round. The tiger likes the mouse. The tiger visits the mouse. If someone likes the mouse then they are cold. If someone visits the mouse and the mouse is cold then the mouse likes the tiger. If someone eats the mouse and they eat the tiger then the tiger visits the mouse. If someone visits the mouse then the mouse is young. If someone likes the mouse then they visit the tiger. If someone is round and big then they like the mouse. If someone eats the tiger then they visit the mouse. If the tiger does not like the mouse and the mouse does not like the tiger then the tiger is rough.

No proof: neither the statement nor its negation follows.

wrong

The bald eagle is red.

Correct answer: unknown · Model said: true (97.5% sure)

The problem and its proof

The bald eagle likes the lion. The bald eagle sees the bear. The bald eagle sees the lion. The bear chases the mouse. The bear is cold. The bear likes the mouse. The bear sees the lion. The lion sees the mouse. The mouse chases the bear. The mouse is cold. If someone sees the bear and they do not chase the bald eagle then the bear sees the lion. If someone is nice then they like the bald eagle. If someone likes the bear then the bear likes the bald eagle. If someone likes the mouse and the mouse is green then they are nice. If someone likes the bald eagle then they are red. If someone chases the lion then the lion sees the mouse. If someone is red then they chase the lion. If someone likes the mouse then the mouse is green. If someone likes the bear and they are not green then they chase the mouse.

No proof: neither the statement nor its negation follows.

right

Charlie is young.

Correct answer: unknown · Model said: unknown (52.6% sure)

The problem and its proof

Anne is cold. Bob is rough. Charlie is cold. Erin is smart. All kind people are not round. Rough people are round. All blue, smart people are round. Round people are young. Smart people are rough. If someone is cold then they are not smart.

No proof: neither the statement nor its negation follows.

right

Harry is nice.

Correct answer: true · Model said: true (68.2% sure)

paraphrased
The problem and its proof

That guy Eric sure is nice. Gary may be round, but he is also kind. Harry is colored red and blue. He is also young, quite round, and cold. A person who is feeling rough, has red cheeks and feeling green will also tend to feel blue. People,though kind and rough are also really nice. Someone that is cold rough and red is also considered to be kind. A young person with red hands is rough around the edges. If a person acts cold yet nice and green, they will be kind.

  1. Harry is red. + Harry is young.If someone is red and young then they are rough.Harry is rough.
  2. + Harry is red. + Harry is cold.If someone is red and rough and cold then they are kind.Harry is kind.
  3. If someone is kind and rough then they are nice.Harry is nice.

Depth 4, open-world task

38.6%
right on 117 of 303 problems wrong; chance 33% (dashed line)
right

Gary is not white.

Correct answer: unknown · Model said: unknown (60.7% sure)

The problem and its proof

Charlie is furry. Charlie is young. Gary is big. Gary is furry. Harry is big. Harry is kind. Harry is white. If someone is kind and white then they are cold. If Gary is nice then Gary is furry. Nice people are young. All nice, big people are kind. Kind people are cold. If someone is furry and cold then they are white. Young people are kind. If Harry is kind and Harry is white then Harry is nice.

No proof: neither the statement nor its negation follows.

wrong

The mouse does not chase the lion.

Correct answer: unknown · Model said: true (55.2% sure)

The problem and its proof

The bald eagle likes the lion. The bald eagle sees the bear. The bald eagle sees the lion. The bear chases the mouse. The bear is cold. The bear likes the mouse. The bear sees the lion. The lion sees the mouse. The mouse chases the bear. The mouse is cold. If someone sees the bear and they do not chase the bald eagle then the bear sees the lion. If someone is nice then they like the bald eagle. If someone likes the bear then the bear likes the bald eagle. If someone likes the mouse and the mouse is green then they are nice. If someone likes the bald eagle then they are red. If someone chases the lion then the lion sees the mouse. If someone is red then they chase the lion. If someone likes the mouse then the mouse is green. If someone likes the bear and they are not green then they chase the mouse.

No proof: neither the statement nor its negation follows.

wrong

The dog does not visit the dog.

Correct answer: false · Model said: unknown (59.6% sure)

The problem and its proof

The bald eagle chases the dog. The bald eagle is cold. The dog chases the bald eagle. The rabbit is nice. The tiger is cold. The tiger needs the dog. The tiger visits the dog. If someone is nice and cold then they chase the tiger. If the bald eagle chases the tiger and the tiger needs the bald eagle then the bald eagle needs the tiger. If someone visits the dog then they chase the dog. If someone chases the bald eagle then the bald eagle visits the dog. If the tiger chases the dog then the dog is cold. If someone is round then they visit the dog. If someone chases the dog and they visit the bald eagle then the bald eagle is cold. If someone visits the dog and the dog is cold then the dog is round.

The proof that the statement's negation follows:

  1. The tiger visits the dog.If someone visits the dog then they chase the dog.The tiger chases the dog.
  2. If the tiger chases the dog then the dog is cold.The dog is cold.
  3. + The tiger visits the dog.If someone visits the dog and the dog is cold then the dog is round.The dog is round.
  4. If someone is round then they visit the dog.The dog visits the dog.
right

The rabbit sees the cat.

Correct answer: true · Model said: true (58.1% sure)

The problem and its proof

The cat eats the mouse. The cat is blue. The cat is rough. The cat sees the squirrel. The mouse is kind. The mouse is young. The mouse needs the rabbit. The mouse sees the cat. The rabbit eats the cat. The rabbit eats the mouse. The rabbit eats the squirrel. The rabbit is rough. The rabbit needs the cat. The rabbit sees the mouse. The squirrel eats the mouse. The squirrel is kind. If someone is young and they eat the squirrel then they need the mouse. If someone sees the cat and the cat is rough then they see the squirrel. If someone is rough and they eat the cat then they are big. Big people are young. If someone needs the mouse then they see the cat.

  1. The rabbit is rough. + The rabbit eats the cat.If someone is rough and they eat the cat then they are big.The rabbit is big.
  2. Big people are young.The rabbit is young.
  3. + The rabbit eats the squirrel.If someone is young and they eat the squirrel then they need the mouse.The rabbit needs the mouse.
  4. If someone needs the mouse then they see the cat.The rabbit sees the cat.

Depth 5, open-world task

41.2%
right on 119 of 289 problems wrong; chance 33% (dashed line)
right

Gary is not kind.

Correct answer: true · Model said: true (83.1% sure)

The problem and its proof

Anne is not blue. Anne is cold. Anne is kind. Anne is nice. Anne is quiet. Erin is quiet. Gary is not nice. Gary is rough. Harry is not blue. Harry is cold. Harry is nice. Harry is quiet. If Gary is not nice then Gary is cold. Rough, cold things are blue. If something is kind then it is cold. If something is white and not nice then it is quiet. If something is cold and blue then it is white. All kind, blue things are not white. If something is quiet and rough then it is not kind.

  1. Gary is not nice.If Gary is not nice then Gary is cold.Gary is cold.
  2. + Gary is rough.Rough, cold things are blue.Gary is blue.
  3. If something is cold and blue then it is white.Gary is white.
  4. + Gary is not nice.If something is white and not nice then it is quiet.Gary is quiet.
  5. + Gary is rough.If something is quiet and rough then it is not kind.Gary is not kind.
wrong

Erin is red.

Correct answer: false · Model said: unknown (87.3% sure)

The problem and its proof

Bob is big. Bob is red. Bob is smart. Bob is young. Erin is round. Erin is smart. Gary is round. Harry is nice. Harry is red. Harry is not round. Harry is smart. Harry is young. Smart people are furry. Red people are furry. If Erin is young then Erin is big. Big people are nice. All round, furry people are young. All nice people are young. Nice, round people are not red.

The proof that the statement's negation follows:

  1. Erin is smart.Smart people are furry.Erin is furry.
  2. + Erin is round.All round, furry people are young.Erin is young.
  3. If Erin is young then Erin is big.Erin is big.
  4. Big people are nice.Erin is nice.
  5. + Erin is round.Nice, round people are not red.Erin is not red.
right

Dave is kind.

Correct answer: true · Model said: true (92.1% sure)

The problem and its proof

Charlie is furry. Charlie is green. Charlie is red. Charlie is round. Dave is green. Fiona is furry. Fiona is round. Gary is green. Gary is kind. Gary is smart. If someone is round then they are cold. Green people are round. If someone is red and furry then they are green. Green, kind people are cold. If someone is cold then they are furry. Furry people are smart. If someone is smart then they are kind.

  1. Dave is green.Green people are round.Dave is round.
  2. If someone is round then they are cold.Dave is cold.
  3. If someone is cold then they are furry.Dave is furry.
  4. Furry people are smart.Dave is smart.
  5. If someone is smart then they are kind.Dave is kind.
wrong

Charlie is not nice.

Correct answer: false · Model said: unknown (48.3% sure)

The problem and its proof

Charlie is blue. Dave is quiet. Erin is blue. Erin is green. Erin is quiet. Erin is round. Gary is blue. All green people are kind. All kind, round people are red. If someone is kind then they are quiet. All quiet, red people are nice. If someone is blue then they are green. If someone is green and kind then they are blue. If someone is kind then they are round. Nice, blue people are red. All red, quiet people are kind.

The proof that the statement's negation follows:

  1. Charlie is blue.If someone is blue then they are green.Charlie is green.
  2. All green people are kind.Charlie is kind.
  3. If someone is kind then they are quiet.Charlie is quiet.
  4. If someone is kind then they are round.Charlie is round.
  5. All kind, round people are red.Charlie is red.
  6. All quiet, red people are nice.Charlie is nice.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

89.7%
right on 269 of 300 problems wrong; chance 50% (dashed line)
right

The mouse is blue.

Correct answer: true · Model said: true (95.4% sure)

The problem and its proof

The mouse is blue. If someone is round and blue then they are cold. Round people are kind. If someone is blue then they are kind. Red people are kind. All cold people are kind. If someone is kind then they are round.

Stated in the text: "The mouse is blue."

right

The mouse is red.

Correct answer: true · Model said: true (97.8% sure)

The problem and its proof

The bear eats the mouse. The mouse is red. If someone needs the mouse and they chase the bear then the mouse chases the bear. If the bear does not need the mouse then the mouse needs the bear.

Stated in the text: "The mouse is red."

right

The dog likes the bear.

Correct answer: false · Model said: false (81.8% sure)

The problem and its proof

The bear needs the dog. The dog needs the bear. If someone needs the bear and they do not like the dog then the bear sees the dog. If someone sees the dog then the dog is big. If someone needs the bear and they are big then they need the dog.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Charlie is not red.

Correct answer: false · Model said: false (87.0% sure)

The problem and its proof

Charlie is red. If something is nice and not young then it is blue. If Charlie is red then Charlie is kind. All kind things are big.

Stated in the text: "Charlie is red."

Depth 1, closed-world task

77.0%
right on 231 of 300 problems wrong; chance 50% (dashed line)
wrong

The bear chases the tiger.

Correct answer: true · Model said: false (72.0% sure)

The problem and its proof

The bear is big. The bear is blue. The bear is rough. The bear likes the tiger. The bear needs the tiger. The tiger chases the bear. The tiger needs the bear. If the tiger needs the bear and the bear chases the tiger then the tiger is young. If the tiger does not need the bear then the bear needs the tiger. If someone chases the tiger and the tiger chases the bear then the bear likes the tiger. If someone chases the bear then the bear likes the tiger. If someone is young then they like the bear. If someone is blue then they chase the tiger. If the bear likes the tiger and the tiger chases the bear then the tiger likes the bear. If someone is young and they do not need the tiger then they like the tiger.

  1. The bear is blue.If someone is blue then they chase the tiger.The bear chases the tiger.
right

Harry is kind.

Correct answer: false · Model said: false (76.4% sure)

paraphrased
The problem and its proof

Bob who frequently wears red clothes always feels very cold because of his young age on the team and his rough skin. Gary is a red, rough, green and blue man. Harry is a young, round shaped young man who is also very cold. People who are very nice and easily flush red are often wearing green concealer. When rough and kind can describe a person, then cold will describe them, too. Anyone having rough, cold and green qualities will also have a big quality. A nice, green, big person is also sure to be a red person. Someone with rough and green feet is invariably kind.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The rabbit does not eat the bear.

Correct answer: false · Model said: false (72.1% sure)

The problem and its proof

The bald eagle chases the rabbit. The bear needs the rabbit. The cat is nice. The rabbit chases the bald eagle. If someone eats the bear and they do not eat the cat then they chase the cat. If the bear is cold and the bear eats the rabbit then the rabbit is nice. If the cat is nice and the cat does not chase the rabbit then the rabbit eats the bear.

  1. The cat is nice.If the cat is nice and the cat does not chase the rabbit then the rabbit eats the bear.The rabbit eats the bear.
right

Erin is quiet.

Correct answer: true · Model said: true (84.0% sure)

The problem and its proof

Bob is green. Dave is furry. Dave is green. Dave is red. Erin is red. Gary is green. Gary is rough. If someone is rough and green then they are blue. If someone is furry and quiet then they are red. All rough, kind people are red. If someone is red then they are quiet. If someone is furry and blue then they are rough. Furry, quiet people are rough. If Erin is kind then Erin is green. All blue people are furry. If Gary is furry and Gary is green then Gary is kind.

  1. Erin is red.If someone is red then they are quiet.Erin is quiet.

Depth 2, closed-world task

64.0%
right on 192 of 300 problems wrong; chance 50% (dashed line)
wrong

The squirrel chases the cat.

Correct answer: false · Model said: true (97.0% sure)

The problem and its proof

The cat chases the rabbit. The rabbit chases the tiger. The rabbit is nice. The squirrel is big. The squirrel sees the rabbit. The squirrel sees the tiger. The tiger is blue. If something sees the tiger then the tiger eats the cat. If something sees the cat then the cat chases the squirrel. If something is green and it chases the cat then it sees the rabbit. If something chases the cat then it is green. If something sees the tiger then the tiger eats the squirrel. If something eats the cat then it sees the squirrel. If something sees the squirrel then it chases the cat. If something eats the rabbit then the rabbit is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The dog is not kind.

Correct answer: true · Model said: true (72.3% sure)

The problem and its proof

The dog is round. If the dog is kind and the dog is round then the dog is cold. If the dog is young and the dog is cold then the dog is kind. All blue things are young.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Fred is not round.

Correct answer: false · Model said: false (62.3% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. That guy Eric sure is nice. Fred is a big guy who is colored red from a sunburn. He also has a cold. Gary is young, round and kind with green ideas and wears blue clothes. Kind people that are red and round are usually big. Nice round big people are kind to others. Rough, cold people are blue. Anyone feeling cold is bound to act nice. When someone is green and round, they can be rough but kind. A nice person is inevitably round as well. Young and red people look round.

  1. Fred is cold.If someone is cold then they are nice.Fred is nice.
  2. If someone is nice then they are round.Fred is round.
right

Anne is not rough.

Correct answer: true · Model said: true (91.3% sure)

The problem and its proof

Anne is big. Anne is round. Dave is nice. Dave is red. Fiona is red. Fiona is rough. Harry is round. If someone is green and big then they are round. Red people are rough. Rough people are green. Nice, round people are green. If Harry is green and Harry is big then Harry is kind. All nice people are red. If someone is rough and green then they are kind. Kind, rough people are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

58.7%
right on 176 of 300 problems wrong; chance 50% (dashed line)
wrong

Harry is not big.

Correct answer: false · Model said: true (66.0% sure)

paraphrased
The problem and its proof

Green, big and red are qualities which all describe Alan. Bob seems to be round. Eric is cold but nice with red hair and dresses green. Harry may be round, but he is also kind. People,though kind and rough are also really nice. A kind person will certainly be rough as well. A person that is both nice and rough is someone who is also big. Anyone rough in texture and young in age is a cold person. I bet anyone who is round and kind is also pretty young.

  1. Harry is kind.If someone is kind then they are rough.Harry is rough.
  2. + Harry is kind.If someone is kind and rough then they are nice.Harry is nice.
  3. If someone is rough and nice then they are big.Harry is big.
right

The mouse visits the bear.

Correct answer: false · Model said: false (80.6% sure)

The problem and its proof

The bear sees the lion. The bear visits the lion. The bear visits the squirrel. The lion chases the bear. The lion chases the mouse. The lion is big. The lion sees the squirrel. The mouse chases the bear. The mouse visits the lion. The squirrel is young. The squirrel visits the lion. The squirrel visits the mouse. If someone sees the lion and the lion is kind then they are red. If someone visits the lion and they are red then the lion visits the squirrel. If someone is kind then they see the lion. If the lion visits the squirrel then the lion chases the bear. If the squirrel chases the lion then the lion is kind. If someone is big then they chase the lion. If someone chases the bear then the bear is big. If someone chases the lion then the lion is kind. If someone is red then they visit the bear.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The cow is cold.

Correct answer: true · Model said: false (82.0% sure)

The problem and its proof

The bear eats the cow. The bear eats the mouse. The bear eats the rabbit. The bear visits the cow. The bear visits the mouse. The cow chases the mouse. The mouse chases the bear. The mouse is cold. The mouse is kind. The mouse is young. The mouse visits the cow. The rabbit chases the mouse. The rabbit eats the cow. The rabbit eats the mouse. The rabbit is cold. The rabbit visits the bear. If something is green then it chases the cow. If something chases the cow then it is cold. If the rabbit is young and the rabbit visits the bear then the rabbit eats the bear. If something is cold then it is kind. If something visits the cow then it is kind. If something is kind then it chases the mouse. If something is cold then it eats the rabbit. If something chases the mouse then it is green.

  1. The cow chases the mouse.If something chases the mouse then it is green.The cow is green.
  2. If something is green then it chases the cow.The cow chases the cow.
  3. If something chases the cow then it is cold.The cow is cold.
right

The tiger is not round.

Correct answer: true · Model said: true (52.3% sure)

The problem and its proof

The tiger is kind. All kind, red people are round. If someone is cold then they are red. If the tiger is red and the tiger is nice then the tiger is cold. Nice, round people are cold. If someone is kind then they are nice. If the tiger is round then the tiger is kind.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 4, closed-world task

50.3%
right on 151 of 300 problems wrong; chance 50% (dashed line)
wrong

Bob is cold.

Correct answer: true · Model said: false (50.7% sure)

paraphrased
The problem and its proof

Alan may be young but he is nice, wears green shoes and is cold. Bob is green with envy, and rough with his kids, but overall is just a nice, round individual. That guy Dave sure is nice. Fred is green to the job, which is why he turned blue from being so cold. Every time you meet someone kind and nice, they'll be green, too. When green, young and round fits a person, you'll see that rough will also fit. If a person has a rough and round shape and is nice, then they will be big too. Truth be told, every big, rough person out there that is green, will also be colored blue. People who are feeling blue and green are said to be red. Kind people that are round are on the big side. If a person is blue and red and big, then that person is cold.

  1. Bob is rough. + Bob is round. + Bob is nice.If someone is rough and round and nice then they are big.Bob is big.
  2. + Bob is rough. + Bob is green.If someone is rough and big and green then they are blue.Bob is blue.
  3. + Bob is green.If someone is blue and green then they are red.Bob is red.
  4. If someone is blue and big and red then they are cold.Bob is cold.
right

Eric is not round.

Correct answer: false · Model said: false (51.9% sure)

paraphrased
The problem and its proof

Bob may be round, but he is also kind. Charlie may be round, but he is also kind. Eric came out and showed us how young, kind, cold, and big he was. Harry is known for being big and rough, but few know he is also always red with cold. A person who is cold and blue is nice. Rough people who are red in color and big are usally round in shape. When someone is kind yet can be cold and blue, they will also be very big. Anyone feeling cold is bound to act nice. Young people that are relatively nice and kind to others, can also tend to be rough. When someone is blue, nice and young they will also be as big as the sun. A person who is rough and young while being kind tends to be red.

  1. Eric is cold.If someone is cold then they are nice.Eric is nice.
  2. + Eric is kind. + Eric is young.If someone is nice and kind and young then they are rough.Eric is rough.
  3. + Eric is kind. + Eric is young.If someone is kind and rough and young then they are red.Eric is red.
  4. + Eric is big.If someone is red and rough and big then they are round.Eric is round.
right

Charlie is big.

Correct answer: true · Model said: true (52.1% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Charlie is green with envy, and rough with his kids, but overall is just a nice, round individual. Eric may be round, but he is also kind. The young person who is always feeling cold is named Fred. People who have green body paint and act kind to others are quite young. When you meet someone who is nice and kind, they wont feel blue. Nice people that are very green and even round shaped will be very young. A nice but rough person will also be green and cold. Most young kind people tend to be red too. When someone is blue, nice and young they will also be as big as the sun. If a person acts cold yet nice and green, they will be kind.

  1. Charlie is green. + Charlie is round. + Charlie is nice.If someone is green and round and nice then they are young.Charlie is young.
  2. + Charlie is nice. + Charlie is green. + Charlie is rough.If someone is nice and green and rough then they are cold.Charlie is cold.
  3. + Charlie is nice. + Charlie is green.If someone is nice and green and cold then they are kind.Charlie is kind.
  4. + Charlie is nice.If someone is kind and nice then they are blue.Charlie is blue.
  5. + Charlie is nice.If someone is young and blue and nice then they are big.Charlie is big.
wrong

The mouse does not need the rabbit.

Correct answer: false · Model said: true (87.9% sure)

The problem and its proof

The dog is rough. The mouse is round. The mouse visits the dog. The rabbit is kind. The rabbit is round. The rabbit visits the tiger. The tiger is big. The tiger is rough. The tiger visits the dog. The tiger visits the mouse. The tiger visits the rabbit. If someone is round then they visit the rabbit. If the rabbit does not need the dog then the dog sees the tiger. If the mouse is big then the mouse needs the rabbit. If someone is kind and big then they are rough. If someone needs the rabbit then they need the mouse. If someone is green and round then they see the rabbit. If someone needs the mouse then the mouse is big. If someone visits the dog then the dog needs the rabbit. If someone is green and they do not see the tiger then they visit the mouse.

  1. The mouse visits the dog.If someone visits the dog then the dog needs the rabbit.The dog needs the rabbit.
  2. If someone needs the rabbit then they need the mouse.The dog needs the mouse.
  3. If someone needs the mouse then the mouse is big.The mouse is big.
  4. If the mouse is big then the mouse needs the rabbit.The mouse needs the rabbit.

Depth 5, closed-world task

43.0%
right on 129 of 300 problems wrong; chance 50% (dashed line)
right

Fiona is not smart.

Correct answer: true · Model said: true (79.5% sure)

The problem and its proof

Anne is furry. Charlie is blue. Charlie is kind. Charlie is rough. Fiona is kind. Gary is kind. Gary is nice. Furry people are smart. If someone is nice and not kind then they are smart. If someone is nice then they are rough. If someone is furry and kind then they are rough. Smart people are rough. All round people are blue. All round, blue people are furry. All rough people are round. If Gary is not nice then Gary is rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The bald eagle is not cold.

Correct answer: false · Model said: true (76.9% sure)

The problem and its proof

The bald eagle likes the squirrel. The bald eagle needs the cat. The cat eats the squirrel. The cat is young. The cat likes the cow. The cat likes the squirrel. The cow eats the cat. The cow is blue. The cow is round. The cow is young. The cow likes the squirrel. The cow needs the bald eagle. The cow needs the squirrel. The squirrel eats the bald eagle. The squirrel is rough. The squirrel needs the cat. If someone likes the cat then they are cold. If someone likes the cat and they are rough then the cat likes the squirrel. If someone is rough then they like the cat. If someone eats the bald eagle and they are cold then the bald eagle is rough. If someone is cold and they eat the cow then the cow likes the cat.

  1. The squirrel is rough.If someone is rough then they like the cat.The squirrel likes the cat.
  2. If someone likes the cat then they are cold.The squirrel is cold.
  3. + The squirrel eats the bald eagle.If someone eats the bald eagle and they are cold then the bald eagle is rough.The bald eagle is rough.
  4. If someone is rough then they like the cat.The bald eagle likes the cat.
  5. If someone likes the cat then they are cold.The bald eagle is cold.
wrong

Gary is not smart.

Correct answer: false · Model said: true (82.3% sure)

The problem and its proof

Anne is cold. Bob is round. Erin is rough. Erin is round. Erin is white. Gary is blue. Gary is rough. If someone is round then they are white. All white, blue people are young. If someone is white and young then they are smart. All rough people are cold. All white people are round. If someone is young then they are round. All blue, cold people are round. If someone is cold then they are blue.

  1. Gary is rough.All rough people are cold.Gary is cold.
  2. + Gary is blue.All blue, cold people are round.Gary is round.
  3. If someone is round then they are white.Gary is white.
  4. + Gary is blue.All white, blue people are young.Gary is young.
  5. If someone is white and young then they are smart.Gary is smart.
right

Fiona is green.

Correct answer: false · Model said: false (77.8% sure)

The problem and its proof

Anne is green. Anne is nice. Anne is young. Bob is big. Bob is furry. Bob is green. Bob is nice. Erin is young. Fiona is nice. Fiona is quiet. If something is big then it is green. Big, quiet things are furry. If something is young then it is green. If something is young and blue then it is big. If something is nice and furry then it is big. All young, green things are big. Quiet, furry things are young. If Erin is blue then Erin is furry.

No proof to show: under the closed world this answer holds because something cannot be derived.

What Kev-4B got right and wrong, depth by depth

At each proof depth, Kev-4B's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

87.7%
right on 263 of 300 problems wrong; chance 33% (dashed line)
right

Harry is round.

Correct answer: true · Model said: true (90.8% sure)

The problem and its proof

Gary is young. Harry is nice. Harry is round. All quiet things are smart. All kind things are nice. Nice things are quiet.

Stated in the text: "Harry is round."

right

Erin is big.

Correct answer: unknown · Model said: unknown (61.6% sure)

The problem and its proof

Anne is quiet. Anne is smart. Erin is green. Smart things are kind. Young, green things are kind. If Anne is young and Anne is green then Anne is smart. If Anne is kind then Anne is young. If something is kind and young then it is not red. Big things are young. All red, big things are not smart. If Erin is not young and Erin is not green then Erin is quiet.

No proof: neither the statement nor its negation follows.

right

Erin is rough.

Correct answer: false · Model said: false (84.7% sure)

The problem and its proof

Anne is cold. Anne is not kind. Anne is red. Anne is round. Bob is cold. Bob is kind. Erin is nice. Erin is red. Erin is not rough. Erin is not round. Harry is nice. Harry is smart. If someone is rough then they are red.

Stated in the text: "Erin is not rough."

right

Dave is not kind.

Correct answer: true · Model said: true (90.1% sure)

The problem and its proof

Bob is green. Dave is not kind. Fiona is cold. Gary is not white. If someone is cold and not green then they are round. Cold, green people are red. White people are not red. If someone is white and not cold then they are red. Green people are white. Red people are kind.

Stated in the text: "Dave is not kind."

Depth 1, open-world task

62.9%
right on 190 of 302 problems wrong; chance 33% (dashed line)
right

The cow is kind.

Correct answer: unknown · Model said: unknown (87.6% sure)

The problem and its proof

The cow needs the mouse. The mouse is big. The squirrel visits the mouse. If someone visits the mouse then the mouse visits the squirrel. If someone eats the squirrel then the squirrel eats the cow. If the mouse is big then the mouse is nice. If someone is kind then they eat the squirrel. If someone needs the cow and the cow visits the mouse then they eat the cow. All nice people are kind.

No proof: neither the statement nor its negation follows.

wrong

Dave is not rough.

Correct answer: false · Model said: unknown (75.9% sure)

paraphrased
The problem and its proof

Alan seems to be round. Bob may be round, but he is also kind. That person with the super big round body wearing red and green clothes who is acting quite nice towards you while being cold and feeling blue is named Dave. Fred may be round, but he is also kind. If someone you know is big, cold, and young, you know they are also kind. A red hued and rough looking person is definitely a young person. Most round and cold people are often rough. A young aged and big blue person will definitely be cold.

The proof that the statement's negation follows:

  1. Dave is round. + Dave is cold.If someone is round and cold then they are rough.Dave is rough.
right

Bob is nice.

Correct answer: unknown · Model said: unknown (80.0% sure)

The problem and its proof

Bob is cold. Erin is cold. Erin is nice. If Erin is kind and Erin is nice then Erin is round. All green, kind people are round. If Erin is kind and Erin is round then Erin is white. Nice people are kind. Green, cold people are nice. All nice, furry people are cold. If Erin is round then Erin is cold. Nice, cold people are furry.

No proof: neither the statement nor its negation follows.

right

Alan is big.

Correct answer: unknown · Model said: unknown (88.3% sure)

paraphrased
The problem and its proof

Alan seems to be round. Fred can be rough and cold, but he is also green, an avid gardener. That guy Gary sure is nice. Harry who frequently wears red clothes always feels very cold because of his young age on the team and his rough skin. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Big people who are red in color are almost always young. A person that is both nice and rough is someone who is also big. A nice and green man or woman is also red in color.

No proof: neither the statement nor its negation follows.

Depth 2, open-world task

52.1%
right on 158 of 303 problems wrong; chance 33% (dashed line)
right

The dog does not like the bear.

Correct answer: true · Model said: true (49.5% sure)

The problem and its proof

The bear chases the dog. The dog is big. The dog needs the bear. If someone is cold and they do not need the bear then they do not like the bear. If someone needs the bear then the bear likes the dog. If someone likes the dog then the dog does not like the bear.

  1. The dog needs the bear.If someone needs the bear then the bear likes the dog.The bear likes the dog.
  2. If someone likes the dog then the dog does not like the bear.The dog does not like the bear.
wrong

Eric is big.

Correct answer: true · Model said: unknown (78.8% sure)

paraphrased
The problem and its proof

Dave is green and cold too. As much as Eric is round, he is also rough, green and blue. Harry is young and green. This makes him cold to others. A big, rough person who is green in color is definitely young aged. A nice, green, big person is also sure to be a red person. Blue eyed people, green with sickness and rough around the edges quickly turn red and blush when stepping ashore. People that are rough and red in the face, are usually big in size. If someone shows that they are red, then they are also showing that they are green. When you run into someone who is rough and green at the same time, they will also be red.

  1. Eric is rough. + Eric is green.If someone is rough and green then they are red.Eric is red.
  2. + Eric is rough.If someone is red and rough then they are big.Eric is big.
wrong

The cat is rough.

Correct answer: false · Model said: unknown (79.2% sure)

The problem and its proof

The cat visits the dog. The dog needs the cat. The dog visits the cat. If someone visits the cat and the cat visits the dog then the cat likes the dog. If someone visits the dog then the dog is blue. If someone needs the cat and they are big then the cat does not visit the dog. If someone likes the dog and the dog visits the cat then they are not rough. If someone is nice then they like the cat. If someone is blue then they visit the cat.

The proof that the statement's negation follows:

  1. The dog visits the cat. + The cat visits the dog.If someone visits the cat and the cat visits the dog then the cat likes the dog.The cat likes the dog.
  2. + The dog visits the cat.If someone likes the dog and the dog visits the cat then they are not rough.The cat is not rough.
right

Harry is not furry.

Correct answer: true · Model said: true (65.3% sure)

The problem and its proof

Anne is big. Bob is big. Bob is blue. Bob is young. Fiona is round. Fiona is young. Harry is not big. Harry is blue. Harry is nice. Harry is not round. Harry is young. Green, blue things are young. If something is furry and young then it is nice. All big, round things are nice. All nice, blue things are green. If something is green and young then it is not furry. If Harry is young then Harry is nice. Big things are blue. Big things are round.

  1. Harry is nice. + Harry is blue.All nice, blue things are green.Harry is green.
  2. + Harry is young.If something is green and young then it is not furry.Harry is not furry.

Depth 3, open-world task

44.6%
right on 135 of 303 problems wrong; chance 33% (dashed line)
wrong

The dog does not need the dog.

Correct answer: true · Model said: unknown (50.6% sure)

The problem and its proof

The dog is not round. The dog likes the lion. The dog needs the rabbit. The dog does not visit the lion. The lion is big. The lion does not like the mouse. The mouse is nice. The mouse likes the rabbit. The mouse does not need the lion. The mouse visits the rabbit. The rabbit is big. The rabbit is nice. The rabbit is not round. The rabbit likes the dog. The rabbit likes the lion. The rabbit does not visit the lion. If someone visits the mouse then they are big. If someone likes the mouse then the mouse likes the rabbit. If someone visits the mouse then the mouse needs the rabbit. If someone is nice then they need the mouse. If the dog likes the rabbit and the rabbit needs the lion then the rabbit likes the lion. If someone is big then they do not need the dog. If someone needs the rabbit then they visit the mouse.

  1. The dog needs the rabbit.If someone needs the rabbit then they visit the mouse.The dog visits the mouse.
  2. If someone visits the mouse then they are big.The dog is big.
  3. If someone is big then they do not need the dog.The dog does not need the dog.
right

The mouse does not need the tiger.

Correct answer: unknown · Model said: unknown (47.4% sure)

The problem and its proof

The dog does not eat the mouse. The dog does not eat the tiger. The dog is round. The dog likes the lion. The dog likes the mouse. The lion eats the tiger. The mouse is not cold. The mouse is round. The mouse needs the dog. The tiger is blue. The tiger is rough. The tiger does not need the lion. If someone likes the mouse and they are blue then the mouse eats the dog. If someone likes the mouse and the mouse is blue then the mouse is rough. If someone is cold then they need the tiger. If someone eats the lion then they need the tiger. If someone likes the dog then they are rough. If someone is blue and they eat the lion then the lion is rough. If someone is rough then they eat the lion. If someone needs the tiger then they do not like the lion.

No proof: neither the statement nor its negation follows.

wrong

Dave is not round.

Correct answer: false · Model said: unknown (54.6% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Bob may be round, but he is also kind. Green, big and red are qualities which all describe Dave. That guy Eric sure is nice. Anyone feeling cold is bound to act nice. Big people with red hair are cold because they cannot find coats that fit. People who are said to be big and nice are round. Having cold and red qualities means a person will also have a rough quality.

The proof that the statement's negation follows:

  1. Dave is red. + Dave is big.If someone is red and big then they are cold.Dave is cold.
  2. If someone is cold then they are nice.Dave is nice.
  3. + Dave is big.If someone is big and nice then they are round.Dave is round.
right

Charlie is rough.

Correct answer: false · Model said: false (49.9% sure)

The problem and its proof

Charlie is nice. Charlie is young. Dave is cold. Cold people are furry. All nice people are cold. If someone is young then they are white. If someone is white and not cold then they are not young. If Charlie is young then Charlie is white. If someone is young and not cold then they are rough. All furry people are not rough. Young, blue people are rough.

The proof that the statement's negation follows:

  1. Charlie is nice.All nice people are cold.Charlie is cold.
  2. Cold people are furry.Charlie is furry.
  3. All furry people are not rough.Charlie is not rough.

Depth 4, open-world task

37.3%
right on 113 of 303 problems wrong; chance 33% (dashed line)
right

Fred is green.

Correct answer: unknown · Model said: unknown (86.8% sure)

paraphrased
The problem and its proof

That guy Dave sure is nice. Eric is kind. He is also very cold and blue. That guy Fred sure is nice. Gary is green and cold too. A person who is cold and blue is nice. Every single big person is a little green in some areas. Even though young people are rough, they are still very red. If you pass someone who is red and round and kind, you'll notice they act in a cold manner. A person that is both nice and rough is someone who is also big. Young people that are relatively nice and kind to others, can also tend to be rough. A nice person with cold skin is going to be rough.

No proof: neither the statement nor its negation follows.

wrong

Fiona is not cold.

Correct answer: true · Model said: unknown (60.8% sure)

The problem and its proof

Charlie is not red. Erin is big. Erin is not red. Erin is not white. Fiona is nice. Harry is not cold. Harry is white. Cold people are not big. If Erin is big then Erin is not green. Nice people are quiet. If someone is green and quiet then they are white. All white people are not cold. If Erin is nice then Erin is white. All quiet people are green. If someone is quiet and not cold then they are red.

  1. Fiona is nice.Nice people are quiet.Fiona is quiet.
  2. All quiet people are green.Fiona is green.
  3. If someone is green and quiet then they are white.Fiona is white.
  4. All white people are not cold.Fiona is not cold.
wrong

The lion is young.

Correct answer: true · Model said: unknown (72.5% sure)

The problem and its proof

The bear eats the cow. The bear likes the lion. The bear likes the mouse. The cow chases the bear. The cow chases the mouse. The cow eats the bear. The cow is nice. The lion chases the cow. The lion chases the mouse. The lion likes the bear. The mouse chases the cow. The mouse chases the lion. If someone is young then they are round. If someone eats the lion then they are young. If someone eats the bear then they eat the lion. If someone is nice and they chase the lion then they like the bear. If someone eats the lion then the lion eats the bear.

  1. The cow eats the bear.If someone eats the bear then they eat the lion.The cow eats the lion.
  2. If someone eats the lion then the lion eats the bear.The lion eats the bear.
  3. If someone eats the bear then they eat the lion.The lion eats the lion.
  4. If someone eats the lion then they are young.The lion is young.
wrong

The cow is not blue.

Correct answer: false · Model said: unknown (58.0% sure)

The problem and its proof

The bear is big. The bear is red. The bear likes the dog. The bear does not like the mouse. The bear needs the cow. The bear needs the dog. The bear does not see the mouse. The cow is not big. The cow is not red. The dog is big. The dog is cold. The dog likes the mouse. The dog needs the bear. The dog needs the mouse. The dog sees the cow. The mouse is blue. If something needs the dog then the dog is blue. If the dog is red and the dog needs the cow then the dog sees the mouse. All blue things are kind. If the dog likes the cow then the cow is blue. If something sees the bear and it is not cold then it needs the dog. If something is blue and not red then it likes the mouse. If something needs the cow then it likes the cow. If something is kind then it likes the cow. If something likes the cow and it is not blue then the cow sees the mouse.

The proof that the statement's negation follows:

  1. The bear needs the dog.If something needs the dog then the dog is blue.The dog is blue.
  2. All blue things are kind.The dog is kind.
  3. If something is kind then it likes the cow.The dog likes the cow.
  4. If the dog likes the cow then the cow is blue.The cow is blue.

Depth 5, open-world task

36.3%
right on 105 of 289 problems wrong; chance 33% (dashed line)
wrong

The mouse is not round.

Correct answer: false · Model said: unknown (76.6% sure)

The problem and its proof

The bald eagle eats the tiger. The mouse visits the bald eagle. The squirrel is kind. The squirrel needs the bald eagle. The squirrel needs the tiger. The tiger eats the squirrel. The tiger is red. If someone is kind then they visit the tiger. If someone eats the mouse then the mouse visits the tiger. If the tiger eats the mouse then the tiger needs the mouse. If someone visits the tiger then they are round. If the squirrel is round then the squirrel eats the mouse. If the squirrel visits the tiger then the tiger is round. If someone is round then they visit the tiger.

The proof that the statement's negation follows:

  1. The squirrel is kind.If someone is kind then they visit the tiger.The squirrel visits the tiger.
  2. If someone visits the tiger then they are round.The squirrel is round.
  3. If the squirrel is round then the squirrel eats the mouse.The squirrel eats the mouse.
  4. If someone eats the mouse then the mouse visits the tiger.The mouse visits the tiger.
  5. If someone visits the tiger then they are round.The mouse is round.
wrong

The tiger is red.

Correct answer: true · Model said: unknown (51.7% sure)

The problem and its proof

The bear chases the dog. The bear is young. The dog chases the tiger. The dog likes the bear. The dog likes the lion. The dog likes the tiger. The lion chases the tiger. The lion eats the bear. The lion likes the bear. The lion likes the dog. The tiger eats the lion. The tiger is cold. The tiger is young. The tiger likes the bear. If something likes the lion then it chases the lion. If something chases the dog and it is red then the dog is cold. If something likes the tiger then it likes the lion. If something chases the bear and the bear is round then the bear chases the lion. If something chases the lion then it is red. If something eats the dog then the dog chases the lion. If something is round then it likes the tiger. All young things are round.

  1. The tiger is young.All young things are round.The tiger is round.
  2. If something is round then it likes the tiger.The tiger likes the tiger.
  3. If something likes the tiger then it likes the lion.The tiger likes the lion.
  4. If something likes the lion then it chases the lion.The tiger chases the lion.
  5. If something chases the lion then it is red.The tiger is red.
right

Dave is not young.

Correct answer: unknown · Model said: unknown (65.5% sure)

The problem and its proof

Dave is furry. Gary is young. Harry is white. If something is nice then it is young. All kind, rough things are young. All kind things are not young. All white things are nice. If something is white then it is not kind. Young things are rough. If Gary is furry then Gary is rough. If something is rough then it is white.

No proof: neither the statement nor its negation follows.

wrong

Anne is nice.

Correct answer: false · Model said: unknown (51.0% sure)

The problem and its proof

Anne is red. Anne is rough. Charlie is not blue. Charlie is green. Charlie is white. Gary is not blue. Gary is furry. Gary is not green. Gary is white. Harry is not blue. Harry is furry. Harry is nice. Harry is not red. Harry is white. Red people are blue. All furry, nice people are not rough. Green people are not nice. If Anne is blue and Anne is white then Anne is green. Furry, blue people are white. Green people are white. If Gary is green then Gary is not blue. Blue, red people are furry. If Harry is not nice then Harry is white.

The proof that the statement's negation follows:

  1. Anne is red.Red people are blue.Anne is blue.
  2. + Anne is red.Blue, red people are furry.Anne is furry.
  3. Furry, blue people are white.Anne is white.
  4. If Anne is blue and Anne is white then Anne is green.Anne is green.
  5. Green people are not nice.Anne is not nice.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

85.7%
right on 257 of 300 problems wrong; chance 50% (dashed line)
wrong

Fred is not kind.

Correct answer: true · Model said: false (67.3% sure)

paraphrased
The problem and its proof

Alan is green and cold too. For being so cold, it's good Fred can remain nice. Gary seems to be round. No one knows Harry like me and he is a kind guy, very round in the belly and green as grass. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Kind red people are green on the inside. Even though young people are rough, they are still very red. Nice people that are very green and even round shaped will be very young. Round people who are kind tend to be nice. A green person that is rough and cold is often also nice. Someone who is a young age and looks like they are round are also red.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The squirrel is not green.

Correct answer: true · Model said: true (78.1% sure)

The problem and its proof

The bear chases the lion. The bear eats the lion. The bear is kind. The bear is rough. The bear needs the lion. The bear needs the squirrel. The lion chases the bear. The lion is rough. The lion is young. The lion needs the squirrel. The squirrel chases the lion. The squirrel is young. If something is young then it is rough. If something is kind then it needs the bear. Rough things are kind.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The rabbit is not nice.

Correct answer: true · Model said: true (53.3% sure)

The problem and its proof

The cat eats the rabbit. The cat is big. The cat is cold. The cat is round. The cat visits the rabbit. The rabbit eats the cat. The rabbit is red. The rabbit is round. The rabbit needs the cat. The rabbit visits the cat. If the cat eats the rabbit and the cat needs the rabbit then the cat is red. If something needs the cat and the cat eats the rabbit then the cat needs the rabbit.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Gary is cold.

Correct answer: false · Model said: false (78.0% sure)

paraphrased
The problem and its proof

The young person who is always feeling cold is named Dave. Thankfully Eric is nice. People have realized that he is big and rough. Since he is young he has tendency to become red when they mention it. Gary seems to be round. Harry seems to be round. If someone plays rough and is nice and round, they will be big. A big round young person is often blue. Nice people who feel blue often look green when ill. A nice person is inevitably round as well. If a person is both cold and round, that person is also someone who is red. If you know someone who is nice, green, and big, you know a young person. A person who is cold and red and blue is big.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

68.3%
right on 205 of 300 problems wrong; chance 50% (dashed line)
right

The dog is big.

Correct answer: true · Model said: true (70.3% sure)

The problem and its proof

The dog is rough. Rough things are big.

  1. The dog is rough.Rough things are big.The dog is big.
right

The lion likes the tiger.

Correct answer: true · Model said: true (84.0% sure)

The problem and its proof

The bald eagle is blue. The lion visits the bald eagle. The mouse visits the lion. The tiger eats the mouse. If something visits the bald eagle and it likes the tiger then the bald eagle is big. If something visits the lion then the lion likes the tiger. If something visits the tiger and the tiger visits the mouse then the mouse eats the lion. Rough things are big. If the lion is rough then the lion is nice. If something likes the mouse then it is rough.

  1. The mouse visits the lion.If something visits the lion then the lion likes the tiger.The lion likes the tiger.
wrong

The bald eagle likes the bald eagle.

Correct answer: true · Model said: false (78.5% sure)

The problem and its proof

The bald eagle eats the cat. The bald eagle eats the cow. The bald eagle is big. The bald eagle is cold. The bald eagle is red. The bald eagle is round. The bald eagle likes the cat. The bald eagle likes the cow. The cat chases the bald eagle. The cat chases the cow. The cat likes the cow. The cow chases the cat. The cow eats the cat. The cow is kind. The cow likes the cat. If something eats the cow then it likes the bald eagle. If something is big and it likes the bald eagle then the bald eagle chases the cat.

  1. The bald eagle eats the cow.If something eats the cow then it likes the bald eagle.The bald eagle likes the bald eagle.
right

The lion does not like the lion.

Correct answer: false · Model said: false (52.2% sure)

The problem and its proof

The lion is blue. The lion is kind. The lion likes the mouse. The lion needs the mouse. The mouse eats the lion. The mouse is blue. The mouse is kind. The mouse is young. The mouse likes the lion. The mouse needs the lion. If something likes the mouse then it likes the lion. If something likes the lion then it needs the lion.

  1. The lion likes the mouse.If something likes the mouse then it likes the lion.The lion likes the lion.

Depth 2, closed-world task

54.3%
right on 163 of 300 problems wrong; chance 50% (dashed line)
wrong

Fred is red.

Correct answer: true · Model said: false (67.2% sure)

paraphrased
The problem and its proof

Bob is kind and nice and looks green. Eric, who is relatively young, is also pretty big and tends to be cold. Fred is kind. He is also very cold and blue. That guy Harry sure is nice. People here that are both big and kind while being cold are always red. Kind red people are green on the inside. When someone is kind yet can be cold and blue, they will also be very big. Kind red and round people often look green. Is a kind person if feeling cold and looking red then they are usually young.

  1. Fred is kind. + Fred is cold. + Fred is blue.If someone is kind and cold and blue then they are big.Fred is big.
  2. + Fred is kind. + Fred is cold.If someone is big and kind and cold then they are red.Fred is red.
wrong

Gary is not green.

Correct answer: true · Model said: false (52.8% sure)

paraphrased
The problem and its proof

Bob is cold to his ex-wife. She knows he is round and red. He is kind to everyone else. Eric is known for being big and rough, but few know he is also always red with cold. For being so cold, it's good Gary can remain nice. For being so cold, it's good Harry can remain nice. Round people who feel blue and are green in color are often young in age. Rough people who are red in color and big are usally round in shape. Kind red people are green on the inside. A rather big person is a red person. A nice but rough person will also be green and cold. A big and round individual is sure to be a kind individual, too. A person that is red, rough and kind often feels cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Bob is blue.

Correct answer: false · Model said: false (75.8% sure)

paraphrased
The problem and its proof

Alan is green and cold too. Bob seems to be round. Eric who is young and round is also big and rough. When a person is nice and round and cold, they look blue. Even though young people are rough, they are still very red. A person feeling blue because of their big size will always be cold. Young people who feel cold and are still kind usually look rough. Red people, who are also round and young, are often quite blue as well. I bet anyone who is round and kind is also pretty young.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The bear likes the tiger.

Correct answer: false · Model said: false (61.1% sure)

The problem and its proof

The bear needs the mouse. The dog is big. The dog is cold. The dog is nice. The dog likes the mouse. The dog needs the mouse. The mouse eats the tiger. The mouse needs the dog. The tiger likes the bear. The tiger likes the mouse. If something is nice then it likes the mouse. If something is nice then it likes the tiger. If something likes the mouse then the mouse needs the tiger. If something likes the bear then the bear is cold. If something is cold and it likes the mouse then it likes the tiger. Nice things are cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

56.0%
right on 168 of 300 problems wrong; chance 50% (dashed line)
right

Fred is not blue.

Correct answer: true · Model said: true (59.7% sure)

paraphrased
The problem and its proof

For being so cold, it's good Alan can remain nice. Charlie is a kind person and he is also often cold. Eric seems to be round. Fred is green and cold too. People here that are both big and kind while being cold are always red. Some cold people can be nice at times,and red at at other times. Every time you meet someone kind and nice, they'll be green, too. Rough, cold people are blue. A person who is blue, red and nice is usually considered to be a young person. Having cold and red qualities means a person will also have a rough quality. Big people with rough, green skin are cold because of it.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The cat eats the mouse.

Correct answer: true · Model said: false (86.5% sure)

The problem and its proof

The cat is big. The cat is rough. The cat is round. The cat needs the mouse. The cat needs the tiger. The mouse eats the rabbit. The mouse is kind. The mouse is round. The mouse visits the cat. The rabbit eats the tiger. The rabbit is kind. The rabbit needs the mouse. The tiger eats the cat. The tiger is rough. The tiger is young. The tiger visits the cat. If someone needs the tiger then they eat the rabbit. If someone is big then they need the tiger. If someone is kind then they eat the mouse. If someone needs the cat then the cat visits the mouse. If the mouse does not visit the cat then the mouse eats the rabbit. If someone eats the rabbit then they are kind. If someone eats the rabbit and they visit the rabbit then they are round. If the cat is rough then the cat needs the rabbit.

  1. The cat needs the tiger.If someone needs the tiger then they eat the rabbit.The cat eats the rabbit.
  2. If someone eats the rabbit then they are kind.The cat is kind.
  3. If someone is kind then they eat the mouse.The cat eats the mouse.
right

Harry is not red.

Correct answer: true · Model said: true (62.1% sure)

The problem and its proof

Erin is green. Erin is white. Fiona is green. Fiona is nice. Harry is cold. Harry is green. Harry is smart. Red things are blue. If Erin is cold then Erin is white. Nice things are red. If Harry is blue then Harry is nice. If something is smart and cold then it is green. All green, white things are nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Bob is big.

Correct answer: true · Model said: false (70.0% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Young Bob has a round form, believes in green and is rough. Fred seems to be round. For being so cold, it's good Gary can remain nice. All young and kind people that feel blue are described as red. You'll always find rough, cold, green people to also be red people. Interesting that all round people are cold. People that are rough and red in the face, are usually big in size.

  1. Bob is round.If someone is round then they are cold.Bob is cold.
  2. + Bob is rough. + Bob is green.If someone is cold and rough and green then they are red.Bob is red.
  3. + Bob is rough.If someone is red and rough then they are big.Bob is big.

Depth 4, closed-world task

45.0%
right on 135 of 300 problems wrong; chance 50% (dashed line)
wrong

The cat does not like the cat.

Correct answer: true · Model said: false (51.2% sure)

The problem and its proof

The cat likes the dog. The cat likes the lion. The cat sees the dog. The dog likes the rabbit. The dog needs the cat. The lion likes the dog. The lion likes the rabbit. The lion needs the dog. The lion sees the cat. The rabbit is round. The rabbit likes the cat. The rabbit likes the dog. If someone is young then they need the dog. All nice people are young. If someone likes the cat then they see the dog. If someone is round and they need the dog then the dog likes the cat. If the rabbit needs the cat and the rabbit is not kind then the rabbit is young. If someone needs the cat then they are nice. If someone needs the dog and the dog needs the cat then they like the cat.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Charlie is not nice.

Correct answer: true · Model said: false (59.0% sure)

The problem and its proof

Bob is big. Bob is blue. Bob is cold. Bob is white. Bob is young. Charlie is big. Charlie is red. Charlie is white. Dave is blue. Dave is red. Dave is white. Fiona is big. Fiona is blue. Fiona is red. Fiona is white. If Charlie is cold and Charlie is young then Charlie is big. If someone is cold and young then they are nice. If Fiona is blue and Fiona is cold then Fiona is young. If someone is blue then they are cold. All nice, big people are young. All cold people are young. All nice, big people are blue. Young people are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The mouse is not green.

Correct answer: false · Model said: false (59.5% sure)

The problem and its proof

The bear chases the cow. The bear is green. The cow chases the mouse. The cow chases the rabbit. The mouse chases the bear. The mouse chases the cow. The mouse likes the bear. The rabbit chases the bear. The rabbit chases the cow. The rabbit likes the bear. The rabbit sees the mouse. If someone likes the mouse then the mouse likes the cow. If someone likes the cow then they chase the mouse. If someone chases the mouse then they are green. If someone chases the bear and they see the bear then the bear likes the mouse. If the bear is green and the bear sees the cow then the cow sees the bear. If someone is blue and they like the cow then they are rough. If someone is green then they like the mouse. If someone likes the rabbit then the rabbit sees the bear.

  1. The bear is green.If someone is green then they like the mouse.The bear likes the mouse.
  2. If someone likes the mouse then the mouse likes the cow.The mouse likes the cow.
  3. If someone likes the cow then they chase the mouse.The mouse chases the mouse.
  4. If someone chases the mouse then they are green.The mouse is green.
right

Eric is not young.

Correct answer: false · Model said: false (53.8% sure)

paraphrased
The problem and its proof

Even though Bob is big and rough, he is kind. Bob is green with envy. That guy Dave sure is nice. Eric was proud of being round, yet rough. His red cheeks glowed. Fred who is young and round is also big and rough. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Any person that's blue, young and green will turn out to be a nice person, too. A red, nice person will definitely be a blue person. Why is it that every round person that is blue is so young. Rough people who look red are cold. A nice person with cold skin is going to be rough. A person who inherits genes that make them big and green will also express a blue color.

  1. Eric is red. + Eric is rough.If someone is red and rough then they are cold.Eric is cold.
  2. + Eric is rough.If someone is cold and rough then they are nice.Eric is nice.
  3. + Eric is red.If someone is red and nice then they are blue.Eric is blue.
  4. + Eric is round.If someone is blue and round then they are young.Eric is young.

Depth 5, closed-world task

43.3%
right on 130 of 300 problems wrong; chance 50% (dashed line)
wrong

Anne is not big.

Correct answer: false · Model said: true (53.4% sure)

The problem and its proof

Anne is blue. Anne is nice. Bob is kind. Bob is nice. Fiona is blue. Fiona is furry. Fiona is kind. Fiona is nice. Fiona is smart. Gary is blue. Gary is furry. Gary is smart. Big, furry things are green. Kind, nice things are big. All smart, green things are kind. All nice things are furry. If Anne is kind and Anne is big then Anne is green. If Fiona is blue then Fiona is green. Furry, green things are smart. Furry things are green.

  1. Anne is nice.All nice things are furry.Anne is furry.
  2. Furry things are green.Anne is green.
  3. Furry, green things are smart.Anne is smart.
  4. All smart, green things are kind.Anne is kind.
  5. + Anne is nice.Kind, nice things are big.Anne is big.
right

Dave is not furry.

Correct answer: true · Model said: true (54.8% sure)

The problem and its proof

Charlie is big. Dave is red. Harry is smart. If Dave is green and Dave is smart then Dave is big. Red people are kind. If someone is red and blue then they are big. If someone is big then they are smart. If someone is furry and kind then they are smart. All blue, smart people are furry. If someone is smart then they are green. If someone is smart and big then they are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The cow sees the cow.

Correct answer: true · Model said: false (79.9% sure)

The problem and its proof

The cat chases the cow. The cow chases the lion. The cow is round. The cow sees the lion. The lion chases the mouse. The lion is red. The mouse sees the cat. If someone chases the mouse then they chase the cat. If someone chases the cat then they see the cow. If someone is blue then they chase the cow. If someone chases the lion and they eat the mouse then the lion eats the cat. If someone sees the cow then the cow chases the mouse. If the cat sees the mouse and the cat is not red then the mouse eats the cat.

  1. The lion chases the mouse.If someone chases the mouse then they chase the cat.The lion chases the cat.
  2. If someone chases the cat then they see the cow.The lion sees the cow.
  3. If someone sees the cow then the cow chases the mouse.The cow chases the mouse.
  4. If someone chases the mouse then they chase the cat.The cow chases the cat.
  5. If someone chases the cat then they see the cow.The cow sees the cow.
right

Erin is not green.

Correct answer: true · Model said: true (72.3% sure)

The problem and its proof

Dave is green. Dave is kind. Dave is rough. Erin is cold. Gary is blue. Harry is green. Harry is nice. Green things are kind. If something is rough and not nice then it is green. If something is blue then it is red. If something is green and not nice then it is rough. All red things are rough. All rough, green things are cold. If something is green and not nice then it is cold. If something is kind then it is blue. If Gary is red and Gary is kind then Gary is green.

No proof to show: under the closed world this answer holds because something cannot be derived.

What Kev-0.8B got right and wrong, depth by depth

At each proof depth, Kev-0.8B's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

70.3%
right on 211 of 300 problems wrong; chance 33% (dashed line)
right

Fred is not green.

Correct answer: false · Model said: false (93.5% sure)

paraphrased
The problem and its proof

Rough and cold that is what they say about Blue Bob. Eric, who is relatively young, is also pretty big and tends to be cold. Fred is green and cold too. For being so cold, it's good Harry can remain nice. Rough, cold people are blue. Big, kind folks are green ones. Even though a person is big and rough when they are cold they turn red. Most round and cold people are often rough. Cold, young people are also certain to be rough people. An individual who is big, red and young is also going to be a nice individual.

Stated in the text: "Fred is green."

right

Charlie is green.

Correct answer: false · Model said: false (44.0% sure)

The problem and its proof

Bob is kind. Bob is quiet. Bob is young. Charlie is not green. Dave is round. Fiona is rough. Fiona is round. Kind, blue people are quiet. Rough, round people are quiet. Round people are rough. If someone is blue then they are green. All rough, kind people are green. If someone is quiet then they are blue. All green, round people are young. All kind people are round. If Charlie is rough then Charlie is blue.

Stated in the text: "Charlie is not green."

right

Erin is not nice.

Correct answer: true · Model said: true (58.9% sure)

The problem and its proof

Anne is nice. Anne is white. Charlie is round. Charlie is white. Erin is not nice. Erin is quiet. Gary is not quiet. Blue people are big. If someone is white then they are green. If Anne is quiet then Anne is blue. If Anne is green then Anne is quiet. Blue people are quiet. If Anne is white and Anne is not blue then Anne is nice. Big people are round. If someone is big and not nice then they are round.

Stated in the text: "Erin is not nice."

wrong

The dog does not like the dog.

Correct answer: unknown · Model said: false (55.2% sure)

The problem and its proof

The cat sees the dog. The cat visits the squirrel. The dog is blue. The dog does not like the cat. The dog does not like the lion. The dog likes the squirrel. The lion is blue. The lion is nice. The lion likes the cat. The lion likes the dog. The squirrel is not red. The squirrel sees the cat. Blue things are nice. If something sees the squirrel then the squirrel sees the dog. If something likes the squirrel and it sees the dog then the dog does not visit the squirrel.

No proof: neither the statement nor its negation follows.

Depth 1, open-world task

54.0%
right on 163 of 302 problems wrong; chance 33% (dashed line)
wrong

Fred is not blue.

Correct answer: unknown · Model said: false (71.6% sure)

paraphrased
The problem and its proof

Alan, who is relatively young, is also pretty big and tends to be cold. Bob is green and cold too. Even though Harry is big and rough, he is kind. Harry is green with envy. Anyone who is both red and green must be as cold as Christmas. A big and round individual is sure to be a kind individual, too. A person that is red, rough and kind often feels cold. Blue eyed people, green with sickness and rough around the edges quickly turn red and blush when stepping ashore. Young people who are both nice and cold also tend to be rough. A person who inherits genes that make them big and green will also express a blue color.

No proof: neither the statement nor its negation follows.

right

Harry is not rough.

Correct answer: false · Model said: false (83.7% sure)

paraphrased
The problem and its proof

Rough and cold that is what they say about Blue Eric. Green, big and red are qualities which all describe Fred. Harry may be round, but he is also kind. Young and rough people will most certainly be red. If someone has a kind disposition and looks rather round and rough then you'll find that they're actually quite young. A kind person will certainly be rough as well. Nice people who feel blue often look green when ill.

The proof that the statement's negation follows:

  1. Harry is kind.If someone is kind then they are rough.Harry is rough.
right

Eric is cold.

Correct answer: true · Model said: true (42.9% sure)

paraphrased
The problem and its proof

Bob is green and cold too. Eric might be rough and red but he's actually very kind. Fred is green and cold too. Harry can be rough and cold, but he is also green, an avid gardener. A person who is kind, cold, and rough is big. Every single big person is a little green in some areas. When rough and kind can describe a person, then cold will describe them, too. Big, young people with green color are rather rough. If someone is green and naive they may also have red, rough skin. It is a safe guess then that they are also round. Nice people with red and rough skin are green with envy. Green folks who are nice and rough are a round shape.

  1. Eric is kind. + Eric is rough.If someone is kind and rough then they are cold.Eric is cold.
wrong

Eric is not nice.

Correct answer: unknown · Model said: false (86.4% sure)

paraphrased
The problem and its proof

Alan is somehow red and kind and also green. Green, big and red are qualities which all describe Charlie. Eric is a kind person and he is also often cold. Gary was proud of being round, yet rough. His red cheeks glowed. Red people are almost always rough. Nice round big people are kind to others. Someone with rough and green feet is invariably kind. It's a certainty that any green, big and kind individual is going to be nice. Green folks who are nice and rough are a round shape.

No proof: neither the statement nor its negation follows.

Depth 2, open-world task

46.9%
right on 142 of 303 problems wrong; chance 33% (dashed line)
right

The squirrel visits the rabbit.

Correct answer: true · Model said: true (67.5% sure)

The problem and its proof

The bear eats the squirrel. The bear is big. The bear is not nice. The bear visits the cat. The bear visits the rabbit. The cat eats the squirrel. The cat is nice. The rabbit visits the squirrel. The squirrel does not eat the cat. The squirrel is red. The squirrel visits the bear. If the cat is green then the cat does not visit the rabbit. If something needs the rabbit and it visits the cat then the cat visits the bear. If something needs the bear and it is kind then the bear eats the rabbit. If something needs the squirrel and the squirrel needs the bear then the squirrel eats the cat. If something visits the bear and the bear is red then it visits the rabbit. If the rabbit does not need the squirrel then the squirrel eats the bear. If something is red then it needs the rabbit. If something is big and it visits the cat then it is red. If something visits the rabbit then it is not green.

  1. The bear is big. + The bear visits the cat.If something is big and it visits the cat then it is red.The bear is red.
  2. + The squirrel visits the bear.If something visits the bear and the bear is red then it visits the rabbit.The squirrel visits the rabbit.
right

Fiona is not furry.

Correct answer: true · Model said: true (59.9% sure)

The problem and its proof

Anne is furry. Anne is quiet. Erin is furry. Erin is not rough. Fiona is not quiet. Fiona is smart. Harry is quiet. Smart people are white. Quiet people are smart. All smart, rough people are not furry. If someone is white and not quiet then they are not furry. All white people are cold. All furry, quiet people are not rough. If Fiona is smart and Fiona is quiet then Fiona is white. If Harry is cold then Harry is rough. If Erin is white and Erin is not smart then Erin is not cold.

  1. Fiona is smart.Smart people are white.Fiona is white.
  2. + Fiona is not quiet.If someone is white and not quiet then they are not furry.Fiona is not furry.
wrong

Gary is big.

Correct answer: true · Model said: unknown (43.7% sure)

paraphrased
The problem and its proof

Charlie seems to be round. Dave is green and cold too. Fred is a young, round shaped young man who is also very cold. Gary is green and cold too. Young people who are nice and look round are also going to be green. Any person that's blue, young and green will turn out to be a nice person, too. A young, green person in my experience is round shaped and nice. All young and kind people that feel blue are described as red. Anyone feeling cold is bound to act nice. Nice and green people are often found to be big as well.

  1. Gary is cold.If someone is cold then they are nice.Gary is nice.
  2. + Gary is green.If someone is nice and green then they are big.Gary is big.
wrong

Gary is green.

Correct answer: unknown · Model said: true (90.5% sure)

paraphrased
The problem and its proof

Alan was proud of being round, yet rough. His red cheeks glowed. Dave is a young, rough person who is rather round but is also kind to everyone he meets. Eric is quite rough and very kind, but unfortunately cold and also blue. That guy Gary sure is nice. A very big person who is green but also red is a rough person. People who are young and blue are also red. If someone is cold and blue, you will find that they are also quite young. If someone shows that they are red, then they are also showing that they are green. People who are round and green while being cold are also red.

No proof: neither the statement nor its negation follows.

Depth 3, open-world task

48.8%
right on 148 of 303 problems wrong; chance 33% (dashed line)
wrong

Bob is cold.

Correct answer: unknown · Model said: true (69.0% sure)

paraphrased
The problem and its proof

That guy Bob sure is nice. Dave is known for being big and rough, but few know he is also always red with cold. Rough and cold that is what they say about Blue Gary. Harry is somehow red and kind and also green. Kind people that are red and round are usually big. Not a single young person is not also kind. Nice people who are blue and round at the same time are always young. Big people with red hair are cold because they cannot find coats that fit. Most young kind people tend to be red too. If someone is cold and blue, you will find that they are also quite young. Young and red people look round.

No proof: neither the statement nor its negation follows.

right

Fred is young.

Correct answer: unknown · Model said: unknown (45.3% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Eric is nice and kind even though people make fun of his red and green skin. Eric always feels cold. Fred is somehow red and kind and also green. A person that is red, rough and kind often feels cold. Round young people, red with loveliness, are very cold towards others. A young person with red hands is rough around the edges. Is a kind person if feeling cold and looking red then they are usually young. Green folks who are nice and rough are a round shape. A nice and green man or woman is also red in color.

No proof: neither the statement nor its negation follows.

right

The squirrel sees the tiger.

Correct answer: false · Model said: false (37.4% sure)

The problem and its proof

The squirrel likes the tiger. The tiger is cold. If something is young then it chases the tiger. If the squirrel chases the tiger then the squirrel does not see the tiger. If something is cold then it sees the squirrel. If something is round and not cold then it sees the squirrel. If something sees the squirrel then the squirrel chases the tiger. If something likes the squirrel and the squirrel is cold then it is young. If something likes the tiger and the tiger does not see the squirrel then the squirrel likes the tiger. If something chases the squirrel and it is not green then the squirrel likes the tiger.

The proof that the statement's negation follows:

  1. The tiger is cold.If something is cold then it sees the squirrel.The tiger sees the squirrel.
  2. If something sees the squirrel then the squirrel chases the tiger.The squirrel chases the tiger.
  3. If the squirrel chases the tiger then the squirrel does not see the tiger.The squirrel does not see the tiger.
wrong

The tiger needs the mouse.

Correct answer: unknown · Model said: true (63.3% sure)

The problem and its proof

The cat is not red. The cat needs the mouse. The cat needs the tiger. The cow eats the cat. The cow is young. The mouse visits the cat. The tiger eats the cat. The tiger does not eat the cow. The tiger is red. The tiger does not visit the cat. The tiger visits the cow. The tiger visits the mouse. If the tiger visits the mouse then the mouse eats the tiger. If someone is young and they need the mouse then they eat the mouse. If someone eats the tiger then they are green. If someone is green then they are young. If someone is young then they need the mouse.

No proof: neither the statement nor its negation follows.

Depth 4, open-world task

50.5%
right on 153 of 303 problems wrong; chance 33% (dashed line)
wrong

Fiona is not big.

Correct answer: unknown · Model said: false (61.7% sure)

The problem and its proof

Bob is green. Bob is rough. Bob is smart. Dave is big. Dave is blue. Dave is quiet. Dave is rough. Dave is smart. Fiona is rough. Fiona is smart. If someone is smart and big then they are blue. All smart people are furry. Rough, green people are big. All green, rough people are smart. If someone is blue and rough then they are quiet. If Bob is blue and Bob is smart then Bob is big. All quiet people are big. If someone is big then they are furry.

No proof: neither the statement nor its negation follows.

wrong

The mouse eats the rabbit.

Correct answer: unknown · Model said: true (91.4% sure)

The problem and its proof

The lion chases the rabbit. The lion eats the mouse. The lion eats the rabbit. The lion is red. The lion sees the mouse. The lion sees the rabbit. The mouse chases the rabbit. The mouse eats the lion. The mouse is young. The mouse sees the lion. The rabbit chases the mouse. The rabbit eats the lion. The rabbit eats the mouse. The rabbit is big. The rabbit is red. The rabbit sees the lion. If someone sees the rabbit then they chase the lion. If someone chases the mouse and the mouse sees the rabbit then the rabbit eats the mouse. If someone chases the mouse and they eat the lion then they eat the rabbit. Nice people are big. If someone is big then they chase the mouse. If someone sees the mouse then they are nice.

No proof: neither the statement nor its negation follows.

right

The cat does not chase the cat.

Correct answer: false · Model said: false (52.5% sure)

The problem and its proof

The cat does not chase the rabbit. The cat is cold. The cat is red. The cat sees the rabbit. The cat sees the tiger. The mouse is big. The mouse is red. The rabbit does not chase the cat. The rabbit likes the cat. The rabbit does not like the mouse. The tiger likes the mouse. If something likes the mouse then it chases the cat. If something sees the mouse then it chases the cat. If something chases the mouse then the mouse does not like the rabbit. If something chases the cat then it sees the cat. If the tiger sees the cat then the cat sees the mouse.

The proof that the statement's negation follows:

  1. The tiger likes the mouse.If something likes the mouse then it chases the cat.The tiger chases the cat.
  2. If something chases the cat then it sees the cat.The tiger sees the cat.
  3. If the tiger sees the cat then the cat sees the mouse.The cat sees the mouse.
  4. If something sees the mouse then it chases the cat.The cat chases the cat.
right

The squirrel visits the cow.

Correct answer: true · Model said: true (84.6% sure)

The problem and its proof

The cat chases the cow. The cat is blue. The cat is cold. The cat is green. The cat is kind. The cat is red. The cat does not visit the cow. The cow visits the squirrel. The squirrel is cold. The squirrel needs the cow. The tiger is blue. If something is green then it needs the cow. If something chases the cat and the cat chases the cow then it needs the tiger. If the squirrel visits the cat then the squirrel chases the cat. If something chases the squirrel and the squirrel is green then it chases the cat. If something chases the squirrel then the squirrel visits the cat. If something is blue and red then it chases the squirrel. If something chases the cat then it visits the cow. If something visits the cow then the cow does not chase the squirrel.

  1. The cat is blue. + The cat is red.If something is blue and red then it chases the squirrel.The cat chases the squirrel.
  2. If something chases the squirrel then the squirrel visits the cat.The squirrel visits the cat.
  3. If the squirrel visits the cat then the squirrel chases the cat.The squirrel chases the cat.
  4. If something chases the cat then it visits the cow.The squirrel visits the cow.

Depth 5, open-world task

50.9%
right on 147 of 289 problems wrong; chance 33% (dashed line)
right

Charlie is not cold.

Correct answer: true · Model said: true (66.0% sure)

The problem and its proof

Bob is big. Bob is kind. Bob is nice. Charlie is kind. Charlie is young. Erin is young. Harry is furry. If Charlie is kind and Charlie is young then Charlie is furry. If someone is furry and rough then they are big. If someone is young and furry then they are rough. If someone is furry and big then they are nice. If someone is kind and nice then they are not cold.

  1. Charlie is kind. + Charlie is young.If Charlie is kind and Charlie is young then Charlie is furry.Charlie is furry.
  2. + Charlie is young.If someone is young and furry then they are rough.Charlie is rough.
  3. If someone is furry and rough then they are big.Charlie is big.
  4. If someone is furry and big then they are nice.Charlie is nice.
  5. + Charlie is kind.If someone is kind and nice then they are not cold.Charlie is not cold.
wrong

The tiger is red.

Correct answer: true · Model said: unknown (41.9% sure)

The problem and its proof

The bear chases the dog. The bear is young. The dog chases the tiger. The dog likes the bear. The dog likes the lion. The dog likes the tiger. The lion chases the tiger. The lion eats the bear. The lion likes the bear. The lion likes the dog. The tiger eats the lion. The tiger is cold. The tiger is young. The tiger likes the bear. If something likes the lion then it chases the lion. If something chases the dog and it is red then the dog is cold. If something likes the tiger then it likes the lion. If something chases the bear and the bear is round then the bear chases the lion. If something chases the lion then it is red. If something eats the dog then the dog chases the lion. If something is round then it likes the tiger. All young things are round.

  1. The tiger is young.All young things are round.The tiger is round.
  2. If something is round then it likes the tiger.The tiger likes the tiger.
  3. If something likes the tiger then it likes the lion.The tiger likes the lion.
  4. If something likes the lion then it chases the lion.The tiger chases the lion.
  5. If something chases the lion then it is red.The tiger is red.
wrong

The bald eagle visits the bald eagle.

Correct answer: true · Model said: false (47.9% sure)

The problem and its proof

The bald eagle is green. The bald eagle needs the squirrel. The bald eagle sees the tiger. The bald eagle visits the mouse. The mouse is round. The mouse needs the squirrel. The mouse sees the squirrel. The squirrel is big. The squirrel is round. The squirrel needs the mouse. The squirrel visits the mouse. The squirrel visits the tiger. The tiger is big. The tiger is rough. The tiger needs the squirrel. The tiger visits the squirrel. If something visits the bald eagle and it is round then the bald eagle sees the squirrel. If something sees the tiger and it needs the squirrel then the tiger is round. If something is nice then it visits the bald eagle. If something is round then it sees the squirrel. If something sees the squirrel then it is nice.

  1. The mouse sees the squirrel.If something sees the squirrel then it is nice.The mouse is nice.
  2. If something is nice then it visits the bald eagle.The mouse visits the bald eagle.
  3. + The mouse is round.If something visits the bald eagle and it is round then the bald eagle sees the squirrel.The bald eagle sees the squirrel.
  4. If something sees the squirrel then it is nice.The bald eagle is nice.
  5. If something is nice then it visits the bald eagle.The bald eagle visits the bald eagle.
right

The cat is not big.

Correct answer: false · Model said: false (47.3% sure)

The problem and its proof

The bald eagle chases the cat. The bald eagle chases the dog. The bald eagle does not eat the cat. The bald eagle eats the dog. The cat does not chase the bald eagle. The cat eats the bald eagle. The cat is young. The cat sees the bald eagle. The cat sees the tiger. The dog chases the cat. The dog eats the bald eagle. The dog is blue. The tiger chases the bald eagle. The tiger chases the cat. If something is young then it does not chase the cat. If the cat chases the tiger then the tiger is nice. Nice things are big. If something chases the dog and the dog eats the bald eagle then it is nice. If something is big and it chases the cat then the cat chases the dog.

The proof that the statement's negation follows:

  1. The bald eagle chases the dog. + The dog eats the bald eagle.If something chases the dog and the dog eats the bald eagle then it is nice.The bald eagle is nice.
  2. Nice things are big.The bald eagle is big.
  3. + The bald eagle chases the cat.If something is big and it chases the cat then the cat chases the dog.The cat chases the dog.
  4. + The dog eats the bald eagle.If something chases the dog and the dog eats the bald eagle then it is nice.The cat is nice.
  5. Nice things are big.The cat is big.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

68.0%
right on 204 of 300 problems wrong; chance 50% (dashed line)
right

Bob is kind.

Correct answer: true · Model said: true (93.4% sure)

paraphrased
The problem and its proof

Bob may be round, but he is also kind. Rough and cold that is what they say about Blue Charlie. Gary seems to be round. Harry, who is relatively young, is also pretty big and tends to be cold. People,though kind and rough are also really nice. When green, young and round fits a person, you'll see that rough will also fit. Any person that's blue, young and green will turn out to be a nice person, too. A person that is red, rough and kind often feels cold. Do not fear big cold people for they are often kind. Watch out for young people that are kind for they are rough. A person who is rough and young while being kind tends to be red.

Stated in the text: "Bob is kind."

wrong

Fred is not cold.

Correct answer: false · Model said: true (72.2% sure)

paraphrased
The problem and its proof

Dave may be round, but he is also kind. The young person who is always feeling cold is named Fred. Harry woke up one day to find he had become round. He was still rough, red, and green, but now he was also big. Red people are almost always rough. People,though kind and rough are also really nice. Nice round big people are kind to others. Young and rough people will most certainly be red. If someone has a kind disposition and looks rather round and rough then you'll find that they're actually quite young. Anyone feeling cold is bound to act nice. Young people who are both nice and cold also tend to be rough.

Stated in the text: "Fred is cold."

right

The dog is young.

Correct answer: true · Model said: true (56.9% sure)

The problem and its proof

The dog is kind. The dog is red. The dog is rough. The dog is round. The dog is young. If the dog is red and the dog is not rough then the dog is kind.

Stated in the text: "The dog is young."

right

The cow is rough.

Correct answer: false · Model said: false (68.8% sure)

The problem and its proof

The cat eats the cow. The cat is green. The cat is kind. The cat is nice. The cat is rough. The cat needs the cow. The cat sees the cow. The cow is green. The cow is nice. The cow is young. The cow needs the cat. The cow sees the cat. If the cat is kind then the cat is nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

59.7%
right on 179 of 300 problems wrong; chance 50% (dashed line)
wrong

The squirrel does not see the bear.

Correct answer: true · Model said: false (82.9% sure)

The problem and its proof

The bear is blue. The bear is kind. The bear sees the lion. The lion chases the squirrel. The lion is cold. The lion sees the bear. The squirrel is blue. The tiger is cold. The tiger is kind. The tiger is round. The tiger sees the bear. The tiger sees the lion. If something is blue and not round then it chases the lion. If the squirrel visits the lion then the squirrel sees the bear.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Erin is not nice.

Correct answer: false · Model said: false (82.0% sure)

The problem and its proof

Erin is green. If something is nice then it is kind. All nice, kind things are rough. If Erin is green then Erin is nice.

  1. Erin is green.If Erin is green then Erin is nice.Erin is nice.
right

Fred is not rough.

Correct answer: false · Model said: false (76.8% sure)

paraphrased
The problem and its proof

Alan can be rough and cold, but he is also green, an avid gardener. Bob may be round, but he is also kind. Charlie is a kind person and he is also often cold. Fred, who is relatively young, is also pretty big and tends to be cold. A kind person who is down in the dumps and blue tends to have a rough side. Most big and cold people are also rough. People who are young are also blue. A kind person will certainly be young.

  1. Fred is big. + Fred is cold.If someone is big and cold then they are rough.Fred is rough.
wrong

Charlie is nice.

Correct answer: true · Model said: false (59.0% sure)

paraphrased
The problem and its proof

After Charlie got wet in the rain, he feels cold. He also looks green but big. Eric may be round, but he is also kind. Fred is green and cold too. Harry seems to be round. Red people are almost always rough. All those who are nice and big are red. You must be aware of big, red, young people being rough. If a person happens to be big kind and red at the same time they are a young person. A person who is kind and rough and blue is young. People that are green and big while also being cold are always nice.

  1. Charlie is green. + Charlie is big. + Charlie is cold.If someone is green and big and cold then they are nice.Charlie is nice.

Depth 2, closed-world task

51.3%
right on 154 of 300 problems wrong; chance 50% (dashed line)
right

The squirrel is not young.

Correct answer: false · Model said: false (51.2% sure)

The problem and its proof

The squirrel is red. If the squirrel is red then the squirrel is cold. All kind, green people are cold. Young, green people are kind. If someone is red then they are green. All red, green people are cold. All cold, green people are young. If the squirrel is kind and the squirrel is red then the squirrel is cold. All kind, red people are young.

  1. The squirrel is red.If the squirrel is red then the squirrel is cold.The squirrel is cold.
  2. + The squirrel is red.If someone is red then they are green.The squirrel is green.
  3. All cold, green people are young.The squirrel is young.
wrong

Fiona is not cold.

Correct answer: false · Model said: true (83.4% sure)

The problem and its proof

Anne is big. Anne is cold. Anne is furry. Anne is kind. Anne is round. Anne is smart. Anne is young. Fiona is furry. Fiona is kind. Fiona is round. Fiona is smart. Fiona is young. All big, kind people are cold. If Fiona is round and Fiona is kind then Fiona is big.

  1. Fiona is round. + Fiona is kind.If Fiona is round and Fiona is kind then Fiona is big.Fiona is big.
  2. + Fiona is kind.All big, kind people are cold.Fiona is cold.
wrong

The rabbit does not eat the mouse.

Correct answer: true · Model said: false (62.6% sure)

The problem and its proof

The dog eats the mouse. The dog is blue. The mouse chases the squirrel. The mouse needs the rabbit. The rabbit eats the squirrel. The squirrel is cold. The squirrel is kind. If someone needs the squirrel and they eat the dog then the dog is green. If the rabbit eats the squirrel and the squirrel is blue then the rabbit is cold. If someone chases the mouse then they chase the dog. If the squirrel needs the rabbit and the squirrel eats the dog then the squirrel chases the rabbit. If the squirrel does not need the mouse then the mouse needs the squirrel. If someone is green and they do not chase the rabbit then the rabbit eats the mouse.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The dog sees the tiger.

Correct answer: true · Model said: true (92.4% sure)

The problem and its proof

The dog chases the tiger. The dog is big. The dog is nice. The dog is young. The dog likes the tiger. The mouse is nice. The mouse sees the tiger. The tiger likes the dog. The tiger likes the mouse. The tiger sees the mouse. If the mouse does not chase the dog then the dog likes the mouse. If the dog likes the mouse then the dog sees the tiger.

  1. If the mouse does not chase the dog then the dog likes the mouse.The dog likes the mouse.
  2. If the dog likes the mouse then the dog sees the tiger.The dog sees the tiger.

Depth 3, closed-world task

53.0%
right on 159 of 300 problems wrong; chance 50% (dashed line)
right

The dog eats the dog.

Correct answer: false · Model said: false (65.1% sure)

The problem and its proof

The bear eats the dog. The bear is round. The bear likes the dog. The bear likes the mouse. The bear sees the dog. The dog eats the mouse. The mouse eats the dog. If someone is round then they like the mouse. If someone likes the mouse then the mouse is round. If the dog does not eat the mouse then the mouse is blue. If someone is blue and they eat the mouse then the mouse eats the bear. If someone likes the mouse and they are not red then they see the mouse. If someone eats the bear then the bear likes the mouse. If someone sees the mouse and they do not eat the bear then they eat the dog. If someone eats the dog and they eat the bear then they like the dog.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The bald eagle likes the squirrel.

Correct answer: false · Model said: true (91.1% sure)

The problem and its proof

The bald eagle is cold. The bald eagle likes the rabbit. The bald eagle sees the rabbit. The bald eagle sees the squirrel. The mouse sees the rabbit. The rabbit is red. The rabbit is round. The rabbit is young. The rabbit likes the bald eagle. The rabbit likes the squirrel. The rabbit sees the mouse. The rabbit sees the squirrel. The rabbit visits the mouse. The squirrel is young. The squirrel sees the mouse. The squirrel visits the bald eagle. If someone visits the squirrel and the squirrel likes the bald eagle then the squirrel is round. All round people are cold. If someone likes the mouse then the mouse is round. If someone is young then they like the mouse. If someone visits the rabbit then they like the squirrel. All round people are young. All cold people are green. If someone likes the mouse then they like the squirrel. If someone sees the mouse and they visit the squirrel then the squirrel likes the rabbit.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The bear is kind.

Correct answer: true · Model said: false (59.3% sure)

The problem and its proof

The bear eats the squirrel. The bear needs the rabbit. The lion eats the bear. The lion sees the squirrel. The rabbit eats the bear. The rabbit is big. The rabbit needs the lion. The rabbit sees the lion. The squirrel eats the rabbit. The squirrel is cold. The squirrel is kind. The squirrel needs the bear. If someone is kind and they see the bear then the bear is green. If someone eats the squirrel then they are green. If someone needs the rabbit then they eat the bear. If someone needs the rabbit and they are green then the rabbit eats the squirrel. If the squirrel does not see the bear and the bear does not need the squirrel then the squirrel sees the rabbit. If someone is cold then they are kind. If the bear needs the rabbit and the rabbit needs the lion then the bear sees the squirrel. If someone is green then they are cold.

  1. The bear eats the squirrel.If someone eats the squirrel then they are green.The bear is green.
  2. If someone is green then they are cold.The bear is cold.
  3. If someone is cold then they are kind.The bear is kind.
right

The cat eats the mouse.

Correct answer: true · Model said: true (94.9% sure)

The problem and its proof

The cat is big. The cat is rough. The cat is round. The cat needs the mouse. The cat needs the tiger. The mouse eats the rabbit. The mouse is kind. The mouse is round. The mouse visits the cat. The rabbit eats the tiger. The rabbit is kind. The rabbit needs the mouse. The tiger eats the cat. The tiger is rough. The tiger is young. The tiger visits the cat. If someone needs the tiger then they eat the rabbit. If someone is big then they need the tiger. If someone is kind then they eat the mouse. If someone needs the cat then the cat visits the mouse. If the mouse does not visit the cat then the mouse eats the rabbit. If someone eats the rabbit then they are kind. If someone eats the rabbit and they visit the rabbit then they are round. If the cat is rough then the cat needs the rabbit.

  1. The cat needs the tiger.If someone needs the tiger then they eat the rabbit.The cat eats the rabbit.
  2. If someone eats the rabbit then they are kind.The cat is kind.
  3. If someone is kind then they eat the mouse.The cat eats the mouse.

Depth 4, closed-world task

51.0%
right on 153 of 300 problems wrong; chance 50% (dashed line)
wrong

Gary is furry.

Correct answer: false · Model said: true (55.0% sure)

The problem and its proof

Anne is green. Erin is red. Erin is round. Gary is red. Harry is green. Harry is round. Harry is smart. If something is green then it is blue. All round things are blue. All red, blue things are green. Blue, red things are smart. If something is round and not red then it is furry. All blue things are furry.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Bob is young.

Correct answer: true · Model said: true (62.2% sure)

paraphrased
The problem and its proof

Even though Bob is big and rough, he is kind. Bob is green with envy. Fred is green and cold too. That guy Gary sure is nice. Harry is shaped big and round like a balloon. His red color shows how kind he is. Round people who feel blue and are green in color are often young in age. All young and kind people that feel blue are described as red. People turn green when they are feeling blue. If someone is green and naive they may also have red, rough skin. It is a safe guess then that they are also round. Even though a person is big and rough when they are cold they turn red. Big people with rough, green skin are cold because of it.

  1. Bob is green. + Bob is big. + Bob is rough.If someone is green and big and rough then they are cold.Bob is cold.
  2. + Bob is rough. + Bob is big.If someone is rough and big and cold then they are red.Bob is red.
  3. + Bob is rough. + Bob is green.If someone is red and rough and green then they are round.Bob is round.
  4. + Bob is green.If someone is green then they are blue.Bob is blue.
  5. + Bob is green.If someone is round and blue and green then they are young.Bob is young.
right

Bob is not smart.

Correct answer: true · Model said: true (50.7% sure)

The problem and its proof

Bob is white. Charlie is cold. Dave is furry. Dave is nice. Dave is quiet. Dave is round. Harry is nice. Nice people are furry. If someone is smart then they are cold. If Dave is nice then Dave is white. If someone is furry then they are nice. If someone is furry then they are quiet. If someone is quiet then they are smart.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Gary is kind.

Correct answer: false · Model said: true (62.4% sure)

The problem and its proof

Charlie is rough. Dave is quiet. Gary is rough. Harry is red. If something is red then it is kind. Quiet things are kind. If Gary is kind then Gary is furry. Furry, nice things are quiet. If something is nice then it is red. If something is red then it is nice. All red things are rough. Red things are young.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 5, closed-world task

51.7%
right on 155 of 300 problems wrong; chance 50% (dashed line)
right

The rabbit needs the squirrel.

Correct answer: true · Model said: true (72.9% sure)

The problem and its proof

The cow is green. The cow needs the lion. The cow needs the rabbit. The cow visits the lion. The cow visits the rabbit. The lion is round. The lion sees the squirrel. The lion visits the cow. The rabbit is nice. The squirrel is kind. The squirrel visits the cow. If someone visits the lion then they are kind. If the squirrel is nice then the squirrel is big. If someone is round then they see the cow. If the cow is kind and the cow needs the rabbit then the rabbit is green. If someone is kind and not round then they need the squirrel. If the rabbit is nice and the rabbit is green then the rabbit visits the lion. If someone needs the lion and they are not kind then the lion sees the cow.

  1. The cow visits the lion.If someone visits the lion then they are kind.The cow is kind.
  2. + The cow needs the rabbit.If the cow is kind and the cow needs the rabbit then the rabbit is green.The rabbit is green.
  3. + The rabbit is nice.If the rabbit is nice and the rabbit is green then the rabbit visits the lion.The rabbit visits the lion.
  4. If someone visits the lion then they are kind.The rabbit is kind.
  5. If someone is kind and not round then they need the squirrel.The rabbit needs the squirrel.
right

The cat chases the dog.

Correct answer: true · Model said: true (93.2% sure)

The problem and its proof

The bald eagle chases the dog. The bald eagle is kind. The bald eagle is red. The bald eagle sees the tiger. The cat chases the bald eagle. The cat is kind. The cat likes the dog. The cat sees the bald eagle. The cat sees the tiger. The dog is big. The dog likes the tiger. The dog sees the cat. The tiger is cold. The tiger is red. The tiger likes the bald eagle. The tiger sees the bald eagle. If something likes the tiger then it chases the bald eagle. If the tiger chases the dog then the dog chases the cat. If something is round then it chases the dog. If something sees the cat and it chases the bald eagle then it likes the cat. If something is red then it is round. If the dog sees the bald eagle then the bald eagle sees the dog. If something chases the cat then the cat is round.

  1. The tiger is red.If something is red then it is round.The tiger is round.
  2. If something is round then it chases the dog.The tiger chases the dog.
  3. If the tiger chases the dog then the dog chases the cat.The dog chases the cat.
  4. If something chases the cat then the cat is round.The cat is round.
  5. If something is round then it chases the dog.The cat chases the dog.
wrong

Fiona is red.

Correct answer: true · Model said: false (54.9% sure)

The problem and its proof

Anne is blue. Anne is red. Anne is smart. Dave is big. Dave is cold. Dave is nice. Fiona is smart. Harry is blue. Harry is cold. Harry is nice. Harry is red. Harry is smart. Kind things are nice. All cold, nice things are blue. Cold things are kind. Smart things are cold. All blue, smart things are red.

  1. Fiona is smart.Smart things are cold.Fiona is cold.
  2. Cold things are kind.Fiona is kind.
  3. Kind things are nice.Fiona is nice.
  4. All cold, nice things are blue.Fiona is blue.
  5. + Fiona is smart.All blue, smart things are red.Fiona is red.
wrong

Erin is not cold.

Correct answer: false · Model said: true (72.2% sure)

The problem and its proof

Bob is green. Erin is rough. Gary is cold. Gary is smart. Harry is green. Harry is smart. Harry is white. If something is white then it is green. All rough things are quiet. If something is green and smart then it is cold. All quiet things are blue. All quiet things are white. All white things are rough. Quiet, green things are smart.

  1. Erin is rough.All rough things are quiet.Erin is quiet.
  2. All quiet things are white.Erin is white.
  3. If something is white then it is green.Erin is green.
  4. Quiet, green things are smart.Erin is smart.
  5. If something is green and smart then it is cold.Erin is cold.

Does a bigger Kev help?

The Kev sizes are built the same way on bigger base models. Every size we have scored is listed, smallest first; a size that is not scored yet appears here when it is.

SizeParametersOpen worldClosed worldDepth 0, open worldDepth 5, open worldMemory peak
Kev-0.8B0.8B base53.6%55.8%70.3%50.9%3.2 GB
Kev-4B4B base53.6%58.8%87.7%36.3%17.0 GB
Kev-9B9B base58.6%63.8%89.3%41.2%18.0 GB

Kev-9B against Kev-0.8B: +5.0 points on the open-world task and +8.0 points on the closed-world task overall. By depth on the open-world task the change is +19.0 points at depth 0 and −9.7 points at depth 5: the bigger model is better on easy problems and worse on the deepest ones. The harness compares each Kev with Jev on the same items, not the sizes with each other, so read the two sizes' separate intervals with care.

Where each one is better

Open world, Jev vs Kev-9B. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers, true answers and unknown answers. Kev-9B is not clearly ahead on any depth or answer class.

Closed world, Jev vs Kev-9B. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. Kev-9B is not clearly ahead on any depth or answer class.

Open world, Jev vs Kev-4B. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. Kev-4B is clearly ahead on unknown answers (+4.6 points).

Closed world, Jev vs Kev-4B. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. Kev-4B is not clearly ahead on any depth or answer class.

Open world, Jev vs Kev-0.8B. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, true answers and unknown answers. Kev-0.8B is not clearly ahead on any depth or answer class. Too close to call: false answers.

Closed world, Jev vs Kev-0.8B. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. Kev-0.8B is not clearly ahead on any depth or answer class.

A slice counts as clearly ahead when its paired 95% interval excludes zero.

True, false and unknown, open world

By correct answer, Open world
Correct answerItemsJevKev-9BKev-4BKev-0.8BBest constant
false60588.3% 85.5–90.948.6% 44.6–52.635.9% 31.9–39.589.6% 87.1–91.7one answer
true60588.8% 86.1–91.159.2% 55.4–63.146.6% 42.8–50.658.2% 54.0–62.0one answer
unknown59074.2% 70.7–77.668.1% 64.6–71.778.8% 75.4–81.911.9% 9.3–14.6one answer

True, false and unknown, closed world

By correct answer, Closed world
Correct answerItemsJevKev-9BKev-4BKev-0.8BBest constant
false90088.9% 86.9–90.962.9% 59.9–65.975.3% 72.8–78.266.1% 63.0–69.3one answer
true90089.7% 87.7–91.664.7% 61.7–67.642.2% 39.2–45.445.4% 42.1–48.6one answer

Paraphrased vs templated rules, open world

By wording, Open world
WordingItemsJevKev-9BKev-4BKev-0.8BBest constant
templated1,29886.1% 84.2–87.959.6% 57.0–62.255.9% 53.2–58.653.8% 51.2–56.434.5%
paraphrased (NatLang)50278.1% 74.5–82.155.8% 51.8–60.047.6% 43.4–52.053.0% 48.8–57.235.9%

Paraphrased vs templated rules, closed world

By wording, Closed world
WordingItemsJevKev-9BKev-4BKev-0.8BBest constant
templated1,30694.0% 92.8–95.364.8% 62.0–67.560.2% 57.7–62.756.6% 53.8–59.350.2%
paraphrased (NatLang)49476.7% 73.1–80.461.1% 56.7–65.455.1% 50.4–59.553.6% 49.6–58.150.6%

Answer by answer: Jev, Kev-9B, Kev-4B and Kev-0.8B on the same problems

Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
Kev-9B, open-world task
Correct ↓ · said →truefalseunknown
true35837210
false74294237
unknown13157402
Kev-4B, open-world task
Correct ↓ · said →truefalseunknown
true28212311
false45217343
unknown8936465
Kev-0.8B, open-world task
Correct ↓ · said →truefalseunknown
true35220944
false595424
unknown15037070

Jev said true on 37% of problems; the correct answer is true on 34%. Kev-9B said unknown on 47% of problems; the correct answer is unknown on 33%. Kev-4B said unknown on 62% of problems; the correct answer is unknown on 33%. Kev-0.8B said false on 62% of problems; the correct answer is false on 34%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
Kev-9B, closed-world task
Correct ↓ · said →truefalse
true582318
false334566
Kev-4B, closed-world task
Correct ↓ · said →truefalse
true380520
false222678
Kev-0.8B, closed-world task
Correct ↓ · said →truefalse
true409491
false305595

Jev said true on 50% of problems; the correct answer is true on 50%. Kev-9B said true on 51% of problems; the correct answer is true on 50%. Kev-4B said false on 67% of problems; the correct answer is false on 50%. Kev-0.8B said false on 60% of problems; the correct answer is false on 50%.

Where Kev-9B, Kev-4B and Kev-0.8B err

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Kev-9B, open-world task

Kev-9B, open-world task
Correct ↓ · said →truefalseunknownRecall
true3583721059.2%
false7429423748.6%
unknown1315740268.1%

Kev-9B answered false 21.6%, true 31.3%, unknown 47.2% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

Kev-9B, closed-world task

Kev-9B, closed-world task
Correct ↓ · said →truefalseRecall
true58231864.7%
false33456662.9%

Kev-9B answered false 49.1%, true 50.9% of the time; the correct answers are false 50.0%, true 50.0%.

Kev-9B at depth 0 and depth 5

Kev-9B, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9703
false2944
unknown131077
Kev-9B, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true36659
false172361
unknown22560

Open-world task. At depth 0, Kev-9B was right on 89% of problems (the green diagonal); at depth 5, on 41%. The answer whose share moved most is unknown: 28% of its answers at depth 0, 62% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
Kev-9B, closed-world task, depth 0
Correct ↓ · said →truefalse
true12624
false7143
Kev-9B, closed-world task, depth 5
Correct ↓ · said →truefalse
true7773
false9852

Closed-world task. At depth 0, Kev-9B was right on 90% of problems (the green diagonal); at depth 5, on 43%. The answer whose share moved most is true: 44% of its answers at depth 0, 58% at depth 5, against 50% and 50% of the correct answers.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Kev-4B, open-world task

Kev-4B, open-world task
Correct ↓ · said →truefalseunknownRecall
true2821231146.6%
false4521734335.9%
unknown893646578.8%

Kev-4B answered false 14.7%, true 23.1%, unknown 62.2% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

Kev-4B, closed-world task

Kev-4B, closed-world task
Correct ↓ · said →truefalseRecall
true38052042.2%
false22267875.3%

Kev-4B answered false 66.6%, true 33.4% of the time; the correct answers are false 50.0%, true 50.0%.

Kev-4B at depth 0 and depth 5

Kev-4B, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9604
false18217
unknown10585
Kev-4B, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true22277
false111278
unknown12471

Open-world task. At depth 0, Kev-4B was right on 88% of problems (the green diagonal); at depth 5, on 36%. The answer whose share moved most is unknown: 35% of its answers at depth 0, 78% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
Kev-4B, closed-world task, depth 0
Correct ↓ · said →truefalse
true11634
false9141
Kev-4B, closed-world task, depth 5
Correct ↓ · said →truefalse
true42108
false6288

Closed-world task. At depth 0, Kev-4B was right on 86% of problems (the green diagonal); at depth 5, on 43%. The answer whose share moved most is true: 42% of its answers at depth 0, 35% at depth 5, against 50% and 50% of the correct answers.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Kev-0.8B, open-world task

Kev-0.8B, open-world task
Correct ↓ · said →truefalseunknownRecall
true3522094458.2%
false59542489.6%
unknown1503707011.9%

Kev-0.8B answered false 62.3%, true 31.2%, unknown 6.6% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

Kev-0.8B, closed-world task

Kev-0.8B, closed-world task
Correct ↓ · said →truefalseRecall
true40949145.4%
false30559566.1%

Kev-0.8B answered false 60.3%, true 39.7% of the time; the correct answers are false 50.0%, true 50.0%.

Kev-0.8B at depth 0 and depth 5

Kev-0.8B, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9163
false8920
unknown195328
Kev-0.8B, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true53426
false10901
unknown20634

Open-world task. At depth 0, Kev-0.8B was right on 70% of problems (the green diagonal); at depth 5, on 51%. The answer whose share moved most is false: 50% of its answers at depth 0, 67% at depth 5, against 33% and 35% of the correct answers.

The closed-world task at depth 0 and depth 5
Kev-0.8B, closed-world task, depth 0
Correct ↓ · said →truefalse
true8664
false32118
Kev-0.8B, closed-world task, depth 5
Correct ↓ · said →truefalse
true6189
false5694

Closed-world task. At depth 0, Kev-0.8B was right on 68% of problems (the green diagonal); at depth 5, on 52%. The answer whose share moved most is true: 39% of its answers at depth 0, 39% at depth 5, against 50% and 50% of the correct answers.

Kev-9B, Kev-4B and Kev-0.8B across every breakdown

For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownKev-9BKev-4BKev-0.8B
Proof depthdepth 4 38.6% · depth 0 89.3% (50.7 points)depth 5 36.3% · depth 0 87.7% (51.3 points)depth 2 46.9% · depth 0 70.3% (23.5 points)
Gold answerfalse 48.6% · unknown 68.1% (19.5 points)false 35.9% · unknown 78.8% (42.9 points)unknown 11.9% · false 89.6% (77.7 points)
Theory kindrelation 57.5% · attribute 59.3% (1.8 points)relation 52.4% · attribute 54.4% (2.0 points)relation 53.2% · attribute 53.8% (0.7 points)
Negation in the theorywithout negation 56.8% · with negation 60.4% (3.6 points)without negation 49.9% · with negation 57.3% (7.4 points)without negation 50.6% · with negation 56.6% (6.0 points)
Negated statementnegated statement 57.4% · plain statement 59.7% (2.4 points)negated statement 53.5% · plain statement 53.6% (0.1 points)plain statement 48.5% · negated statement 58.6% (10.1 points)
Question strategyinv-proof 48.6% · random 78.0% (29.4 points)inv-proof 35.9% · random 88.0% (52.1 points)inv-rconc 0.8% · inv-proof 89.6% (88.8 points)
Paraphrased rulesparaphrased (NatLang) 55.8% · templated 59.6% (3.9 points)paraphrased (NatLang) 47.6% · templated 55.9% (8.2 points)paraphrased (NatLang) 53.0% · templated 53.8% (0.8 points)
Theory's deepest proof4 42.6% · 0 87.3% (44.7 points)5 38.0% · 0 85.9% (47.9 points)0 42.3% · 5 58.0% (15.7 points)
Theory length110+ 47.9% · 0-49 77.7% (29.8 points)110+ 41.7% · 0-49 75.7% (34.0 points)110+ 52.6% · 0-49 56.3% (3.7 points)
Number of rules3-5 51.6% · 0-2 78.9% (27.3 points)3-5 45.9% · 0-2 78.9% (33.0 points)8+ 49.8% · 0-2 58.8% (9.0 points)
Number of facts13+ 46.5% · 0-3 71.9% (25.3 points)13+ 40.3% · 0-3 69.8% (29.5 points)0-3 52.4% · 13+ 56.6% (4.2 points)
Proof size5+ 31.8% · 2 84.1% (52.3 points)5+ 20.3% · 0-1 81.4% (61.1 points)0-1 32.0% · 2 77.6% (45.5 points)

Exactly what Kev-9B, Kev-4B and Kev-0.8B were asked

One open-world problem at depth 0, as each model received it. The correct answer is unknown.

The bear is green. Green things are not young.

Statement: The bear is not big.
Jevright answered unknown (100.0%); correct answer unknown

POST https://api.typesafe.ai/v1/systemone

Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "jev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "jev-1.13.0",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 1,
      "probabilities": {
        "false": 0,
        "true": 0,
        "unknown": 1
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 460,
    "output_tokens": 38
  }
}
Kev-9Bright answered unknown (98.7%); correct answer unknown

POST http://127.0.0.1:8009/v1/systemone

Kev's own server running Kev-9B on our laptop, answering Jev's System One format.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.9%, true 0.4%, unknown 98.7%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "kev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "kev-latest",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 0.9803,
      "probabilities": {
        "false": 0.0093,
        "true": 0.0038,
        "unknown": 0.9869
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 162,
    "output_tokens": 63
  }
}
Kev-4Bright answered unknown (99.4%); correct answer unknown

POST http://127.0.0.1:8009/v1/systemone

Kev's own server running Kev-4B on our laptop, answering Jev's System One format.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.3%, true 0.3%, unknown 99.4%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "kev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "kev-latest",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 0.9909,
      "probabilities": {
        "false": 0.0034,
        "true": 0.0027,
        "unknown": 0.9939
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 162,
    "output_tokens": 63
  }
}
Kev-0.8Bright answered unknown (42.8%); correct answer unknown

POST http://127.0.0.1:8009/v1/systemone

Kev's own server running Kev-0.8B on our laptop, answering Jev's System One format.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 40.7%, true 16.5%, unknown 42.8%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "kev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "kev-latest",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 0.1413,
      "probabilities": {
        "false": 0.4072,
        "true": 0.1653,
        "unknown": 0.4275
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 162,
    "output_tokens": 63
  }
}

Consistency and calibration

Kev-9B

Kev-9B's rerun has not been scored yet.

Calibration on the open world task: Kev-9B's probability for its own answer averaged 74.3% against 58.6% accuracy; expected calibration error 15.8 points.

Calibration on the closed world task: Kev-9B's probability for its own answer averaged 77.3% against 63.8% accuracy; expected calibration error 13.5 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.72 on the open world task against Jev's 0.85; 0.63 on the closed world task against Jev's 0.86.

Kev-4B

Asked every open world question a second time, Kev-4B gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 53.6% the first time and 53.6% the second.

Asked every closed world question a second time, Kev-4B gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 58.8% the first time and 58.8% the second.

Calibration on the open world task: Kev-4B's probability for its own answer averaged 69.9% against 53.6% accuracy; expected calibration error 16.4 points.

Calibration on the closed world task: Kev-4B's probability for its own answer averaged 66.4% against 58.8% accuracy; expected calibration error 7.6 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.71 on the open world task against Jev's 0.85; 0.59 on the closed world task against Jev's 0.86.

Kev-0.8B

Asked every open world question a second time, Kev-0.8B gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 53.6% the first time and 53.6% the second.

Asked every closed world question a second time, Kev-0.8B gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 55.8% the first time and 55.8% the second.

Calibration on the open world task: Kev-0.8B's probability for its own answer averaged 65.0% against 53.6% accuracy; expected calibration error 11.4 points.

Calibration on the closed world task: Kev-0.8B's probability for its own answer averaged 72.8% against 55.8% accuracy; expected calibration error 17.0 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.66 on the open world task against Jev's 0.85; 0.57 on the closed world task against Jev's 0.86.

Size, hardware and speed

ModelKindParametersRan onMedian time per decisionMemory peak
Jevhosted decision modelundisclosedvendor's servers189 msnot applicable (hosted)
Kev-9Bopen decision model9B base (Qwen3.5-9B-Base) + rank-16 LoRA adapter and pointer headour laptop (Apple M1 Max, 32 GB)not yet timed18.0 GB
Kev-4Bopen decision model4B base (Qwen3.5-4B-Base) + rank-16 LoRA adapter (33.8M trainable) and pointer headour laptop (Apple M1 Max, 32 GB)897 ms17.0 GB
Kev-0.8Bopen decision model0.8B base (Qwen3.5-0.8B-Base) + rank-16 LoRA adapter and pointer headour laptop (Apple M1 Max, 32 GB)123 ms3.2 GB

Times are open-world, one request at a time. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.

Kev-9B

Parameters
9B base (Qwen3.5-9B-Base) + rank-16 LoRA adapter and pointer head
Architecture
as kev-0.8b
Weights
19.31 GB base (Qwen/Qwen3.5-9B-Base@68c46c4b) + 0.18 GB adapter and head (jaredpalmer/kev-9b@b5d8c18e)
Precision
bfloat16 (MLX), fp32 pointer head
Where it ran
this machine, Kev server c9c1f855 on MLX
Maker's stated hardware
"32 GB Mac, L40S, H100" (Kev README)
Memory measured
17.0 GB in use, 18.0 GB peak
Source
huggingface.co/jaredpalmer/kev-9b

Kev-4B

Parameters
4B base (Qwen3.5-4B-Base) + rank-16 LoRA adapter (33.8M trainable) and pointer head
Architecture
as kev-0.8b
Weights
9.32 GB base (Qwen/Qwen3.5-4B-Base@1001bb4d) + 0.14 GB adapter and head (jaredpalmer/kev-4b@139fdd94)
Precision
bfloat16 (MLX), fp32 pointer head
Where it ran
this machine, Kev server c9c1f855 on MLX
Maker's stated hardware
"32 GB Mac, L40S, H100" (Kev README)
Memory measured
8.6 GB in use, 17.0 GB peak
Time per decision, open world
median 897 ms, 90th percentile 1,100 ms, on a laptop
Time per decision, closed world
median 802 ms, 90th percentile 1,044 ms, on a laptop
Source
huggingface.co/jaredpalmer/kev-4b

Kev-0.8B

Parameters
0.8B base (Qwen3.5-0.8B-Base) + rank-16 LoRA adapter and pointer head
Architecture
Qwen3.5 hybrid (Gated DeltaNet + attention) base, frozen; LoRA r16; pointer head
Weights
1.75 GB base (Qwen/Qwen3.5-0.8B-Base@dc7cdfe2) + 0.05 GB adapter and head (jaredpalmer/kev-0.8b@54f4f877)
Precision
bfloat16 (MLX), fp32 pointer head
Where it ran
this machine, Kev server c9c1f855 on MLX
Maker's stated hardware
"Any Apple Silicon Mac, L4" (Kev README)
Memory measured
2.1 GB in use, 3.2 GB peak
Time per decision, open world
median 123 ms, 90th percentile 152 ms, on a laptop
Time per decision, closed world
median 121 ms, 90th percentile 145 ms, on a laptop
Source
github.com/jaredpalmer/kev

The models

Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

Kev-9B Kev-9B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 9-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

Kev-4B Kev-4B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 4-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

Kev-0.8B Kev-0.8B is an open decision model made by Jared Palmer: a pointer head and a rank-16 LoRA adapter on a frozen 0.8-billion-parameter Qwen3.5 base model. We ran it on our own laptop through the Kev server on MLX, in bfloat16.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.