Skip to main content
HardDecisions

Model · hosted decision model

Jev accuracy on multi-hop reasoning

Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

Open world83.8%82.2 to 85.5 (95%) on 1,800 problems; place 1 of 10
Closed world89.3%87.8 to 90.7 (95%) on 1,800 problems; place 1 of 10
Depth 0 to depth 5−16.7points on the open-world task, 97.7% to 81.0%

Jev accuracy by proof depth

Each point is about 300 problems that need that many inference steps.

Open world: true, false or unknown
  • Jev
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Open world: true, false or unknown

Jev by depth, Open world
Proof depthItemsJevBest constant
depth 030097.7% 96.0–99.333.3%
depth 130287.7% 83.8–91.433.4%
depth 230384.8% 80.2–88.433.3%
depth 330379.9% 75.2–84.533.3%
depth 430371.9% 66.7–77.233.3%
depth 528981.0% 76.5–85.534.9%

Closed world: true or false

Jev by depth, Closed world
Proof depthItemsJevBest constant
depth 030099.3% 98.3–100.050.0%
depth 130093.7% 91.0–96.350.0%
depth 230092.0% 88.7–94.750.0%
depth 330085.0% 81.3–89.350.0%
depth 430076.3% 71.7–81.050.0%
depth 530089.3% 85.7–92.350.0%

What Jev got right and wrong, depth by depth

At each proof depth, Jev's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

97.7%
right on 293 of 300 problems wrong; chance 33% (dashed line)
right

Bob is not smart.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

Bob is not smart. Bob is white. Harry is green. Round people are white. White people are blue. If Bob is white and Bob is blue then Bob is round. Round, white people are not quiet. If someone is blue and not white then they are smart. If someone is white then they are not smart. If someone is smart and not quiet then they are green. All smart, round people are not green.

Stated in the text: "Bob is not smart."

right

Alan is nice.

Correct answer: unknown · Model said: unknown (100.0% sure)

paraphrased
The problem and its proof

Alan is a kind person and he is also often cold. Charlie is a kind person and he is also often cold. Fred is feeling cold -- he is also feeling quite rough, red, nice and round. That guy Harry sure is nice. When a person is nice and round and cold, they look blue. A blue colored person who is nice is a red person. A round person who is on the young side is also likely to treat people in a kind manner. Why is it that every round person that is blue is so young.

No proof: neither the statement nor its negation follows.

right

The cat does not like the dog.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

The cat does not like the dog. The dog likes the cat. The mouse does not chase the dog. If something visits the cat then the cat chases the dog.

Stated in the text: "The cat does not like the dog."

right

Anne is not big.

Correct answer: false · Model said: false (100.0% sure)

The problem and its proof

Anne is big. Anne is green. Bob is green. Charlie is blue. Charlie is nice. Dave is nice. Dave is quiet. Green people are blue. All blue people are big. All big, nice people are kind. Kind people are round. All round people are green. All quiet people are big. Blue people are quiet. Big, quiet people are kind. All blue, quiet people are kind.

Stated in the text: "Anne is big."

Depth 1, open-world task

87.7%
right on 265 of 302 problems wrong; chance 33% (dashed line)
right

Dave is not rough.

Correct answer: false · Model said: false (57.0% sure)

paraphrased
The problem and its proof

Alan seems to be round. Bob may be round, but he is also kind. That person with the super big round body wearing red and green clothes who is acting quite nice towards you while being cold and feeling blue is named Dave. Fred may be round, but he is also kind. If someone you know is big, cold, and young, you know they are also kind. A red hued and rough looking person is definitely a young person. Most round and cold people are often rough. A young aged and big blue person will definitely be cold.

The proof that the statement's negation follows:

  1. Dave is round. + Dave is cold.If someone is round and cold then they are rough.Dave is rough.
right

The bear is not blue.

Correct answer: unknown · Model said: unknown (99.0% sure)

The problem and its proof

The bear eats the dog. The bear is not rough. The dog does not need the bear. If the bear likes the dog then the bear is blue.

No proof: neither the statement nor its negation follows.

right

Harry is round.

Correct answer: true · Model said: true (62.0% sure)

paraphrased
The problem and its proof

Eric seems big and rough, but he's actually quite kind. Young Harry here is always rough but nice and yes he is big. Young, red people are usually quite blue too. Young people who are cold and blue are actually kind. A big, round person who is kind is also red. People with big smiles and round eyes will have red hair. Someone who is young at heart and age are very round. A person that is round and somewhat green while being nice tends to be red as well. If a person is both cold and round, that person is also someone who is red.

  1. Harry is young.If someone is young then they are round.Harry is round.
right

Harry is red.

Correct answer: true · Model said: true (94.0% sure)

paraphrased
The problem and its proof

Bob is green to the job, which is why he turned blue from being so cold. Green, big and red are qualities which all describe Charlie. Green, big and red are qualities which all describe Eric. Harry can be rough and cold, but he is also green, an avid gardener. People,though kind and rough are also really nice. People that are green and big while also being cold are always nice. A kind person will certainly be young. When you run into someone who is rough and green at the same time, they will also be red. A person who inherits genes that make them big and green will also express a blue color. If a person is blue and red and big, then that person is cold.

  1. Harry is rough. + Harry is green.If someone is rough and green then they are red.Harry is red.

Depth 2, open-world task

84.8%
right on 257 of 303 problems wrong; chance 33% (dashed line)
right

Dave is rough.

Correct answer: false · Model said: false (97.0% sure)

The problem and its proof

Charlie is nice. Charlie is young. Dave is cold. Cold people are furry. All nice people are cold. If someone is young then they are white. If someone is white and not cold then they are not young. If Charlie is young then Charlie is white. If someone is young and not cold then they are rough. All furry people are not rough. Young, blue people are rough.

The proof that the statement's negation follows:

  1. Dave is cold.Cold people are furry.Dave is furry.
  2. All furry people are not rough.Dave is not rough.
right

Dave is young.

Correct answer: true · Model said: true (55.0% sure)

paraphrased
The problem and its proof

Alan seems to be round. Bob may be round, but he is also kind. That person with the super big round body wearing red and green clothes who is acting quite nice towards you while being cold and feeling blue is named Dave. Fred may be round, but he is also kind. If someone you know is big, cold, and young, you know they are also kind. A red hued and rough looking person is definitely a young person. Most round and cold people are often rough. A young aged and big blue person will definitely be cold.

  1. Dave is round. + Dave is cold.If someone is round and cold then they are rough.Dave is rough.
  2. + Dave is red.If someone is red and rough then they are young.Dave is young.
wrong

Eric is rough.

Correct answer: true · Model said: unknown (88.0% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Eric is nice and kind even though people make fun of his red and green skin. Eric always feels cold. Fred is somehow red and kind and also green. A person that is red, rough and kind often feels cold. Round young people, red with loveliness, are very cold towards others. A young person with red hands is rough around the edges. Is a kind person if feeling cold and looking red then they are usually young. Green folks who are nice and rough are a round shape. A nice and green man or woman is also red in color.

  1. Eric is cold. + Eric is red. + Eric is kind.If someone is cold and red and kind then they are young.Eric is young.
  2. + Eric is red.If someone is red and young then they are rough.Eric is rough.
right

Fiona is quiet.

Correct answer: false · Model said: false (100.0% sure)

The problem and its proof

Bob is smart. Bob is white. Fiona is white. Nice, quiet things are white. All nice things are not quiet. If something is white then it is not furry. All white things are nice. Big things are not kind. If something is nice and not quiet then it is not big. If Fiona is white and Fiona is big then Fiona is smart. If something is kind then it is smart.

The proof that the statement's negation follows:

  1. Fiona is white.All white things are nice.Fiona is nice.
  2. All nice things are not quiet.Fiona is not quiet.

Depth 3, open-world task

79.9%
right on 242 of 303 problems wrong; chance 33% (dashed line)
wrong

The dog is big.

Correct answer: unknown · Model said: true (60.0% sure)

The problem and its proof

The dog visits the tiger. The tiger eats the dog. If the tiger visits the dog and the dog chases the tiger then the dog is big. If the tiger is cold then the tiger does not chase the dog. If something visits the tiger and the tiger eats the dog then it eats the tiger. If something chases the tiger then it visits the dog. If something is big and not kind then it does not visit the dog. If the tiger is kind and the tiger visits the dog then the dog does not chase the tiger. If something eats the dog then the dog chases the tiger. If something is kind and it eats the dog then it chases the tiger.

No proof: neither the statement nor its negation follows.

right

Gary is not green.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

Bob is not young. Dave is not green. Erin is nice. Gary is white. Round things are young. If something is white and young then it is not green. White things are round.

  1. Gary is white.White things are round.Gary is round.
  2. Round things are young.Gary is young.
  3. + Gary is white.If something is white and young then it is not green.Gary is not green.
right

Fred is not blue.

Correct answer: false · Model said: false (78.0% sure)

paraphrased
The problem and its proof

Bob is a kind person and he is also often cold. Eric is green and cold too. Fred is green because he is young. Fred is also very kind but he can be rough when people make fun of him. For being so cold, it's good Gary can remain nice. Young round people who are green are usually blue. Not a single young person is not also kind. Nice people with blue and rough skin are very young and cannot be trusted with responsibility. Most round and cold people are often rough. Most young kind people tend to be red too. Young and red people look round.

The proof that the statement's negation follows:

  1. Fred is young. + Fred is kind.If someone is young and kind then they are red.Fred is red.
  2. + Fred is young.If someone is young and red then they are round.Fred is round.
  3. + Fred is young. + Fred is green.If someone is round and young and green then they are blue.Fred is blue.
right

Dave is not big.

Correct answer: unknown · Model said: unknown (77.0% sure)

paraphrased
The problem and its proof

For being so cold, it's good Alan can remain nice. Dave is kind and nice and looks green. Eric is big and rough around the edges. Even though he is feeling ill and green, he is nice. Fred is nice and kind,though he's also rather cold. Young people who are cold to others and green with envy are actually nice. A person feeling blue because of their big size will always be cold. A person that is both nice and rough is someone who is also big. Someone being nice, even when they are cold without a doubt means that they are round about the middle. Most round and cold people are often rough.

No proof: neither the statement nor its negation follows.

Depth 4, open-world task

71.9%
right on 218 of 303 problems wrong; chance 33% (dashed line)
right

Anne is not rough.

Correct answer: true · Model said: true (85.0% sure)

The problem and its proof

Anne is blue. Anne is nice. Anne is white. Charlie is blue. Erin is blue. Erin is quiet. Fiona is not cold. Fiona is not kind. Fiona is nice. Fiona is quiet. All cold things are kind. All rough things are kind. Nice, cold things are blue. Nice things are quiet. All blue things are nice. Cold, blue things are nice. Quiet, blue things are cold. Rough, kind things are cold. If something is kind then it is not rough.

  1. Anne is nice.Nice things are quiet.Anne is quiet.
  2. + Anne is blue.Quiet, blue things are cold.Anne is cold.
  3. All cold things are kind.Anne is kind.
  4. If something is kind then it is not rough.Anne is not rough.
right

Charlie is not round.

Correct answer: false · Model said: false (97.0% sure)

The problem and its proof

Anne is quiet. Anne is not rough. Anne is round. Bob is not green. Bob is kind. Charlie is green. Dave is round. Quiet, rough people are cold. All rough, kind people are cold. All round people are kind. All green people are nice. Nice people are quiet. If Anne is round then Anne is kind. All nice people are rough. All cold people are round.

The proof that the statement's negation follows:

  1. Charlie is green.All green people are nice.Charlie is nice.
  2. Nice people are quiet.Charlie is quiet.
  3. All nice people are rough.Charlie is rough.
  4. Quiet, rough people are cold.Charlie is cold.
  5. All cold people are round.Charlie is round.
right

Bob is not green.

Correct answer: true · Model said: true (82.0% sure)

The problem and its proof

Anne is big. Anne is green. Anne is red. Anne is not young. Bob is nice. Dave is big. Dave is green. Dave is not nice. Dave is red. Gary is not young. All round people are big. If someone is red and green then they are big. Green people are not furry. All nice, red people are not green. If Bob is not nice then Bob is not green. If someone is nice then they are round. If someone is round and not nice then they are red. If someone is big and nice then they are red. If someone is red and not green then they are young.

  1. Bob is nice.If someone is nice then they are round.Bob is round.
  2. All round people are big.Bob is big.
  3. + Bob is nice.If someone is big and nice then they are red.Bob is red.
  4. + Bob is nice.All nice, red people are not green.Bob is not green.
wrong

Charlie is not cold.

Correct answer: false · Model said: unknown (81.0% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Charlie discovered that even though he was round, rough, and nice and green, he was actually big. Dave is a cold and round man who has red and green skin. The young person who is always feeling cold is named Harry. People who have green body paint and act kind to others are quite young. A very nice person who is also green is certainly kind. Big people with red hair are cold because they cannot find coats that fit. Most young kind people tend to be red too.

The proof that the statement's negation follows:

  1. Charlie is green. + Charlie is nice.If someone is green and nice then they are kind.Charlie is kind.
  2. + Charlie is green.If someone is kind and green then they are young.Charlie is young.
  3. If someone is young and kind then they are red.Charlie is red.
  4. + Charlie is big.If someone is red and big then they are cold.Charlie is cold.

Depth 5, open-world task

81.0%
right on 234 of 289 problems wrong; chance 33% (dashed line)
right

Bob is young.

Correct answer: true · Model said: true (87.0% sure)

The problem and its proof

Anne is blue. Anne is nice. Anne is quiet. Anne is rough. Anne is white. Bob is big. Charlie is rough. Erin is big. Erin is nice. Erin is young. Quiet, nice things are white. If something is rough then it is quiet. If Bob is white then Bob is young. If Anne is big and Anne is blue then Anne is rough. If Bob is rough and Bob is quiet then Bob is nice. Big things are rough. If Erin is nice and Erin is young then Erin is big.

  1. Bob is big.Big things are rough.Bob is rough.
  2. If something is rough then it is quiet.Bob is quiet.
  3. If Bob is rough and Bob is quiet then Bob is nice.Bob is nice.
  4. Quiet, nice things are white.Bob is white.
  5. If Bob is white then Bob is young.Bob is young.
right

Harry is furry.

Correct answer: unknown · Model said: unknown (65.0% sure)

The problem and its proof

Anne is blue. Anne is furry. Anne is kind. Anne is quiet. Anne is rough. Anne is round. Anne is young. Erin is blue. Erin is quiet. Fiona is kind. Fiona is rough. Fiona is round. Fiona is young. Harry is blue. Harry is round. If someone is rough and furry then they are kind. Furry people are rough. All quiet people are kind. If someone is kind and rough then they are round. If Harry is blue then Harry is young. If someone is quiet then they are furry. All rough people are quiet. All kind people are quiet.

No proof: neither the statement nor its negation follows.

wrong

The bear chases the cat.

Correct answer: false · Model said: unknown (94.0% sure)

The problem and its proof

The bear is nice. The bear is red. The bear does not need the mouse. The cat chases the bear. The cat chases the mouse. The lion eats the bear. The lion eats the cat. The lion needs the bear. The lion needs the cat. The mouse eats the lion. If someone eats the mouse then the mouse chases the bear. If the mouse chases the bear then the bear eats the cat. If someone needs the cat and the cat is red then they eat the bear. If someone chases the bear then they do not eat the bear. If someone is nice then they do not eat the bear. If someone eats the cat then they do not chase the cat. If someone chases the lion and they need the lion then they do not eat the mouse. If someone needs the lion then they eat the mouse. If someone chases the bear then the bear needs the lion.

The proof that the statement's negation follows:

  1. The cat chases the bear.If someone chases the bear then the bear needs the lion.The bear needs the lion.
  2. If someone needs the lion then they eat the mouse.The bear eats the mouse.
  3. If someone eats the mouse then the mouse chases the bear.The mouse chases the bear.
  4. If the mouse chases the bear then the bear eats the cat.The bear eats the cat.
  5. If someone eats the cat then they do not chase the cat.The bear does not chase the cat.
right

Gary is not big.

Correct answer: false · Model said: false (52.0% sure)

paraphrased
The problem and its proof

That guy Bob sure is nice. Dave is known for being big and rough, but few know he is also always red with cold. Rough and cold that is what they say about Blue Gary. Harry is somehow red and kind and also green. Kind people that are red and round are usually big. Not a single young person is not also kind. Nice people who are blue and round at the same time are always young. Big people with red hair are cold because they cannot find coats that fit. Most young kind people tend to be red too. If someone is cold and blue, you will find that they are also quite young. Young and red people look round.

The proof that the statement's negation follows:

  1. Gary is blue. + Gary is cold.If someone is blue and cold then they are young.Gary is young.
  2. If someone is young then they are kind.Gary is kind.
  3. If someone is young and kind then they are red.Gary is red.
  4. If someone is young and red then they are round.Gary is round.
  5. If someone is kind and round and red then they are big.Gary is big.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

99.3%
right on 298 of 300 problems wrong; chance 50% (dashed line)
right

Charlie is round.

Correct answer: false · Model said: false (100.0% sure)

paraphrased
The problem and its proof

A bit overweight, Bob is big and round. He's a nice guy who is often cold. That guy Charlie sure is nice. The young person who is always feeling cold is named Eric. Fred is still young, which explains why he is round and cold, however, most people still think Fred is nice. People who are very nice and easily flush red are often wearing green concealer. A rather big person is a red person. Rough people who are kind and green with envy are red with their toughened skin. A red, nice person will definitely be a blue person. Most big and cold people are also rough. Someone who is currently blue, big, and cold, is also kind.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Fiona is not cold.

Correct answer: false · Model said: false (100.0% sure)

The problem and its proof

Erin is big. Erin is cold. Erin is green. Erin is red. Fiona is cold. Fiona is green. Fiona is nice. If Fiona is round then Fiona is red. All green, red things are rough. If something is red and not nice then it is rough. If Erin is round and Erin is red then Erin is big. If Fiona is nice then Fiona is rough. Rough things are round. If something is big then it is cold. If something is round and not rough then it is cold.

Stated in the text: "Fiona is cold."

right

The lion is not young.

Correct answer: false · Model said: false (100.0% sure)

The problem and its proof

The lion is blue. The lion is cold. The lion is nice. The lion is rough. The lion is young. If someone is rough and blue then they are cold. If the lion is blue and the lion is rough then the lion is cold.

Stated in the text: "The lion is young."

right

The mouse likes the cow.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

The cow chases the mouse. The cow is green. The cow is rough. The cow is round. The cow likes the mouse. The cow needs the mouse. The mouse chases the cow. The mouse is green. The mouse likes the cow. The mouse needs the cow. If someone is green then they need the mouse. If the cow needs the mouse then the mouse needs the cow. If the mouse likes the cow and the mouse chases the cow then the mouse needs the cow.

Stated in the text: "The mouse likes the cow."

Depth 1, closed-world task

93.7%
right on 281 of 300 problems wrong; chance 50% (dashed line)
right

Charlie is kind.

Correct answer: true · Model said: true (85.0% sure)

paraphrased
The problem and its proof

Charlie, who is relatively young, is also pretty big and tends to be cold. Fred woke up one day to find he had become round. He was still rough, red, and green, but now he was also big. For being so cold, it's good Harry can remain nice. Even though young people are rough, they are still very red. You must be aware of big, red, young people being rough. Do not fear big cold people for they are often kind. Big people with rough, green skin are cold because of it. Is a kind person if feeling cold and looking red then they are usually young. An individual who is big, red and young is also going to be a nice individual. If a person is blue and red and big, then that person is cold.

  1. Charlie is cold. + Charlie is big.If someone is cold and big then they are kind.Charlie is kind.
right

Charlie is not white.

Correct answer: false · Model said: false (99.0% sure)

The problem and its proof

Anne is nice. Anne is young. Charlie is green. Charlie is nice. Erin is green. Erin is nice. Harry is white. Furry people are red. If someone is white then they are nice. If someone is red then they are green. If someone is young then they are furry. White, nice people are young. Furry people are round. All green, nice people are white. Green, young people are white.

  1. Charlie is green. + Charlie is nice.All green, nice people are white.Charlie is white.
right

The cow is not big.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

The bear eats the dog. The bear is rough. The bear likes the cow. The bear likes the dog. The bear needs the cow. The bear needs the tiger. The cow eats the bear. The dog eats the cow. The dog is round. The dog likes the bear. The dog likes the tiger. The dog needs the bear. The tiger likes the dog. The tiger needs the bear. The tiger needs the cow. The tiger needs the dog. If someone eats the dog and they are round then they are big. If someone needs the dog then the dog needs the tiger. If someone needs the bear then they eat the tiger. If someone eats the tiger then the tiger likes the dog. If someone is red and they need the cow then they need the dog. All red people are nice. If someone likes the dog and the dog needs the tiger then they are red. If someone needs the dog then they are round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Gary is cold.

Correct answer: true · Model said: true (90.0% sure)

paraphrased
The problem and its proof

Charlie is a round and rough around the edges, and he is also big. Eric may be round, but he is also kind. Even though Gary is big and rough, he is kind. Gary is green with envy. Harry is a big fellow, often blue and sad, but he is nice. A green person that is red and blue are known for their rough skin. A young, green person in my experience is round shaped and nice. A green person that is rough and cold is often also nice. Big people with rough, green skin are cold because of it. Young people who are both nice and cold also tend to be rough. If you know someone who is nice, green, and big, you know a young person.

  1. Gary is green. + Gary is big. + Gary is rough.If someone is green and big and rough then they are cold.Gary is cold.

Depth 2, closed-world task

92.0%
right on 276 of 300 problems wrong; chance 50% (dashed line)
right

Erin is nice.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

Erin is blue. Erin is cold. Erin is young. Young, blue people are big. Big, young people are nice.

  1. Erin is young. + Erin is blue.Young, blue people are big.Erin is big.
  2. + Erin is young.Big, young people are nice.Erin is nice.
right

The cow is not big.

Correct answer: false · Model said: false (100.0% sure)

The problem and its proof

The cow is red. Big, red things are kind. All rough, big things are red. If something is rough then it is red. All red, young things are big. If something is kind and red then it is big. If something is red then it is young.

  1. The cow is red.If something is red then it is young.The cow is young.
  2. + The cow is red.All red, young things are big.The cow is big.
right

Dave is round.

Correct answer: true · Model said: true (99.0% sure)

paraphrased
The problem and its proof

Charlie seems to be round. Dave is a kind person and he is also often cold. Eric is a round and rough around the edges, and he is also big. Gary is somehow red and kind and also green. Anyone who is both red and green must be as cold as Christmas. Someone being nice, even when they are cold without a doubt means that they are round about the middle. Anyone feeling cold is bound to act nice. Most young kind people tend to be red too.

  1. Dave is cold.If someone is cold then they are nice.Dave is nice.
  2. + Dave is cold.If someone is cold and nice then they are round.Dave is round.
right

The mouse does not see the cow.

Correct answer: true · Model said: true (99.0% sure)

The problem and its proof

The cat chases the cow. The cow chases the lion. The cow is round. The cow sees the lion. The lion chases the mouse. The lion is red. The mouse sees the cat. If someone chases the mouse then they chase the cat. If someone chases the cat then they see the cow. If someone is blue then they chase the cow. If someone chases the lion and they eat the mouse then the lion eats the cat. If someone sees the cow then the cow chases the mouse. If the cat sees the mouse and the cat is not red then the mouse eats the cat.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

85.0%
right on 255 of 300 problems wrong; chance 50% (dashed line)
right

Anne is quiet.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

Anne is rough. Charlie is rough. Charlie is white. Blue people are green. All quiet people are young. If someone is quiet and rough then they are young. If someone is smart then they are blue. All blue people are quiet. All rough people are smart.

  1. Anne is rough.All rough people are smart.Anne is smart.
  2. If someone is smart then they are blue.Anne is blue.
  3. All blue people are quiet.Anne is quiet.
right

Erin is smart.

Correct answer: false · Model said: false (100.0% sure)

The problem and its proof

Charlie is cold. Charlie is quiet. Charlie is young. Dave is furry. Dave is smart. Erin is furry. Harry is cold. If someone is red and furry then they are cold. Young, cold people are red. All quiet people are green. Cold people are smart. Green, young people are smart. All young, green people are smart. If Harry is green then Harry is young. All cold, smart people are furry. If someone is smart then they are quiet.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Harry is not nice.

Correct answer: true · Model said: true (97.0% sure)

The problem and its proof

Harry is young. All round, red people are quiet. All red people are quiet. Young, nice people are white. If Harry is red then Harry is quiet. Round people are nice. Quiet people are round. Smart people are white. Smart, round people are quiet.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Eric is round.

Correct answer: true · Model said: false (88.0% sure)

paraphrased
The problem and its proof

Alan might be rough and red but he's actually very kind. Eric is red and blue colored. He is cold to the touch, big and rough. Fred is young and green. This makes him cold to others. Harry seems to be round. Young, red people are usually quite blue too. A red hued and rough looking person is definitely a young person. People who are said to be big and nice are round. Often a young, blue person is very kind and then will be nice to others. A cold blue person who is rough is also kind.

  1. Eric is red. + Eric is rough.If someone is red and rough then they are young.Eric is young.
  2. + Eric is blue. + Eric is cold. + Eric is rough.If someone is blue and cold and rough then they are kind.Eric is kind.
  3. + Eric is blue.If someone is blue and young and kind then they are nice.Eric is nice.
  4. + Eric is big.If someone is big and nice then they are round.Eric is round.

Depth 4, closed-world task

76.3%
right on 229 of 300 problems wrong; chance 50% (dashed line)
wrong

Dave is cold.

Correct answer: true · Model said: false (66.0% sure)

paraphrased
The problem and its proof

Bob seems to be round. For being so cold, it's good Charlie can remain nice. Dave is a young, rough person who is rather round but is also kind to everyone he meets. Even though Fred is round and big, he is very kind. When you meet someone who is nice and kind, they wont feel blue. All young and kind people that feel blue are described as red. A person that is red, rough and kind often feels cold. Round people who are kind tend to be nice.

  1. Dave is round. + Dave is kind.If someone is round and kind then they are nice.Dave is nice.
  2. + Dave is kind.If someone is kind and nice then they are blue.Dave is blue.
  3. + Dave is young. + Dave is kind.If someone is blue and young and kind then they are red.Dave is red.
  4. + Dave is rough. + Dave is kind.If someone is rough and red and kind then they are cold.Dave is cold.
right

The lion does not see the lion.

Correct answer: true · Model said: true (65.0% sure)

The problem and its proof

The lion chases the squirrel. The lion is kind. The lion is round. The lion sees the squirrel. The mouse is blue. The mouse is round. The mouse sees the lion. The mouse sees the squirrel. The mouse visits the rabbit. The rabbit chases the lion. The rabbit is blue. The rabbit visits the lion. The squirrel chases the mouse. The squirrel is kind. The squirrel visits the lion. The squirrel visits the rabbit. If the rabbit is green then the rabbit sees the mouse. If something sees the rabbit then it sees the lion. If something visits the squirrel and it is not green then it sees the rabbit. If something sees the squirrel then the squirrel is blue. If something sees the lion and it is young then the lion sees the rabbit. If something is blue then it chases the rabbit. If something chases the rabbit then it visits the squirrel. If something visits the mouse and it is blue then the mouse sees the rabbit.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Bob is not young.

Correct answer: true · Model said: true (100.0% sure)

The problem and its proof

Anne is round. Anne is young. Bob is white. Fiona is red. Fiona is young. Gary is smart. Gary is young. All smart things are round. All red, young things are round. If Bob is quiet then Bob is young. All young, round things are kind. If something is red and not white then it is quiet. If something is kind and not red then it is quiet. If something is red and smart then it is quiet. If something is round and kind then it is young.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Charlie is red.

Correct answer: true · Model said: true (97.0% sure)

paraphrased
The problem and its proof

Charlie is a young, round shaped young man who is also very cold. Dave is green with envy, and rough with his kids, but overall is just a nice, round individual. That guy Gary sure is nice. Even though Harry is round and big, he is very kind. Rough people who are red in color and big are usally round in shape. Young round people who are green are usually blue. Being young and cold also means you're green. Anybody young person covered in blue and green will be kind. Most young kind people tend to be red too. Rough people who look red are cold. Young and red people look round.

  1. Charlie is young. + Charlie is cold.If someone is young and cold then they are green.Charlie is green.
  2. + Charlie is round. + Charlie is young.If someone is round and young and green then they are blue.Charlie is blue.
  3. + Charlie is young.If someone is young and blue and green then they are kind.Charlie is kind.
  4. + Charlie is young.If someone is young and kind then they are red.Charlie is red.

Depth 5, closed-world task

89.3%
right on 268 of 300 problems wrong; chance 50% (dashed line)
right

The mouse needs the mouse.

Correct answer: true · Model said: true (57.0% sure)

The problem and its proof

The cat sees the cow. The cat sees the mouse. The cow likes the dog. The cow likes the mouse. The cow sees the dog. The dog is blue. The dog is nice. The dog likes the mouse. The mouse is green. The mouse needs the cat. If someone sees the dog and they do not like the cow then they are nice. If someone needs the dog and the dog is kind then they like the dog. If someone is nice and they do not like the cow then they need the mouse. If someone needs the mouse and they are not blue then the mouse sees the dog. If someone likes the cow and they are not nice then they are green. If someone needs the cow and they are not green then they are big.

  1. The cow sees the dog.If someone sees the dog and they do not like the cow then they are nice.The cow is nice.
  2. If someone is nice and they do not like the cow then they need the mouse.The cow needs the mouse.
  3. If someone needs the mouse and they are not blue then the mouse sees the dog.The mouse sees the dog.
  4. If someone sees the dog and they do not like the cow then they are nice.The mouse is nice.
  5. If someone is nice and they do not like the cow then they need the mouse.The mouse needs the mouse.
right

Harry is not rough.

Correct answer: false · Model said: false (94.0% sure)

The problem and its proof

Anne is rough. Anne is young. Bob is furry. Bob is smart. Erin is big. Erin is blue. Erin is furry. Erin is quiet. Erin is rough. Harry is smart. If someone is furry and rough then they are smart. All smart people are furry. All rough people are furry. Young people are rough. Furry, big people are young. Blue people are young. Furry people are quiet. If someone is big then they are quiet. All quiet, smart people are big.

  1. Harry is smart.All smart people are furry.Harry is furry.
  2. Furry people are quiet.Harry is quiet.
  3. + Harry is smart.All quiet, smart people are big.Harry is big.
  4. Furry, big people are young.Harry is young.
  5. Young people are rough.Harry is rough.
right

Bob is not nice.

Correct answer: false · Model said: false (97.0% sure)

The problem and its proof

Anne is cold. Anne is green. Anne is nice. Anne is round. Bob is cold. Bob is white. Dave is cold. Dave is kind. Dave is round. Harry is blue. Harry is cold. Harry is green. Harry is kind. Harry is nice. Harry is round. Harry is white. All white things are green. If something is blue then it is kind. Green, cold things are round. Round, cold things are blue. Green things are white. Kind things are nice. White, green things are cold.

  1. Bob is white.All white things are green.Bob is green.
  2. + Bob is cold.Green, cold things are round.Bob is round.
  3. + Bob is cold.Round, cold things are blue.Bob is blue.
  4. If something is blue then it is kind.Bob is kind.
  5. Kind things are nice.Bob is nice.
right

The squirrel is green.

Correct answer: true · Model said: true (90.0% sure)

The problem and its proof

The bald eagle chases the squirrel. The bald eagle eats the squirrel. The bald eagle is big. The bald eagle is green. The cow eats the bald eagle. The cow is cold. The cow sees the bald eagle. The cow sees the dog. The dog chases the squirrel. The dog is green. The dog is round. The dog sees the squirrel. The squirrel is cold. The squirrel sees the cow. If someone is red and they eat the dog then they see the squirrel. If someone is round then they eat the squirrel. If someone eats the squirrel and they do not eat the bald eagle then they chase the bald eagle. If someone chases the bald eagle and they do not see the dog then they are green. If someone chases the dog and they are big then they are green. If the dog is round and the dog eats the squirrel then the squirrel is round.

  1. The dog is round.If someone is round then they eat the squirrel.The dog eats the squirrel.
  2. + The dog is round.If the dog is round and the dog eats the squirrel then the squirrel is round.The squirrel is round.
  3. If someone is round then they eat the squirrel.The squirrel eats the squirrel.
  4. If someone eats the squirrel and they do not eat the bald eagle then they chase the bald eagle.The squirrel chases the bald eagle.
  5. If someone chases the bald eagle and they do not see the dog then they are green.The squirrel is green.

True, false and unknown: where Jev errs

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what Jev said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknownRecall
true53756388.8%
false105346188.3%
unknown1242843874.2%

Jev answered false 31.5%, true 37.3%, unknown 31.2% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalseRecall
true8079389.7%
false10080088.9%

Jev answered false 49.6%, true 50.4% of the time; the correct answers are false 50.0%, true 50.0%.

Jev at depth 0 and depth 5

Jev, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9802
false0991
unknown3196
Jev, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true9407
false2936
unknown31947

Open-world task. At depth 0, Jev was right on 98% of problems (the green diagonal); at depth 5, on 81%. The answer whose share moved most is unknown: 33% of its answers at depth 0, 21% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
Jev, closed-world task, depth 0
Correct ↓ · said →truefalse
true1500
false2148
Jev, closed-world task, depth 5
Correct ↓ · said →truefalse
true13416
false16134

Closed-world task. At depth 0, Jev was right on 99% of problems (the green diagonal); at depth 5, on 89%. The answer whose share moved most is true: 51% of its answers at depth 0, 50% at depth 5, against 50% and 50% of the correct answers.

Jev across every breakdown

For each property of the problems, Jev's weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownWeakest · strongest
Proof depthdepth 4 71.9% · depth 0 97.7% (25.7 points)
Gold answerunknown 74.2% · true 88.8% (14.5 points)
Theory kindattribute 82.8% · relation 85.3% (2.5 points)
Negation in the theorywithout negation 80.3% · with negation 87.5% (7.2 points)
Negated statementnegated statement 78.6% · plain statement 89.1% (10.5 points)
Question strategyinv-rconc 50.2% · random 98.0% (47.8 points)
Paraphrased rulesparaphrased (NatLang) 78.1% · templated 86.1% (8.0 points)
Theory's deepest proof4 67.9% · 0 90.1% (22.2 points)
Theory length80-109 79.8% · 0-49 93.1% (13.3 points)
Number of rules8+ 80.0% · 0-2 92.2% (12.2 points)
Number of facts8-12 80.3% · 0-3 90.4% (10.1 points)
Proof size0-1 80.4% · 2 98.1% (17.8 points)

Jev against every other model

Paired differences: both models answered exactly the same items, so the gap has its own 95% interval. Positive numbers mean Jev was more accurate. GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.

Jev minus GLiDE, Oct 3 rerun, open world: +7.4 points overall (+5.2 to +9.9)

depth 0300 items
0.0
depth 1302 items
+0.7
depth 2303 items
+5.0
depth 3303 items
+9.9
depth 4303 items
+10.6
depth 5289 items
+19.0

Jev minus GLiDE, Oct 2, open world: +0.9 points overall (−1.2 to +3.3)

depth 0300 items
0.0
depth 1302 items
0.0
depth 2303 items
+0.7
depth 3303 items
+1.3
depth 4303 items
−2.0
depth 5289 items
+5.9

Jev minus GPT-6 Luna, open world: +19.8 points overall (+17.2 to +22.4)

depth 0300 items
+4.3
depth 1302 items
+9.9
depth 2303 items
+24.4
depth 3303 items
+18.2
depth 4303 items
+27.4
depth 5289 items
+34.9

Jev minus Kev-9B, open world: +25.3 points overall (+22.7 to +27.8)

depth 0300 items
+8.3
depth 1302 items
+13.9
depth 2303 items
+26.4
depth 3303 items
+30.4
depth 4303 items
+33.3
depth 5289 items
+39.8

Jev minus Kev-4B, open world: +30.3 points overall (+27.4 to +32.9)

depth 0300 items
+10.0
depth 1302 items
+24.8
depth 2303 items
+32.7
depth 3303 items
+35.3
depth 4303 items
+34.7
depth 5289 items
+44.6

Jev minus Kev-0.8B, open world: +30.3 points overall (+27.7 to +32.7)

depth 0300 items
+27.3
depth 1302 items
+33.8
depth 2303 items
+38.0
depth 3303 items
+31.0
depth 4303 items
+21.5
depth 5289 items
+30.1

Jev minus GLiNER2.5-Decide, open world: +51.1 points overall (+48.3 to +54.1)

depth 0300 items
+64.3
depth 1302 items
+54.6
depth 2303 items
+51.5
depth 3303 items
+46.5
depth 4303 items
+38.6
depth 5289 items
+50.9

Jev minus GLiNER2.5-Decide, without instructions, open world: +38.3 points overall (+35.4 to +41.3)

depth 0300 items
+52.7
depth 1302 items
+43.4
depth 2303 items
+40.6
depth 3303 items
+35.6
depth 4303 items
+21.5
depth 5289 items
+36.3

Jev minus Laya, open world: +41.4 points overall (+38.8 to +44.0)

depth 0300 items
+35.0
depth 1302 items
+45.7
depth 2303 items
+43.6
depth 3303 items
+41.6
depth 4303 items
+36.0
depth 5289 items
+46.7

Jev minus GLiDE, Oct 3 rerun, closed world: +20.4 points overall (+18.2 to +22.7)

depth 0300 items
+0.3
depth 1300 items
+5.7
depth 2300 items
+23.7
depth 3300 items
+23.7
depth 4300 items
+24.0
depth 5300 items
+45.3

Jev minus GLiDE, Oct 2, closed world: +17.5 points overall (+15.4 to +19.6)

depth 0300 items
0.0
depth 1300 items
+3.7
depth 2300 items
+20.0
depth 3300 items
+20.7
depth 4300 items
+20.7
depth 5300 items
+40.0

Jev minus GPT-6 Luna, closed world: +24.3 points overall (+21.9 to +26.6)

depth 0300 items
+1.3
depth 1300 items
+17.0
depth 2300 items
+26.7
depth 3300 items
+28.0
depth 4300 items
+28.0
depth 5300 items
+44.7

Jev minus Kev-9B, closed world: +25.5 points overall (+23.0 to +27.8)

depth 0300 items
+9.7
depth 1300 items
+16.7
depth 2300 items
+28.0
depth 3300 items
+26.3
depth 4300 items
+26.0
depth 5300 items
+46.3

Jev minus Kev-4B, closed world: +30.5 points overall (+27.9 to +33.1)

depth 0300 items
+13.7
depth 1300 items
+25.3
depth 2300 items
+37.7
depth 3300 items
+29.0
depth 4300 items
+31.3
depth 5300 items
+46.0

Jev minus Kev-0.8B, closed world: +33.5 points overall (+30.7 to +36.3)

depth 0300 items
+31.3
depth 1300 items
+34.0
depth 2300 items
+40.7
depth 3300 items
+32.0
depth 4300 items
+25.3
depth 5300 items
+37.7

Jev minus GLiNER2.5-Decide, closed world: +38.9 points overall (+36.1 to +41.7)

depth 0300 items
+49.7
depth 1300 items
+43.0
depth 2300 items
+43.0
depth 3300 items
+34.0
depth 4300 items
+24.7
depth 5300 items
+39.3

Jev minus GLiNER2.5-Decide, without instructions, closed world: +35.0 points overall (+32.1 to +37.9)

depth 0300 items
+47.0
depth 1300 items
+39.3
depth 2300 items
+38.0
depth 3300 items
+31.7
depth 4300 items
+26.7
depth 5300 items
+27.3

Jev minus Laya, closed world: +33.7 points overall (+30.7 to +36.5)

depth 0300 items
+27.3
depth 1300 items
+42.7
depth 2300 items
+37.3
depth 3300 items
+33.7
depth 4300 items
+17.7
depth 5300 items
+43.3

Consistency and calibration

Asked every open world question a second time, Jev gave the same answer on 97.2% of 1,800 items (50 changed; Gwet's AC1 0.958, interval 0.948 to 0.969). Accuracy was 83.8% the first time and 84.2% the second.

Asked every closed world question a second time, Jev gave the same answer on 97.9% of 1,800 items (38 changed; Gwet's AC1 0.958, interval 0.944 to 0.971). Accuracy was 89.3% the first time and 89.7% the second.

Calibration on the open world task: Jev's probability for its own answer averaged 87.8% against 83.8% accuracy; expected calibration error 4.1 points.

Calibration on the closed world task: Jev's probability for its own answer averaged 92.1% against 89.3% accuracy; expected calibration error 2.9 points.

Size, hardware and speed

Parameters
undisclosed
Architecture
undisclosed
Weights
not available (hosted)
Where it ran
TypeSafe API (typesafe-sdk); version recorded per row (jev-1.13.0)
Maker's stated hardware
not applicable (hosted)
Price
$42 per billion input tokens; output free
Time per decision, open world
median 189 ms, 90th percentile 241 ms, over the network
Time per decision, closed world
median 170 ms, 90th percentile 207 ms, over the network

Hosted: timed over the network against the vendor's servers, so its latency is not like for like with a model on a laptop. Speed, size and memory for every model.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.