Skip to main content
HardDecisions

Comparison · GLiNER2.5-Decide vs Jev

Jev vs GLiNER2.5-Decide: a hosted decision model against Fastino's open one

GLiNER2.5-Decide is Fastino's open 340M-parameter decision model, released under Apache 2.0, that answers in one pass. We ran it on a laptop against Jev on the same 3,600 problems.

Every model received the same problem, the same question and the same meaning for each answer. GLiNER2.5-Decide is shown two ways. GLiNER2.5-Decide: the instructions and the meaning of each answer are in its text, after the problem, because Fastino's API for this model has no separate field for them; given them, it answered unknown to every open-world problem and false to almost every closed-world one. GLiNER2.5-Decide, without instructions: only the problem and the three answer words; its answers vary but stay near chance, even at depth 0, where the answer is written in the text. Fastino's hosted API for this model gave the same answer as our copy on 7,197 of the 7,200 problems in both forms, and the same setup scores 63.7% on Fastino's own benchmark data (Fastino reports 60.2%), so the model is working as published. Asked a plain yes-or-no question in the form its model card shows (“Is it true that …?”, with the facts as the passage), it does better: 76% at depth 0 on held-out ProofWriter problems whose answer is true or false, against Jev's 98% on the benchmark. That form leaves out the instructions, so it is not a benchmark entry.

Overall: Jev, GLiNER2.5-Decide and GLiNER2.5-Decide, without instructions

Open world: true, false or unknown

  • Jev82.2 to 85.5 (95%), 1,800 items83.8%
  • GLiNER2.5-Decide30.7 to 34.9 (95%), 1,800 items32.8%
  • GLiNER2.5-Decide, without instructions43.1 to 47.8 (95%), 1,800 items45.5%

Jev minus GLiNER2.5-Decide: +51.1 points on the same 1,800 items, 95% interval +48.3 to +54.1: Jev is clearly ahead.

Jev minus GLiNER2.5-Decide, without instructions: +38.3 points on the same 1,800 items, 95% interval +35.4 to +41.3: Jev is clearly ahead.

Closed world: true or false

  • Jev87.8 to 90.7 (95%), 1,800 items89.3%
  • GLiNER2.5-Decide48.1 to 52.6 (95%), 1,800 items50.3%
  • GLiNER2.5-Decide, without instructions52.0 to 56.6 (95%), 1,800 items54.3%

Jev minus GLiNER2.5-Decide: +38.9 points on the same 1,800 items, 95% interval +36.1 to +41.7: Jev is clearly ahead.

Jev minus GLiNER2.5-Decide, without instructions: +35.0 points on the same 1,800 items, 95% interval +32.1 to +37.9: Jev is clearly ahead.

Jev vs GLiNER2.5-Decide and GLiNER2.5-Decide, without instructions by proof depth

Each point is about 300 problems that need that many inference steps, with its 95% interval.

Open world: true, false or unknown
  • Jev
  • GLiNER2.5-Decide
  • GLiNER2.5-Decide, without instructions
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
  • GLiNER2.5-Decide
  • GLiNER2.5-Decide, without instructions
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Jev minus GLiNER2.5-Decide by depth, open world

depth 0300 items
+64.3
depth 1302 items
+54.6
depth 2303 items
+51.5
depth 3303 items
+46.5
depth 4303 items
+38.6
depth 5289 items
+50.9

Jev minus GLiNER2.5-Decide by depth, closed world

depth 0300 items
+49.7
depth 1300 items
+43.0
depth 2300 items
+43.0
depth 3300 items
+34.0
depth 4300 items
+24.7
depth 5300 items
+39.3

Jev minus GLiNER2.5-Decide, without instructions by depth, open world

depth 0300 items
+52.7
depth 1302 items
+43.4
depth 2303 items
+40.6
depth 3303 items
+35.6
depth 4303 items
+21.5
depth 5289 items
+36.3

Jev minus GLiNER2.5-Decide, without instructions by depth, closed world

depth 0300 items
+47.0
depth 1300 items
+39.3
depth 2300 items
+38.0
depth 3300 items
+31.7
depth 4300 items
+26.7
depth 5300 items
+27.3

Paired differences: positive means Jev was more accurate on the same items.

Accuracy by depth, open world

By depth, Open world
Proof depthItemsJevGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsBest constant
depth 030097.7% 96.0–99.333.3% 27.7–38.745.0% 39.7–51.033.3%
depth 130287.7% 83.8–91.433.1% 27.8–38.744.4% 38.7–49.733.4%
depth 230384.8% 80.2–88.433.3% 27.7–38.644.2% 38.6–50.233.3%
depth 330379.9% 75.2–84.533.3% 27.7–38.644.2% 38.6–50.233.3%
depth 430371.9% 66.7–77.233.3% 27.7–38.650.5% 45.2–56.433.3%
depth 528981.0% 76.5–85.530.1% 24.9–35.344.6% 38.4–50.534.9%

Accuracy by depth, closed world

By depth, Closed world
Proof depthItemsJevGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsBest constant
depth 030099.3% 98.3–100.049.7% 44.0–55.052.3% 46.7–58.050.0%
depth 130093.7% 91.0–96.350.7% 45.0–56.354.3% 49.0–59.750.0%
depth 230092.0% 88.7–94.749.0% 43.0–54.354.0% 48.3–60.050.0%
depth 330085.0% 81.3–89.351.0% 45.3–56.353.3% 47.3–59.050.0%
depth 430076.3% 71.7–81.051.7% 46.0–57.049.7% 44.3–55.350.0%
depth 530089.3% 85.7–92.350.0% 44.3–55.762.0% 56.3–67.350.0%

What GLiNER2.5-Decide got right and wrong, depth by depth

At each proof depth, GLiNER2.5-Decide's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

33.3%
right on 100 of 300 problems wrong; chance 33% (dashed line)
wrong

Gary is blue.

Correct answer: true · Model said: unknown (92.0% sure)

paraphrased
The problem and its proof

Dave seems to be round. Fred is green and cold too. Gary, a kind young person is always feeling blue. Harry is a kind person and he is also often cold. People who are very nice and easily flush red are often wearing green concealer. Most young kind people tend to be red too. A kind person who looks blue because he is is cold is usually big in stature. Is a kind person if feeling cold and looking red then they are usually young. Any red sort of person is a nice person.

Stated in the text: "Gary is blue."

right

The dog does not like the dog.

Correct answer: unknown · Model said: unknown (93.9% sure)

The problem and its proof

The cat sees the dog. The cat visits the squirrel. The dog is blue. The dog does not like the cat. The dog does not like the lion. The dog likes the squirrel. The lion is blue. The lion is nice. The lion likes the cat. The lion likes the dog. The squirrel is not red. The squirrel sees the cat. Blue things are nice. If something sees the squirrel then the squirrel sees the dog. If something likes the squirrel and it sees the dog then the dog does not visit the squirrel.

No proof: neither the statement nor its negation follows.

wrong

The bear does not eat the mouse.

Correct answer: true · Model said: unknown (93.9% sure)

The problem and its proof

The bear does not eat the mouse. The bear is blue. The bear is not cold. The bear is green. The bear is red. The bear does not need the mouse. The bear visits the mouse. The mouse does not eat the bear. The mouse is nice. The mouse is red. The mouse does not need the bear. The mouse does not visit the bear. If the bear is nice and the mouse does not eat the bear then the bear visits the mouse.

Stated in the text: "The bear does not eat the mouse."

wrong

Harry is round.

Correct answer: false · Model said: unknown (95.0% sure)

The problem and its proof

Erin is round. Erin is smart. Fiona is nice. Fiona is smart. Harry is blue. Harry is red. Harry is not round. If Harry is red and Harry is not round then Harry is rough. If something is smart then it is red.

Stated in the text: "Harry is not round."

Depth 1, open-world task

33.1%
right on 100 of 302 problems wrong; chance 33% (dashed line)
wrong

The dog likes the bear.

Correct answer: true · Model said: unknown (95.0% sure)

The problem and its proof

The bald eagle is green. The bald eagle is young. The bald eagle sees the dog. The bear likes the dog. The bear needs the cow. The cow needs the dog. The cow sees the dog. The dog is blue. The dog is green. The dog is young. The dog needs the bear. The dog needs the cow. If someone sees the bald eagle and they are nice then the bald eagle needs the bear. If someone is nice and young then they need the dog. If someone likes the cow and the cow needs the dog then the cow is kind. If someone is young and blue then they like the bear. If someone is blue and they like the bear then the bear likes the cow. If someone is green and they need the bear then they need the dog. If someone sees the bear then they are nice. If someone is kind then they see the bear.

  1. The dog is young. + The dog is blue.If someone is young and blue then they like the bear.The dog likes the bear.
wrong

Dave is not quiet.

Correct answer: false · Model said: unknown (92.7% sure)

The problem and its proof

Dave is big. Dave is blue. Dave is furry. Dave is red. Erin is blue. Erin is not furry. Fiona is not quiet. If Dave is not furry then Dave is rough. All quiet people are not cold. Red, blue people are quiet.

The proof that the statement's negation follows:

  1. Dave is red. + Dave is blue.Red, blue people are quiet.Dave is quiet.
right

Gary is red.

Correct answer: unknown · Model said: unknown (93.6% sure)

paraphrased
The problem and its proof

Eric is round but he is also nice and kind. Fred is quite rough and very kind, but unfortunately cold and also blue. Gary seems to be round. That guy Harry sure is nice. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Every time you meet someone kind and nice, they'll be green, too. Most big and cold people are also rough. Blue eyed people, green with sickness and rough around the edges quickly turn red and blush when stepping ashore. Is a kind person if feeling cold and looking red then they are usually young.

No proof: neither the statement nor its negation follows.

wrong

Bob is cold.

Correct answer: true · Model said: unknown (86.3% sure)

paraphrased
The problem and its proof

Round, nice Bob felt as blue and big as he had ever been. The young person who is always feeling cold is named Charlie. That guy Eric sure is nice. That guy Fred sure is nice. People here that are both big and kind while being cold are always red. A person feeling blue because of their big size will always be cold. You must be aware of big, red, young people being rough. Do not fear big cold people for they are often kind. People that are rough and red in the face, are usually big in size.

  1. Bob is blue. + Bob is big.If someone is blue and big then they are cold.Bob is cold.

Depth 2, open-world task

33.3%
right on 101 of 303 problems wrong; chance 33% (dashed line)
right

The dog is not blue.

Correct answer: unknown · Model said: unknown (91.4% sure)

The problem and its proof

The dog is not red. If the dog is not cold then the dog is green. Green people are blue. If someone is cold and not rough then they are green. Cold people are not green. If the dog is rough then the dog is blue. If someone is blue and not green then they are red.

No proof: neither the statement nor its negation follows.

wrong

Bob is rough.

Correct answer: false · Model said: unknown (90.8% sure)

The problem and its proof

Bob is white. If someone is round and smart then they are furry. If someone is nice then they are furry. If Bob is white then Bob is furry. All furry people are not rough. If Bob is smart then Bob is not white. All rough people are not kind. Rough, round people are kind. If someone is white and not rough then they are not kind.

The proof that the statement's negation follows:

  1. Bob is white.If Bob is white then Bob is furry.Bob is furry.
  2. All furry people are not rough.Bob is not rough.
wrong

Alan is red.

Correct answer: true · Model said: unknown (91.8% sure)

paraphrased
The problem and its proof

Alan seems big and rough, but he's actually quite kind. Even though Charlie is round and big, he is very kind. Dave is green and cold too. That guy Eric sure is nice. Rough people who are red in color and big are usally round in shape. Big, kind folks are green ones. Kind red and round people often look green. When someone is blue, nice and young they will also be as big as the sun. When you run into someone who is rough and green at the same time, they will also be red.

  1. Alan is big. + Alan is kind.If someone is big and kind then they are green.Alan is green.
  2. + Alan is rough.If someone is rough and green then they are red.Alan is red.
wrong

Charlie is blue.

Correct answer: true · Model said: unknown (90.0% sure)

paraphrased
The problem and its proof

For being so cold, it's good Charlie can remain nice. Eric is a young, rough person who is rather round but is also kind to everyone he meets. Fred was proud of being round, yet rough. His red cheeks glowed. A nice but rough person will also be green and cold. Someone being nice, even when they are cold without a doubt means that they are round about the middle. Kind people that are round are on the big side. A nice person who feels blue and looks round is usually kind. If a person is both cold and round, that person is also someone who is red. People who are round and behave in a cold way are surely blue.

  1. Charlie is cold. + Charlie is nice.If someone is cold and nice then they are round.Charlie is round.
  2. + Charlie is cold.If someone is round and cold then they are blue.Charlie is blue.

Depth 3, open-world task

33.3%
right on 101 of 303 problems wrong; chance 33% (dashed line)
right

The bear is nice.

Correct answer: unknown · Model said: unknown (94.4% sure)

The problem and its proof

The bald eagle visits the squirrel. The bear needs the squirrel. The cow is nice. The squirrel is big. If someone is blue then they are nice. If someone visits the squirrel then they are kind. Kind people are blue.

No proof: neither the statement nor its negation follows.

wrong

The bald eagle does not chase the cow.

Correct answer: true · Model said: unknown (95.6% sure)

The problem and its proof

The bald eagle is kind. The bald eagle needs the cow. The cow chases the bald eagle. The cow eats the squirrel. The cow is not green. The cow is red. The squirrel is not rough. If something eats the bald eagle then the bald eagle eats the cow. If the bald eagle is red and the bald eagle does not eat the squirrel then the squirrel is rough. If something eats the cow then it does not chase the cow. If something needs the cow and the cow needs the squirrel then it chases the bald eagle. If something chases the bald eagle then it eats the bald eagle. If something needs the squirrel then the squirrel is red. If something needs the cow then it chases the squirrel. If something chases the cow and it needs the cow then it is not green.

  1. The cow chases the bald eagle.If something chases the bald eagle then it eats the bald eagle.The cow eats the bald eagle.
  2. If something eats the bald eagle then the bald eagle eats the cow.The bald eagle eats the cow.
  3. If something eats the cow then it does not chase the cow.The bald eagle does not chase the cow.
wrong

The squirrel is not round.

Correct answer: true · Model said: unknown (94.6% sure)

The problem and its proof

The cow is red. The cow is not round. The cow likes the squirrel. The cow needs the lion. The lion needs the cow. The lion does not visit the squirrel. The squirrel is blue. The squirrel is red. The squirrel likes the lion. The squirrel needs the lion. If something likes the squirrel and it is round then it visits the cow. If something needs the squirrel then it is not round. If something likes the squirrel then the squirrel visits the lion. If the squirrel visits the cow and the squirrel likes the lion then the cow likes the squirrel. All red things are blue. If something visits the lion then it needs the squirrel.

  1. The cow likes the squirrel.If something likes the squirrel then the squirrel visits the lion.The squirrel visits the lion.
  2. If something visits the lion then it needs the squirrel.The squirrel needs the squirrel.
  3. If something needs the squirrel then it is not round.The squirrel is not round.
wrong

The cow is blue.

Correct answer: true · Model said: unknown (93.9% sure)

The problem and its proof

The bear is rough. The bear needs the dog. The bear visits the cat. The bear visits the dog. The cat is blue. The cat likes the cow. The cow is rough. The cow likes the bear. The dog needs the bear. The dog visits the cat. All blue people are big. If someone visits the cow then they need the cow. If someone is nice then they like the cow. If someone likes the cat and they are nice then they are big. If someone is blue then they visit the cow. If the cat needs the cow then the cow is blue.

  1. The cat is blue.If someone is blue then they visit the cow.The cat visits the cow.
  2. If someone visits the cow then they need the cow.The cat needs the cow.
  3. If the cat needs the cow then the cow is blue.The cow is blue.

Depth 4, open-world task

33.3%
right on 101 of 303 problems wrong; chance 33% (dashed line)
wrong

Fiona is not young.

Correct answer: false · Model said: unknown (94.4% sure)

The problem and its proof

Bob is kind. Bob is quiet. Bob is young. Charlie is not green. Dave is round. Fiona is rough. Fiona is round. Kind, blue people are quiet. Rough, round people are quiet. Round people are rough. If someone is blue then they are green. All rough, kind people are green. If someone is quiet then they are blue. All green, round people are young. All kind people are round. If Charlie is rough then Charlie is blue.

The proof that the statement's negation follows:

  1. Fiona is rough. + Fiona is round.Rough, round people are quiet.Fiona is quiet.
  2. If someone is quiet then they are blue.Fiona is blue.
  3. If someone is blue then they are green.Fiona is green.
  4. + Fiona is round.All green, round people are young.Fiona is young.
right

Bob is green.

Correct answer: unknown · Model said: unknown (92.3% sure)

The problem and its proof

Bob is round. Dave is big. Dave is young. Fiona is quiet. Gary is cold. Gary is red. Gary is young. All cold things are big. If something is quiet then it is young. Round, cold things are quiet. Young, big things are green. All big, green things are round. Young things are cold.

No proof: neither the statement nor its negation follows.

wrong

Harry is cold.

Correct answer: true · Model said: unknown (90.9% sure)

paraphrased
The problem and its proof

Alan is kind and nice and looks green. That guy Gary sure is nice. Young Harry here is always rough but nice and yes he is big. When someone is kind yet can be cold and blue, they will also be very big. A nice young person who is big can be considered round. A red, nice person will definitely be a blue person. It's pretty typical of red, rough, and cold people to be round. Young people are nice and cold are kind to people regardless. Someone who is a young age and looks like they are round are also red. Any person who can be red and blue at the same time is cold.

  1. Harry is nice. + Harry is big. + Harry is young.If someone is nice and big and young then they are round.Harry is round.
  2. + Harry is young.If someone is young and round then they are red.Harry is red.
  3. + Harry is nice.If someone is red and nice then they are blue.Harry is blue.
  4. If someone is red and blue then they are cold.Harry is cold.
wrong

Eric is young.

Correct answer: true · Model said: unknown (93.4% sure)

paraphrased
The problem and its proof

Even though Bob is big and rough, he is kind. Bob is green with envy. That guy Dave sure is nice. Eric was proud of being round, yet rough. His red cheeks glowed. Fred who is young and round is also big and rough. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Any person that's blue, young and green will turn out to be a nice person, too. A red, nice person will definitely be a blue person. Why is it that every round person that is blue is so young. Rough people who look red are cold. A nice person with cold skin is going to be rough. A person who inherits genes that make them big and green will also express a blue color.

  1. Eric is red. + Eric is rough.If someone is red and rough then they are cold.Eric is cold.
  2. + Eric is rough.If someone is cold and rough then they are nice.Eric is nice.
  3. + Eric is red.If someone is red and nice then they are blue.Eric is blue.
  4. + Eric is round.If someone is blue and round then they are young.Eric is young.

Depth 5, open-world task

30.1%
right on 87 of 289 problems wrong; chance 33% (dashed line)
wrong

Gary is kind.

Correct answer: false · Model said: unknown (92.3% sure)

The problem and its proof

Anne is not blue. Anne is cold. Anne is kind. Anne is nice. Anne is quiet. Erin is quiet. Gary is not nice. Gary is rough. Harry is not blue. Harry is cold. Harry is nice. Harry is quiet. If Gary is not nice then Gary is cold. Rough, cold things are blue. If something is kind then it is cold. If something is white and not nice then it is quiet. If something is cold and blue then it is white. All kind, blue things are not white. If something is quiet and rough then it is not kind.

The proof that the statement's negation follows:

  1. Gary is not nice.If Gary is not nice then Gary is cold.Gary is cold.
  2. + Gary is rough.Rough, cold things are blue.Gary is blue.
  3. If something is cold and blue then it is white.Gary is white.
  4. + Gary is not nice.If something is white and not nice then it is quiet.Gary is quiet.
  5. + Gary is rough.If something is quiet and rough then it is not kind.Gary is not kind.
wrong

The tiger is red.

Correct answer: true · Model said: unknown (95.7% sure)

The problem and its proof

The bear chases the dog. The bear is young. The dog chases the tiger. The dog likes the bear. The dog likes the lion. The dog likes the tiger. The lion chases the tiger. The lion eats the bear. The lion likes the bear. The lion likes the dog. The tiger eats the lion. The tiger is cold. The tiger is young. The tiger likes the bear. If something likes the lion then it chases the lion. If something chases the dog and it is red then the dog is cold. If something likes the tiger then it likes the lion. If something chases the bear and the bear is round then the bear chases the lion. If something chases the lion then it is red. If something eats the dog then the dog chases the lion. If something is round then it likes the tiger. All young things are round.

  1. The tiger is young.All young things are round.The tiger is round.
  2. If something is round then it likes the tiger.The tiger likes the tiger.
  3. If something likes the tiger then it likes the lion.The tiger likes the lion.
  4. If something likes the lion then it chases the lion.The tiger chases the lion.
  5. If something chases the lion then it is red.The tiger is red.
right

Gary is not green.

Correct answer: unknown · Model said: unknown (92.9% sure)

The problem and its proof

Bob is cold. Charlie is green. Charlie is quiet. Charlie is white. Charlie is young. Gary is quiet. Gary is white. If something is young then it is red. All white things are quiet. Quiet, young things are cold. If Gary is quiet and Gary is green then Gary is young. If something is red then it is green. If Gary is round then Gary is white. White, young things are green. If something is red then it is young.

No proof: neither the statement nor its negation follows.

wrong

The bear does not need the bear.

Correct answer: false · Model said: unknown (95.4% sure)

The problem and its proof

The bear eats the cow. The bear eats the lion. The cow chases the dog. The cow needs the dog. The dog eats the bear. The dog does not eat the cow. The dog eats the lion. The dog is blue. The dog is kind. The lion chases the dog. The lion eats the cow. The lion is kind. If something chases the lion and it needs the lion then the lion does not chase the cow. Kind things are young. If something needs the bear and it eats the bear then the bear is kind. If something eats the lion then it chases the lion. If something is young then it needs the bear.

The proof that the statement's negation follows:

  1. The dog is kind.Kind things are young.The dog is young.
  2. If something is young then it needs the bear.The dog needs the bear.
  3. + The dog eats the bear.If something needs the bear and it eats the bear then the bear is kind.The bear is kind.
  4. Kind things are young.The bear is young.
  5. If something is young then it needs the bear.The bear needs the bear.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

49.7%
right on 149 of 300 problems wrong; chance 50% (dashed line)
wrong

The rabbit is not cold.

Correct answer: true · Model said: false (68.4% sure)

The problem and its proof

The dog needs the rabbit. The dog visits the rabbit. The rabbit eats the dog. If something visits the dog then it is young. Round things are young. If something needs the dog and it visits the dog then the dog eats the rabbit. If something is nice then it visits the dog. If something visits the rabbit and the rabbit is young then it eats the rabbit. If something is young then it eats the rabbit. If something is cold and it eats the dog then it needs the rabbit. If something visits the rabbit and the rabbit eats the dog then it is round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cat is not rough.

Correct answer: false · Model said: false (65.7% sure)

The problem and its proof

The cat eats the tiger. The cat is blue. The cat is rough. The cat is young. The cat needs the tiger. The cat sees the tiger. The tiger eats the cat. The tiger is blue. The tiger is rough. The tiger is round. The tiger needs the cat. The tiger sees the cat. If the tiger needs the cat then the tiger is rough. If the tiger needs the cat and the cat is rough then the cat needs the tiger.

Stated in the text: "The cat is rough."

right

Fred is not cold.

Correct answer: false · Model said: false (52.8% sure)

paraphrased
The problem and its proof

Bob seems to be round. Dave may be young but he is nice, wears green shoes and is cold. For being so cold, it's good Fred can remain nice. The young person who is always feeling cold is named Gary. Young round people who are green are usually blue. If a person is described as being nice, cold and young it usually follow that they are red. One who is young, red and also cold will definitely be round, too. Someone that is cold rough and red is also considered to be kind. Every person I met that was cold and round was also red. A person who is green, blue, and red in color will be round shaped. A person who inherits genes that make them big and green will also express a blue color.

Stated in the text: "Fred is cold."

wrong

The tiger is red.

Correct answer: true · Model said: false (67.5% sure)

The problem and its proof

The bald eagle chases the tiger. The bald eagle is blue. The bald eagle likes the tiger. The bald eagle sees the tiger. The tiger chases the bald eagle. The tiger is blue. The tiger is kind. The tiger is red. The tiger likes the bald eagle. The tiger sees the bald eagle. If the tiger is blue then the tiger chases the bald eagle. If someone is kind and they like the bald eagle then they like the tiger. If someone sees the bald eagle and they like the bald eagle then the bald eagle sees the tiger.

Stated in the text: "The tiger is red."

Depth 1, closed-world task

50.7%
right on 152 of 300 problems wrong; chance 50% (dashed line)
wrong

The lion is blue.

Correct answer: true · Model said: false (55.7% sure)

The problem and its proof

The lion is green. All blue things are young. If something is green then it is blue. All young things are rough.

  1. The lion is green.If something is green then it is blue.The lion is blue.
right

The squirrel likes the dog.

Correct answer: false · Model said: false (59.6% sure)

The problem and its proof

The cat is kind. The dog chases the cat. The squirrel is blue. If someone likes the dog then they like the squirrel. If someone is kind then they like the dog.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cow eats the cow.

Correct answer: false · Model said: false (65.1% sure)

The problem and its proof

The bear is blue. The bear is kind. The bear needs the dog. The cow visits the bear. The cow visits the dog. The dog is blue. The dog visits the bear. The squirrel eats the cow. The squirrel is kind. The squirrel needs the cow. If something is blue and it needs the cow then it eats the cow. If the bear visits the cow then the cow eats the squirrel. If something needs the cow then it is round. If something is round then it eats the bear. If the cow is kind and the cow does not need the squirrel then the cow eats the bear. If something visits the squirrel then it visits the dog. If something eats the bear then the bear needs the cow. If something is kind and not round then it is nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Gary is not young.

Correct answer: true · Model said: false (69.2% sure)

paraphrased
The problem and its proof

Eric is round but he is also nice and kind. Fred is quite rough and very kind, but unfortunately cold and also blue. Gary seems to be round. That guy Harry sure is nice. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Every time you meet someone kind and nice, they'll be green, too. Most big and cold people are also rough. Blue eyed people, green with sickness and rough around the edges quickly turn red and blush when stepping ashore. Is a kind person if feeling cold and looking red then they are usually young.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 2, closed-world task

49.0%
right on 147 of 300 problems wrong; chance 50% (dashed line)
wrong

The rabbit is not big.

Correct answer: true · Model said: false (62.2% sure)

The problem and its proof

The cat needs the rabbit. The cat sees the rabbit. The cow chases the cat. The cow chases the rabbit. The cow is blue. The rabbit chases the cow. The rabbit is green. If someone sees the rabbit then they see the cow. If someone sees the cow then they are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Eric is rough.

Correct answer: false · Model said: false (65.0% sure)

paraphrased
The problem and its proof

Bob is both young and red, but is nice and kind but he can be cold. Charlie seems to be round. Rough and cold that is what they say about Blue Fred. A person who is kind, cold, and rough is big. A blue, cold natured person will be rough textured. Someone with rough and green feet is invariably kind. Cold and red people are always kind to others. If someone is cold and blue, you will find that they are also quite young. A round shaped kind person who is colored green will be cold natured. People who are young are also blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cat needs the cat.

Correct answer: false · Model said: false (68.8% sure)

The problem and its proof

The bald eagle is rough. The bald eagle needs the cat. The bald eagle needs the dog. The bald eagle visits the cat. The bear sees the bald eagle. The bear visits the dog. The cat is green. The cat sees the bear. The cat sees the dog. The dog needs the bear. The dog sees the bald eagle. If something visits the dog then the dog needs the bald eagle. If something visits the dog then it sees the bald eagle. If something sees the dog then it is red. If something needs the cat and it needs the bear then the cat sees the bear. All red things are nice. If something is nice then it needs the bear. If something needs the cat and it visits the dog then it is blue. If something sees the bald eagle then it needs the cat. If the cat sees the bear and the bear needs the cat then the bear sees the dog.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Erin is not green.

Correct answer: true · Model said: false (63.3% sure)

The problem and its proof

Erin is kind. Erin is quiet. Erin is young. Cold things are white. If something is quiet and red then it is white. If something is white and not red then it is green.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

51.0%
right on 153 of 300 problems wrong; chance 50% (dashed line)
wrong

The mouse is not rough.

Correct answer: true · Model said: false (69.9% sure)

The problem and its proof

The bald eagle chases the mouse. The bald eagle chases the rabbit. The bald eagle is kind. The bald eagle is rough. The bald eagle likes the mouse. The cat chases the mouse. The cat chases the rabbit. The cat is blue. The cat is red. The cat is round. The cat likes the rabbit. The mouse chases the cat. The mouse eats the rabbit. The mouse is blue. The rabbit likes the cat. The rabbit likes the mouse. If someone is red then they like the mouse. If someone is kind and they do not chase the mouse then they are blue. If someone chases the cat and they chase the bald eagle then the bald eagle chases the rabbit. If someone eats the rabbit then they like the rabbit. If the mouse is blue then the mouse eats the bald eagle. If someone likes the cat and they like the bald eagle then they eat the bald eagle. If someone likes the bald eagle then the bald eagle eats the rabbit. If someone likes the mouse and they like the rabbit then they like the bald eagle. If someone likes the bald eagle then they are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Eric is not cold.

Correct answer: false · Model said: false (66.2% sure)

paraphrased
The problem and its proof

Alan is green and cold too. Bob seems to be round. Eric who is young and round is also big and rough. When a person is nice and round and cold, they look blue. Even though young people are rough, they are still very red. A person feeling blue because of their big size will always be cold. Young people who feel cold and are still kind usually look rough. Red people, who are also round and young, are often quite blue as well. I bet anyone who is round and kind is also pretty young.

  1. Eric is rough. + Eric is young.If someone is rough and young then they are red.Eric is red.
  2. + Eric is round. + Eric is young.If someone is red and round and young then they are blue.Eric is blue.
  3. + Eric is big.If someone is blue and big then they are cold.Eric is cold.
wrong

Harry is not big.

Correct answer: true · Model said: false (65.3% sure)

The problem and its proof

Anne is big. Anne is round. Dave is nice. Dave is red. Fiona is red. Fiona is rough. Harry is round. If someone is green and big then they are round. Red people are rough. Rough people are green. Nice, round people are green. If Harry is green and Harry is big then Harry is kind. All nice people are red. If someone is rough and green then they are kind. Kind, rough people are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Harry is furry.

Correct answer: false · Model said: false (64.3% sure)

The problem and its proof

Anne is blue. Anne is nice. Erin is young. Fiona is big. Fiona is nice. Fiona is red. Harry is quiet. If someone is blue then they are red. If Erin is not big then Erin is furry. All young, blue people are furry. If someone is quiet then they are blue. If Erin is nice then Erin is quiet. Big, nice people are young. If Harry is blue then Harry is big. All young, furry people are nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 4, closed-world task

51.7%
right on 155 of 300 problems wrong; chance 50% (dashed line)
right

Fiona is big.

Correct answer: false · Model said: false (65.7% sure)

The problem and its proof

Anne is blue. Anne is furry. Erin is blue. Erin is cold. Erin is rough. Fiona is cold. Fiona is young. Big, red people are young. If someone is furry then they are blue. If someone is blue then they are red. If someone is blue and red then they are big. Cold, red people are furry. All blue people are red. If Anne is red then Anne is rough. All big people are cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The dog needs the bear.

Correct answer: false · Model said: false (68.4% sure)

The problem and its proof

The bald eagle chases the dog. The bald eagle is big. The bald eagle is kind. The bald eagle is nice. The bear eats the bald eagle. The bear eats the cow. The bear is cold. The cow chases the bald eagle. The cow chases the dog. The cow eats the bear. The dog chases the bear. If something chases the dog then it eats the dog. If something eats the dog then it is young. If something chases the bald eagle and the bald eagle chases the cow then it is young. If the bear eats the bald eagle then the bear chases the dog. If something is young then it needs the bear. If the bear chases the bald eagle and the bald eagle is kind then the bald eagle is cold. If something needs the dog and the dog needs the bear then it is kind. If something needs the bear then it chases the cow. If something eats the dog and the dog is not big then it chases the dog.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Dave is not white.

Correct answer: true · Model said: false (66.5% sure)

The problem and its proof

Anne is rough. Dave is quiet. If someone is rough then they are quiet. If Anne is red then Anne is green. If Anne is green then Anne is red. If someone is kind and not rough then they are green. If someone is white and not green then they are blue. Rough, quiet people are blue. If someone is blue and not red then they are white. If someone is green and not kind then they are white.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The lion likes the cow.

Correct answer: true · Model said: false (67.2% sure)

The problem and its proof

The bald eagle is round. The bald eagle is young. The bald eagle likes the lion. The bald eagle needs the cow. The bald eagle sees the lion. The cow likes the lion. The cow likes the mouse. The cow needs the mouse. The lion is red. The lion is round. The mouse is round. The mouse likes the cow. The mouse needs the bald eagle. The mouse sees the bald eagle. If something sees the bald eagle then the bald eagle likes the mouse. If something is green then it likes the lion. If something sees the bald eagle then it likes the bald eagle. If something is red then it likes the bald eagle. If something likes the cow and it needs the bald eagle then it is green. If something is red and it likes the lion then the lion needs the bald eagle. If something needs the bald eagle then it likes the cow. If something needs the bald eagle then it is red.

  1. The mouse needs the bald eagle.If something needs the bald eagle then it is red.The mouse is red.
  2. + The mouse likes the cow. + The mouse needs the bald eagle.If something likes the cow and it needs the bald eagle then it is green.The mouse is green.
  3. If something is green then it likes the lion.The mouse likes the lion.
  4. If something is red and it likes the lion then the lion needs the bald eagle.The lion needs the bald eagle.
  5. If something needs the bald eagle then it likes the cow.The lion likes the cow.

Depth 5, closed-world task

50.0%
right on 150 of 300 problems wrong; chance 50% (dashed line)
wrong

Gary is not young.

Correct answer: true · Model said: false (67.1% sure)

The problem and its proof

Bob is big. Bob is blue. Dave is big. Dave is blue. Dave is furry. Dave is quiet. Dave is white. Gary is furry. Gary is red. Harry is big. Harry is furry. Harry is red. If Gary is quiet then Gary is young. All big things are young. Young, blue things are furry. All quiet things are white. All blue, white things are quiet. Blue, red things are quiet. White things are quiet. Furry things are red. Furry, quiet things are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Fiona is not young.

Correct answer: false · Model said: false (69.4% sure)

The problem and its proof

Anne is big. Anne is blue. Anne is round. Bob is green. Fiona is big. Gary is quiet. Gary is young. Big things are green. If something is round and quiet then it is young. All quiet things are round. If something is green and not red then it is blue. If something is round then it is blue. If something is young and not green then it is quiet. All blue, green things are quiet.

  1. Fiona is big.Big things are green.Fiona is green.
  2. If something is green and not red then it is blue.Fiona is blue.
  3. All blue, green things are quiet.Fiona is quiet.
  4. All quiet things are round.Fiona is round.
  5. If something is round and quiet then it is young.Fiona is young.
right

Gary is round.

Correct answer: false · Model said: false (62.3% sure)

The problem and its proof

Anne is big. Anne is young. Charlie is blue. Charlie is kind. Charlie is round. Charlie is young. Gary is big. Harry is big. Harry is blue. Harry is kind. Harry is nice. Harry is smart. All smart things are big. All kind, smart things are round. All young, kind things are blue. All nice, young things are smart. If something is big then it is kind. If something is blue then it is nice. If Anne is nice and Anne is kind then Anne is smart. If Gary is nice then Gary is young. If something is round and big then it is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Dave is not furry.

Correct answer: true · Model said: false (63.4% sure)

The problem and its proof

Charlie is big. Dave is red. Harry is smart. If Dave is green and Dave is smart then Dave is big. Red people are kind. If someone is red and blue then they are big. If someone is big then they are smart. If someone is furry and kind then they are smart. All blue, smart people are furry. If someone is smart then they are green. If someone is smart and big then they are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

What GLiNER2.5-Decide, without instructions got right and wrong, depth by depth

At each proof depth, GLiNER2.5-Decide, without instructions's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

45.0%
right on 135 of 300 problems wrong; chance 33% (dashed line)
wrong

The bald eagle does not need the bald eagle.

Correct answer: unknown · Model said: false (81.7% sure)

The problem and its proof

The bald eagle is big. The bald eagle is blue. The bald eagle is green. The bald eagle is rough. The bald eagle is young. Green, young people are rough. All young people are big.

No proof: neither the statement nor its negation follows.

right

The mouse is not green.

Correct answer: false · Model said: false (78.1% sure)

The problem and its proof

The lion likes the mouse. The mouse is green. If someone visits the mouse then the mouse sees the lion. If someone sees the lion then the lion sees the mouse. If someone likes the mouse then they visit the mouse.

Stated in the text: "The mouse is green."

right

The tiger is young.

Correct answer: true · Model said: true (49.9% sure)

The problem and its proof

The tiger is young. Young, nice things are round. All nice things are big. Young things are nice.

Stated in the text: "The tiger is young."

wrong

The squirrel is not red.

Correct answer: unknown · Model said: false (82.6% sure)

The problem and its proof

The bald eagle sees the lion. The lion chases the mouse. The lion sees the squirrel. The mouse chases the bald eagle. The mouse is red. The mouse sees the lion. The mouse visits the lion. The mouse visits the squirrel. The squirrel chases the bald eagle. The squirrel chases the mouse. The squirrel is green. If the lion visits the squirrel and the lion sees the squirrel then the lion chases the mouse. All nice people are green. If someone is kind then they chase the lion. If someone chases the lion and the lion chases the squirrel then they visit the mouse. If someone chases the mouse then the mouse is nice. If someone is green and they see the lion then the lion is nice. If someone sees the mouse then they are green. If someone is nice and they see the lion then the lion chases the squirrel. If someone chases the squirrel then they are kind.

No proof: neither the statement nor its negation follows.

Depth 1, open-world task

44.4%
right on 134 of 302 problems wrong; chance 33% (dashed line)
right

Fred is red.

Correct answer: true · Model said: true (58.5% sure)

paraphrased
The problem and its proof

Charlie is a really nice guy with a big round body, usually wearing red. Dave seems to be round. That guy Eric sure is nice. Fred is still young, which explains why he is round and cold, however, most people still think Fred is nice. Some cold people can be nice at times,and red at at other times. Rough people who are red in color and big are usally round in shape. I've noticed that nice people who have rough, green skin have a tendancy to be round. Having cold and red qualities means a person will also have a rough quality. Is a kind person if feeling cold and looking red then they are usually young. Red people, who are also round and young, are often quite blue as well. When someone is blue, nice and young they will also be as big as the sun.

  1. Fred is cold. + Fred is nice.If someone is cold and nice then they are red.Fred is red.
right

Bob is not kind.

Correct answer: false · Model said: false (93.7% sure)

paraphrased
The problem and its proof

Alan is a kind person and he is also often cold. Bob is cold but nice with red hair and dresses green. Eric is a kind person and he is also often cold. For being so cold, it's good Gary can remain nice. Every person I met that was cold and round was also red. Young people who are both nice and cold also tend to be rough. Is a kind person if feeling cold and looking red then they are usually young. If a person acts cold yet nice and green, they will be kind.

The proof that the statement's negation follows:

  1. Bob is nice. + Bob is green. + Bob is cold.If someone is nice and green and cold then they are kind.Bob is kind.
wrong

The lion likes the lion.

Correct answer: unknown · Model said: true (89.6% sure)

The problem and its proof

The lion is not rough. The mouse is nice. The tiger visits the mouse. If something is nice then it likes the lion. If something likes the lion then the lion likes the mouse. If something likes the mouse then it is not cold.

No proof: neither the statement nor its negation follows.

wrong

The tiger does not like the mouse.

Correct answer: true · Model said: false (57.8% sure)

The problem and its proof

The cat chases the tiger. The cat is big. The cat is kind. The cat is nice. The cat sees the mouse. The lion is not big. The lion is red. The lion likes the cat. The mouse chases the cat. The mouse chases the tiger. The mouse likes the lion. The mouse does not see the lion. The tiger is red. The tiger likes the cat. The tiger does not like the lion. If the lion chases the mouse then the mouse does not like the lion. If the mouse chases the tiger then the tiger does not like the mouse.

  1. The mouse chases the tiger.If the mouse chases the tiger then the tiger does not like the mouse.The tiger does not like the mouse.

Depth 2, open-world task

44.2%
right on 134 of 303 problems wrong; chance 33% (dashed line)
wrong

The dog visits the cow.

Correct answer: unknown · Model said: true (59.7% sure)

The problem and its proof

The cow eats the dog. The dog eats the cow. If the cow visits the dog then the cow eats the dog. If someone chases the dog then they visit the cow. If someone eats the dog then they chase the dog.

No proof: neither the statement nor its negation follows.

right

The lion visits the mouse.

Correct answer: true · Model said: true (60.5% sure)

The problem and its proof

The lion does not need the tiger. The mouse visits the tiger. The squirrel visits the mouse. The tiger eats the lion. If someone visits the mouse then they visit the lion. If someone visits the lion then the lion visits the mouse.

  1. The squirrel visits the mouse.If someone visits the mouse then they visit the lion.The squirrel visits the lion.
  2. If someone visits the lion then the lion visits the mouse.The lion visits the mouse.
wrong

Bob is not red.

Correct answer: unknown · Model said: false (84.8% sure)

paraphrased
The problem and its proof

The young person who is always feeling cold is named Bob. That guy Dave sure is nice. The young person who is always feeling cold is named Eric. Gary is kind and nice and looks green. If someone you know is big, cold, and young, you know they are also kind. A blue, cold natured person will be rough textured. Every single big person is a little green in some areas. Nice people that are very green and even round shaped will be very young. Someone who is a young age and looks like they are round are also red. Young people who feel cold and are still kind usually look rough. A nice person is inevitably round as well.

No proof: neither the statement nor its negation follows.

right

The mouse is green.

Correct answer: true · Model said: true (37.2% sure)

The problem and its proof

The mouse is rough. If someone is rough then they are young. If someone is round and young then they are blue. All young people are round. If someone is young and blue then they are green. Green people are blue. If someone is young then they are green.

  1. The mouse is rough.If someone is rough then they are young.The mouse is young.
  2. If someone is young then they are green.The mouse is green.

Depth 3, open-world task

44.2%
right on 134 of 303 problems wrong; chance 33% (dashed line)
wrong

The tiger needs the bald eagle.

Correct answer: false · Model said: true (79.6% sure)

The problem and its proof

The bald eagle eats the cat. The bald eagle likes the dog. The bald eagle likes the tiger. The bald eagle does not need the cat. The bald eagle does not need the dog. The cat likes the bald eagle. The dog is blue. The tiger eats the dog. The tiger is red. The tiger likes the bald eagle. The tiger needs the dog. If something is red and it needs the tiger then the tiger does not like the cat. If something needs the dog and it eats the dog then it likes the dog. If the tiger needs the dog and the dog is big then the tiger needs the bald eagle. If something needs the tiger then the tiger does not need the bald eagle. If something needs the dog then the dog is cold. All blue things are cold. If something is cold then it needs the tiger. If something likes the bald eagle and the bald eagle is big then it likes the tiger. If something likes the bald eagle and it does not need the bald eagle then the bald eagle is big.

The proof that the statement's negation follows:

  1. The tiger needs the dog.If something needs the dog then the dog is cold.The dog is cold.
  2. If something is cold then it needs the tiger.The dog needs the tiger.
  3. If something needs the tiger then the tiger does not need the bald eagle.The tiger does not need the bald eagle.
right

The mouse likes the cow.

Correct answer: true · Model said: true (57.3% sure)

The problem and its proof

The bald eagle eats the cow. The bald eagle is round. The bald eagle likes the cow. The cow eats the bald eagle. The cow likes the dog. The dog eats the bald eagle. The dog is rough. The dog likes the bald eagle. The dog visits the bald eagle. The dog visits the cow. The mouse visits the dog. If someone likes the mouse then they eat the cow. If someone likes the bald eagle then the bald eagle visits the mouse. If the mouse likes the dog then the mouse eats the bald eagle. If someone visits the bald eagle then the bald eagle is blue. If the mouse eats the bald eagle then the bald eagle is green. If someone is blue then they eat the mouse. If someone likes the cow and they visit the dog then they are blue. Round people are green. If someone eats the mouse then the mouse likes the cow.

  1. The dog visits the bald eagle.If someone visits the bald eagle then the bald eagle is blue.The bald eagle is blue.
  2. If someone is blue then they eat the mouse.The bald eagle eats the mouse.
  3. If someone eats the mouse then the mouse likes the cow.The mouse likes the cow.
right

Gary is not green.

Correct answer: false · Model said: false (85.8% sure)

paraphrased
The problem and its proof

Dave seems to be round. Fred is green and cold too. Gary, a kind young person is always feeling blue. Harry is a kind person and he is also often cold. People who are very nice and easily flush red are often wearing green concealer. Most young kind people tend to be red too. A kind person who looks blue because he is is cold is usually big in stature. Is a kind person if feeling cold and looking red then they are usually young. Any red sort of person is a nice person.

The proof that the statement's negation follows:

  1. Gary is young. + Gary is kind.If someone is young and kind then they are red.Gary is red.
  2. If someone is red then they are nice.Gary is nice.
  3. If someone is nice and red then they are green.Gary is green.
wrong

Eric is kind.

Correct answer: unknown · Model said: true (93.8% sure)

paraphrased
The problem and its proof

Dave is cold to his ex-wife. She knows he is round and red. He is kind to everyone else. Eric is a round and rough around the edges, and he is also big. Fred may be round, but he is also kind. Gary is a kind person and he is also often cold. Red people are almost always rough. Someone who is a young age and looks like they are round are also red. Count on anyone you meet who is big, round, and red also being kind. A kind person will certainly be young. When somebody is red, blue and round, you can bet they are also rough. I bet anyone who is round and kind is also pretty young.

No proof: neither the statement nor its negation follows.

Depth 4, open-world task

50.5%
right on 153 of 303 problems wrong; chance 33% (dashed line)
right

The lion needs the tiger.

Correct answer: true · Model said: true (56.9% sure)

The problem and its proof

The dog does not eat the mouse. The dog does not eat the tiger. The dog is round. The dog likes the lion. The dog likes the mouse. The lion eats the tiger. The mouse is not cold. The mouse is round. The mouse needs the dog. The tiger is blue. The tiger is rough. The tiger does not need the lion. If someone likes the mouse and they are blue then the mouse eats the dog. If someone likes the mouse and the mouse is blue then the mouse is rough. If someone is cold then they need the tiger. If someone eats the lion then they need the tiger. If someone likes the dog then they are rough. If someone is blue and they eat the lion then the lion is rough. If someone is rough then they eat the lion. If someone needs the tiger then they do not like the lion.

  1. The tiger is rough.If someone is rough then they eat the lion.The tiger eats the lion.
  2. + The tiger is blue.If someone is blue and they eat the lion then the lion is rough.The lion is rough.
  3. If someone is rough then they eat the lion.The lion eats the lion.
  4. If someone eats the lion then they need the tiger.The lion needs the tiger.
wrong

Bob is not furry.

Correct answer: unknown · Model said: false (67.1% sure)

The problem and its proof

Bob is quiet. Bob is white. Dave is not green. Dave is white. Erin is big. Erin is green. Erin is quiet. Erin is red. Erin is rough. Erin is white. Gary is rough. Gary is white. Furry, white things are red. Rough, white things are red. If something is green and rough then it is big. All green, white things are big. If something is red and quiet then it is green. All white, big things are furry. If something is red then it is quiet.

No proof: neither the statement nor its negation follows.

wrong

Gary is furry.

Correct answer: false · Model said: true (65.7% sure)

The problem and its proof

Bob is kind. Bob is red. Bob is round. Charlie is kind. Fiona is not green. Fiona is smart. Gary is round. All round people are quiet. All smart people are round. If someone is furry and not green then they are smart. Quiet people are smart. If someone is kind and quiet then they are not furry. If someone is smart then they are kind. If someone is kind and not furry then they are red.

The proof that the statement's negation follows:

  1. Gary is round.All round people are quiet.Gary is quiet.
  2. Quiet people are smart.Gary is smart.
  3. If someone is smart then they are kind.Gary is kind.
  4. If someone is kind and quiet then they are not furry.Gary is not furry.
right

The cat is not nice.

Correct answer: false · Model said: false (83.6% sure)

The problem and its proof

The bald eagle chases the cat. The bald eagle chases the dog. The bald eagle does not eat the cat. The bald eagle eats the dog. The cat does not chase the bald eagle. The cat eats the bald eagle. The cat is young. The cat sees the bald eagle. The cat sees the tiger. The dog chases the cat. The dog eats the bald eagle. The dog is blue. The tiger chases the bald eagle. The tiger chases the cat. If something is young then it does not chase the cat. If the cat chases the tiger then the tiger is nice. Nice things are big. If something chases the dog and the dog eats the bald eagle then it is nice. If something is big and it chases the cat then the cat chases the dog.

The proof that the statement's negation follows:

  1. The bald eagle chases the dog. + The dog eats the bald eagle.If something chases the dog and the dog eats the bald eagle then it is nice.The bald eagle is nice.
  2. Nice things are big.The bald eagle is big.
  3. + The bald eagle chases the cat.If something is big and it chases the cat then the cat chases the dog.The cat chases the dog.
  4. + The dog eats the bald eagle.If something chases the dog and the dog eats the bald eagle then it is nice.The cat is nice.

Depth 5, open-world task

44.6%
right on 129 of 289 problems wrong; chance 33% (dashed line)
wrong

Dave is not furry.

Correct answer: unknown · Model said: false (81.1% sure)

The problem and its proof

Anne is nice. Anne is red. Anne is rough. Bob is blue. Bob is cold. Bob is red. Dave is red. Dave is rough. Erin is red. Erin is rough. If Anne is red and Anne is blue then Anne is rough. Furry people are red. All rough, red people are blue. If someone is green and blue then they are nice. If Dave is nice then Dave is furry. If Erin is blue then Erin is furry. If someone is cold and blue then they are furry. If someone is rough and furry then they are green. Nice, green people are cold.

No proof: neither the statement nor its negation follows.

right

Charlie is cold.

Correct answer: false · Model said: false (80.4% sure)

The problem and its proof

Bob is big. Bob is kind. Bob is nice. Charlie is kind. Charlie is young. Erin is young. Harry is furry. If Charlie is kind and Charlie is young then Charlie is furry. If someone is furry and rough then they are big. If someone is young and furry then they are rough. If someone is furry and big then they are nice. If someone is kind and nice then they are not cold.

The proof that the statement's negation follows:

  1. Charlie is kind. + Charlie is young.If Charlie is kind and Charlie is young then Charlie is furry.Charlie is furry.
  2. + Charlie is young.If someone is young and furry then they are rough.Charlie is rough.
  3. If someone is furry and rough then they are big.Charlie is big.
  4. If someone is furry and big then they are nice.Charlie is nice.
  5. + Charlie is kind.If someone is kind and nice then they are not cold.Charlie is not cold.
wrong

The tiger is green.

Correct answer: unknown · Model said: true (42.3% sure)

The problem and its proof

The cat is green. The cat sees the dog. The cat visits the dog. The dog eats the mouse. The dog is young. The dog sees the tiger. The dog visits the cat. The dog visits the mouse. The mouse sees the dog. The mouse visits the cat. The mouse visits the dog. The tiger eats the cat. The tiger is nice. The tiger sees the dog. The tiger sees the mouse. The tiger visits the dog. If someone eats the mouse then they are green. If someone is young and they see the dog then they are green. If someone eats the dog then they see the dog. If the tiger eats the dog and the dog visits the mouse then the tiger is young. If someone visits the cat and the cat is nice then the cat sees the dog. If someone sees the dog and the dog is green then they see the cat. If someone is rough then they eat the dog. All green people are rough. If someone eats the dog then the dog visits the mouse.

No proof: neither the statement nor its negation follows.

right

The bald eagle is not cold.

Correct answer: false · Model said: false (52.7% sure)

The problem and its proof

The bald eagle likes the squirrel. The bald eagle needs the cat. The cat eats the squirrel. The cat is young. The cat likes the cow. The cat likes the squirrel. The cow eats the cat. The cow is blue. The cow is round. The cow is young. The cow likes the squirrel. The cow needs the bald eagle. The cow needs the squirrel. The squirrel eats the bald eagle. The squirrel is rough. The squirrel needs the cat. If someone likes the cat then they are cold. If someone likes the cat and they are rough then the cat likes the squirrel. If someone is rough then they like the cat. If someone eats the bald eagle and they are cold then the bald eagle is rough. If someone is cold and they eat the cow then the cow likes the cat.

The proof that the statement's negation follows:

  1. The squirrel is rough.If someone is rough then they like the cat.The squirrel likes the cat.
  2. If someone likes the cat then they are cold.The squirrel is cold.
  3. + The squirrel eats the bald eagle.If someone eats the bald eagle and they are cold then the bald eagle is rough.The bald eagle is rough.
  4. If someone is rough then they like the cat.The bald eagle likes the cat.
  5. If someone likes the cat then they are cold.The bald eagle is cold.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

52.3%
right on 157 of 300 problems wrong; chance 50% (dashed line)
right

The mouse chases the tiger.

Correct answer: true · Model said: true (86.4% sure)

The problem and its proof

The cow is green. The mouse chases the tiger. The rabbit needs the cow. The tiger chases the rabbit. If something chases the mouse then the mouse is blue. If something is green and it does not visit the rabbit then it chases the mouse.

Stated in the text: "The mouse chases the tiger."

right

Anne is not round.

Correct answer: false · Model said: false (98.5% sure)

The problem and its proof

Anne is nice. Anne is round. Dave is big. Dave is quiet. Dave is round. Erin is green. Erin is quiet. Harry is big. Harry is quiet. Harry is round. All furry, round people are nice. Furry people are nice. All nice people are rough. All green, nice people are furry. Round, big people are furry. If Erin is furry then Erin is big. Round, rough people are big. Quiet people are green. Furry people are quiet.

Stated in the text: "Anne is round."

wrong

Gary is green.

Correct answer: false · Model said: true (96.7% sure)

paraphrased
The problem and its proof

Bob is green to the job, which is why he turned blue from being so cold. Green, big and red are qualities which all describe Charlie. Dave is a kind person and he is also often cold. Gary is a really nice guy with a big round body, usually wearing red. A person with big, round, kind face appears to be rough around the edges because they are naive. A person feeling blue because of their big size will always be cold. Do not fear big cold people for they are often kind. A person that is very big and also the color red they will also be blue. Cold and red people are always kind to others. A person who is cold and red and blue is big.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Dave is rough.

Correct answer: false · Model said: true (53.7% sure)

paraphrased
The problem and its proof

The young person who is always feeling cold is named Dave. Thankfully Eric is nice. People have realized that he is big and rough. Since he is young he has tendency to become red when they mention it. Gary seems to be round. Harry seems to be round. If someone plays rough and is nice and round, they will be big. A big round young person is often blue. Nice people who feel blue often look green when ill. A nice person is inevitably round as well. If a person is both cold and round, that person is also someone who is red. If you know someone who is nice, green, and big, you know a young person. A person who is cold and red and blue is big.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

54.3%
right on 163 of 300 problems wrong; chance 50% (dashed line)
right

Anne is not smart.

Correct answer: false · Model said: false (99.2% sure)

The problem and its proof

Anne is young. Charlie is young. Dave is rough. Fiona is nice. If Charlie is furry and Charlie is not rough then Charlie is big. If something is young and not rough then it is smart. All green, rough things are smart. If Anne is smart then Anne is big. If Fiona is nice then Fiona is furry. Big things are nice. Rough things are nice. If something is young and not big then it is nice.

  1. Anne is young.If something is young and not rough then it is smart.Anne is smart.
wrong

Harry is not cold.

Correct answer: true · Model said: false (65.5% sure)

paraphrased
The problem and its proof

Alan is a really nice guy with a big round body, usually wearing red. The young person who is always feeling cold is named Bob. Eric may be round, but he is also kind. Harry may be round, but he is also kind. A person feeling blue because of their big size will always be cold. People who are feeling blue and green are said to be red. A person that is very big and also the color red they will also be blue. If someone is cold and blue, you will find that they are also quite young.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Charlie is not round.

Correct answer: true · Model said: false (99.1% sure)

The problem and its proof

Anne is red. Anne is rough. Charlie is cold. Charlie is kind. Charlie is quiet. Fiona is kind. Fiona is round. Gary is furry. Gary is kind. Gary is red. Gary is round. Furry people are cold. Red people are cold. All round, red people are kind. Round people are kind. Round, kind people are quiet. If someone is rough and furry then they are round. All round, rough people are red. If someone is cold and rough then they are furry. All cold people are furry.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The squirrel does not see the tiger.

Correct answer: false · Model said: false (98.2% sure)

The problem and its proof

The bear needs the tiger. The cow is big. The squirrel is cold. The tiger needs the cow. If someone is nice and they do not eat the bear then they are cold. If the squirrel is not green then the squirrel sees the tiger.

  1. If the squirrel is not green then the squirrel sees the tiger.The squirrel sees the tiger.

Depth 2, closed-world task

54.0%
right on 162 of 300 problems wrong; chance 50% (dashed line)
right

Gary is quiet.

Correct answer: true · Model said: true (75.8% sure)

The problem and its proof

Anne is blue. Dave is round. Gary is big. If something is red and not round then it is quiet. If something is big then it is red. If something is quiet then it is blue.

  1. Gary is big.If something is big then it is red.Gary is red.
  2. If something is red and not round then it is quiet.Gary is quiet.
right

Bob is not blue.

Correct answer: false · Model said: false (93.1% sure)

The problem and its proof

Bob is cold. Bob is furry. Bob is rough. Bob is round. Bob is young. Dave is blue. Dave is rough. Erin is cold. Erin is furry. Erin is quiet. Erin is rough. Erin is round. Erin is young. Gary is round. Quiet, young things are blue. Round things are quiet. All furry things are young. All round, quiet things are rough. Blue, quiet things are rough. Rough things are furry. Blue things are furry. If Gary is young and Gary is quiet then Gary is rough.

  1. Bob is round.Round things are quiet.Bob is quiet.
  2. + Bob is young.Quiet, young things are blue.Bob is blue.
wrong

Bob is not cold.

Correct answer: false · Model said: true (53.1% sure)

paraphrased
The problem and its proof

Bob discovered that even though he was round, rough, and nice and green, he was actually big. When Charlie walks around the neighborhood being nice and kind, the closer you get to him you can tell he is blue and red. Eric seems to be round. Harry seems to be round. Not a single young person is not also kind. Rough, cold people are blue. Is a kind person if feeling cold and looking red then they are usually young. Rough people who look red are cold. A nice and green man or woman is also red in color.

  1. Bob is nice. + Bob is green.If someone is nice and green then they are red.Bob is red.
  2. + Bob is rough.If someone is red and rough then they are cold.Bob is cold.
wrong

Bob is round.

Correct answer: false · Model said: true (98.2% sure)

paraphrased
The problem and its proof

Big rough Alan is round and rather green. Bob might be rough and red but he's actually very kind. Gary may be round, but he is also kind. Green, big and red are qualities which all describe Harry. A rather big person is a red person. If someone has a kind disposition and looks rather round and rough then you'll find that they're actually quite young. Anyone having rough, cold and green qualities will also have a big quality. Count on anyone you meet who is big, round, and red also being kind. Young and red people look round.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

53.3%
right on 160 of 300 problems wrong; chance 50% (dashed line)
wrong

Harry is not red.

Correct answer: true · Model said: false (89.0% sure)

The problem and its proof

Erin is green. Erin is white. Fiona is green. Fiona is nice. Harry is cold. Harry is green. Harry is smart. Red things are blue. If Erin is cold then Erin is white. Nice things are red. If Harry is blue then Harry is nice. If something is smart and cold then it is green. All green, white things are nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Fiona is not white.

Correct answer: false · Model said: false (90.5% sure)

The problem and its proof

Fiona is quiet. All white, green things are quiet. If something is quiet then it is green. All smart things are white. Green things are smart. All white, red things are furry. Quiet, blue things are red. All red, blue things are furry. All blue things are quiet.

  1. Fiona is quiet.If something is quiet then it is green.Fiona is green.
  2. Green things are smart.Fiona is smart.
  3. All smart things are white.Fiona is white.
right

The lion does not see the squirrel.

Correct answer: false · Model said: false (99.0% sure)

The problem and its proof

The dog chases the lion. The lion chases the squirrel. The squirrel eats the lion. The tiger is blue. If someone eats the lion then they see the dog. If someone chases the dog then they see the squirrel. If someone sees the dog and they eat the lion then the lion chases the dog.

  1. The squirrel eats the lion.If someone eats the lion then they see the dog.The squirrel sees the dog.
  2. + The squirrel eats the lion.If someone sees the dog and they eat the lion then the lion chases the dog.The lion chases the dog.
  3. If someone chases the dog then they see the squirrel.The lion sees the squirrel.
wrong

The mouse is not big.

Correct answer: true · Model said: false (97.1% sure)

The problem and its proof

The bear chases the mouse. The bear is big. The bear is red. The bear likes the cow. The bear likes the mouse. The bear needs the cow. The bear needs the mouse. The cow chases the bear. The cow chases the mouse. The cow is big. The cow is blue. The cow is green. The cow needs the bear. The mouse is blue. The mouse needs the bear. The mouse needs the cow. If something likes the bear then the bear chases the cow. If the mouse is blue then the mouse likes the bear. If the mouse is blue and the mouse chases the cow then the cow needs the mouse. If the cow needs the mouse and the cow chases the bear then the bear is blue. If something likes the bear and the bear is blue then it is big. If something needs the mouse and it chases the bear then the bear chases the cow.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 4, closed-world task

49.7%
right on 149 of 300 problems wrong; chance 50% (dashed line)
wrong

Anne is smart.

Correct answer: false · Model said: true (93.4% sure)

The problem and its proof

Anne is quiet. Bob is rough. Round people are blue. If Anne is blue then Anne is smart. If someone is blue and smart then they are young. If someone is blue then they are round. Rough, round people are smart. If someone is green then they are rough. Green people are rough. Rough people are round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cow visits the cow.

Correct answer: true · Model said: true (90.9% sure)

The problem and its proof

The bear is rough. The bear needs the dog. The bear visits the cat. The bear visits the dog. The cat is blue. The cat likes the cow. The cow is rough. The cow likes the bear. The dog needs the bear. The dog visits the cat. All blue people are big. If someone visits the cow then they need the cow. If someone is nice then they like the cow. If someone likes the cat and they are nice then they are big. If someone is blue then they visit the cow. If the cat needs the cow then the cow is blue.

  1. The cat is blue.If someone is blue then they visit the cow.The cat visits the cow.
  2. If someone visits the cow then they need the cow.The cat needs the cow.
  3. If the cat needs the cow then the cow is blue.The cow is blue.
  4. If someone is blue then they visit the cow.The cow visits the cow.
right

The cow does not visit the cow.

Correct answer: false · Model said: false (99.2% sure)

The problem and its proof

The bear is rough. The bear needs the dog. The bear visits the cat. The bear visits the dog. The cat is blue. The cat likes the cow. The cow is rough. The cow likes the bear. The dog needs the bear. The dog visits the cat. All blue people are big. If someone visits the cow then they need the cow. If someone is nice then they like the cow. If someone likes the cat and they are nice then they are big. If someone is blue then they visit the cow. If the cat needs the cow then the cow is blue.

  1. The cat is blue.If someone is blue then they visit the cow.The cat visits the cow.
  2. If someone visits the cow then they need the cow.The cat needs the cow.
  3. If the cat needs the cow then the cow is blue.The cow is blue.
  4. If someone is blue then they visit the cow.The cow visits the cow.
wrong

Erin is green.

Correct answer: false · Model said: true (67.2% sure)

The problem and its proof

Dave is young. Erin is blue. Fiona is quiet. If someone is white and red then they are blue. All quiet, green people are red. All white people are quiet. All young people are white. If Erin is red and Erin is quiet then Erin is blue. All white people are red. All white people are green. If someone is quiet then they are young.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 5, closed-world task

62.0%
right on 186 of 300 problems wrong; chance 50% (dashed line)
right

The bald eagle is red.

Correct answer: true · Model said: true (54.0% sure)

The problem and its proof

The bald eagle is young. The bald eagle likes the cow. The bald eagle sees the mouse. The cow is big. The cow likes the bald eagle. The cow likes the mouse. The cow likes the squirrel. The cow needs the bald eagle. The mouse is red. The mouse sees the bald eagle. The squirrel is young. If someone needs the mouse then they see the cow. If someone needs the cow then they are big. If someone sees the squirrel then they are big. If someone likes the bald eagle then the bald eagle needs the cow. If someone sees the cow then they are red. If someone is big then they need the mouse. If someone is green then they see the bald eagle.

  1. The cow likes the bald eagle.If someone likes the bald eagle then the bald eagle needs the cow.The bald eagle needs the cow.
  2. If someone needs the cow then they are big.The bald eagle is big.
  3. If someone is big then they need the mouse.The bald eagle needs the mouse.
  4. If someone needs the mouse then they see the cow.The bald eagle sees the cow.
  5. If someone sees the cow then they are red.The bald eagle is red.
wrong

Fiona is not quiet.

Correct answer: true · Model said: false (97.8% sure)

The problem and its proof

Bob is big. Bob is cold. Bob is green. Bob is round. Fiona is big. Gary is big. Gary is cold. Harry is big. Harry is cold. Harry is furry. Harry is green. Harry is kind. Harry is quiet. Harry is round. If Fiona is quiet and Fiona is not cold then Fiona is kind. If something is big and round then it is furry. Cold things are quiet. If something is furry then it is quiet. All kind, furry things are quiet. If something is green then it is round. If something is quiet and big then it is green. If Gary is furry then Gary is kind.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Erin is rough.

Correct answer: false · Model said: true (56.1% sure)

The problem and its proof

Dave is blue. Dave is rough. Dave is young. Erin is blue. Erin is round. Erin is young. Fiona is rough. Gary is green. Gary is nice. Gary is young. Rough things are round. Green, nice things are quiet. Round, rough things are green. All quiet things are green. All green things are nice. All quiet things are rough. All rough, nice things are quiet. All quiet, green things are rough. If something is nice and quiet then it is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The dog is red.

Correct answer: true · Model said: true (55.5% sure)

The problem and its proof

The bear is big. The bear sees the cat. The cat chases the cow. The cat likes the bear. The cow likes the cat. The cow sees the bear. The dog sees the cow. If the cat likes the bear and the cat chases the dog then the dog likes the cat. If someone likes the cat then the cat sees the bear. If someone sees the bear then they chase the dog. If the cat is red then the cat likes the bear. If someone chases the bear then the bear sees the dog. If someone likes the cat then they chase the cow. If someone chases the cow and the cow chases the dog then they are red. If the cow is green then the cow chases the cat. If someone likes the dog and they see the bear then the bear is young.

  1. The cow likes the cat.If someone likes the cat then the cat sees the bear.The cat sees the bear.
  2. If someone sees the bear then they chase the dog.The cat chases the dog.
  3. + The cat likes the bear.If the cat likes the bear and the cat chases the dog then the dog likes the cat.The dog likes the cat.
  4. If someone likes the cat then they chase the cow.The dog chases the cow.
  5. + The cow sees the bear.If someone sees the bear then they chase the dog.The cow chases the dog.
  6. If someone chases the cow and the cow chases the dog then they are red.The dog is red.

Where each one is better

Open world, Jev vs GLiNER2.5-Decide. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GLiNER2.5-Decide is clearly ahead on unknown answers (+25.8 points).

Closed world, Jev vs GLiNER2.5-Decide. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5 and true answers. GLiNER2.5-Decide is clearly ahead on false answers (+9.7 points).

Open world, Jev vs GLiNER2.5-Decide, without instructions. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers, true answers and unknown answers. GLiNER2.5-Decide, without instructions is not clearly ahead on any depth or answer class.

Closed world, Jev vs GLiNER2.5-Decide, without instructions. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GLiNER2.5-Decide, without instructions is not clearly ahead on any depth or answer class.

A slice counts as clearly ahead when its paired 95% interval excludes zero.

True, false and unknown, open world

By correct answer, Open world
Correct answerItemsJevGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsBest constant
false60588.3% 85.5–90.90.0% 0.0–0.069.4% 65.3–72.9one answer
true60588.8% 86.1–91.10.0% 0.0–0.065.3% 61.7–69.1one answer
unknown59074.2% 70.7–77.6100.0% 100.0–100.00.7% 0.2–1.5one answer

True, false and unknown, closed world

By correct answer, Closed world
Correct answerItemsJevGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsBest constant
false90088.9% 86.9–90.998.6% 97.8–99.261.1% 58.0–64.3one answer
true90089.7% 87.7–91.62.1% 1.2–3.147.4% 44.2–50.6one answer

Paraphrased vs templated rules, open world

By wording, Open world
WordingItemsJevGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsBest constant
templated1,29886.1% 84.2–87.934.5% 32.0–37.038.2% 35.7–40.834.5%
paraphrased (NatLang)50278.1% 74.5–82.128.3% 24.5–32.164.3% 60.4–68.335.9%

Paraphrased vs templated rules, closed world

By wording, Closed world
WordingItemsJevGLiNER2.5-DecideGLiNER2.5-Decide, without instructionsBest constant
templated1,30694.0% 92.8–95.350.2% 47.5–52.852.5% 49.8–55.250.2%
paraphrased (NatLang)49476.7% 73.1–80.450.6% 46.2–54.958.9% 54.7–63.250.6%

Answer by answer: Jev, GLiNER2.5-Decide and GLiNER2.5-Decide, without instructions on the same problems

Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
GLiNER2.5-Decide, open-world task
Correct ↓ · said →truefalseunknown
true00605
false00605
unknown00590
GLiNER2.5-Decide, without instructions, open-world task
Correct ↓ · said →truefalseunknown
true3952100
false1854200
unknown2663204

Jev said true on 37% of problems; the correct answer is true on 34%. GLiNER2.5-Decide said unknown on 100% of problems; the correct answer is unknown on 33%. GLiNER2.5-Decide, without instructions said false on 53% of problems; the correct answer is false on 34%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
GLiNER2.5-Decide, closed-world task
Correct ↓ · said →truefalse
true19881
false13887
GLiNER2.5-Decide, without instructions, closed-world task
Correct ↓ · said →truefalse
true427473
false350550

Jev said true on 50% of problems; the correct answer is true on 50%. GLiNER2.5-Decide said false on 98% of problems; the correct answer is false on 50%. GLiNER2.5-Decide, without instructions said false on 57% of problems; the correct answer is false on 50%.

Where GLiNER2.5-Decide and GLiNER2.5-Decide, without instructions err

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

GLiNER2.5-Decide, open-world task

GLiNER2.5-Decide, open-world task
Correct ↓ · said →truefalseunknownRecall
true006050.0%
false006050.0%
unknown00590100.0%

GLiNER2.5-Decide answered unknown 100.0% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

GLiNER2.5-Decide, closed-world task

GLiNER2.5-Decide, closed-world task
Correct ↓ · said →truefalseRecall
true198812.1%
false1388798.6%

GLiNER2.5-Decide answered false 98.2%, true 1.8% of the time; the correct answers are false 50.0%, true 50.0%.

GLiNER2.5-Decide at depth 0 and depth 5

GLiNER2.5-Decide, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true00100
false00100
unknown00100
GLiNER2.5-Decide, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true00101
false00101
unknown0087

Open-world task. At depth 0, GLiNER2.5-Decide was right on 33% of problems (the green diagonal); at depth 5, on 30%. The answer whose share moved most is true: 0% of its answers at depth 0, 0% at depth 5, against 33% and 35% of the correct answers.

The closed-world task at depth 0 and depth 5
GLiNER2.5-Decide, closed-world task, depth 0
Correct ↓ · said →truefalse
true1149
false2148
GLiNER2.5-Decide, closed-world task, depth 5
Correct ↓ · said →truefalse
true0150
false0150

Closed-world task. At depth 0, GLiNER2.5-Decide was right on 50% of problems (the green diagonal); at depth 5, on 50%. The answer whose share moved most is false: 99% of its answers at depth 0, 100% at depth 5, against 50% and 50% of the correct answers.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

GLiNER2.5-Decide, without instructions, open-world task

GLiNER2.5-Decide, without instructions, open-world task
Correct ↓ · said →truefalseunknownRecall
true395210065.3%
false185420069.4%
unknown26632040.7%

GLiNER2.5-Decide, without instructions answered false 52.8%, true 47.0%, unknown 0.2% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

GLiNER2.5-Decide, without instructions, closed-world task

GLiNER2.5-Decide, without instructions, closed-world task
Correct ↓ · said →truefalseRecall
true42747347.4%
false35055061.1%

GLiNER2.5-Decide, without instructions answered false 56.8%, true 43.2% of the time; the correct answers are false 50.0%, true 50.0%.

GLiNER2.5-Decide, without instructions at depth 0 and depth 5

GLiNER2.5-Decide, without instructions, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true67330
false36640
unknown43534
GLiNER2.5-Decide, without instructions, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true60410
false32690
unknown29580

Open-world task. At depth 0, GLiNER2.5-Decide, without instructions was right on 45% of problems (the green diagonal); at depth 5, on 45%. The answer whose share moved most is false: 50% of its answers at depth 0, 58% at depth 5, against 33% and 35% of the correct answers.

The closed-world task at depth 0 and depth 5
GLiNER2.5-Decide, without instructions, closed-world task, depth 0
Correct ↓ · said →truefalse
true7278
false6585
GLiNER2.5-Decide, without instructions, closed-world task, depth 5
Correct ↓ · said →truefalse
true7179
false35115

Closed-world task. At depth 0, GLiNER2.5-Decide, without instructions was right on 52% of problems (the green diagonal); at depth 5, on 62%. The answer whose share moved most is true: 46% of its answers at depth 0, 35% at depth 5, against 50% and 50% of the correct answers.

GLiNER2.5-Decide and GLiNER2.5-Decide, without instructions across every breakdown

For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownGLiNER2.5-DecideGLiNER2.5-Decide, without instructions
Proof depthdepth 5 30.1% · depth 0 33.3% (3.2 points)depth 2 44.2% · depth 4 50.5% (6.3 points)
Gold answerfalse 0.0% · unknown 100.0% (100.0 points)unknown 0.7% · false 69.4% (68.7 points)
Theory kindrelation 31.3% · attribute 33.8% (2.5 points)relation 43.3% · attribute 47.0% (3.7 points)
Negation in the theorywith negation 30.0% · without negation 35.5% (5.5 points)with negation 31.0% · without negation 59.4% (28.4 points)
Negated statementplain statement 32.4% · negated statement 33.2% (0.8 points)plain statement 43.4% · negated statement 47.6% (4.2 points)
Question strategyinv-proof 0.0% · inv-random 100.0% (100.0 points)inv-rconc 0.0% · inv-proof 69.4% (69.4 points)
Paraphrased rulesparaphrased (NatLang) 28.3% · templated 34.5% (6.2 points)templated 38.2% · paraphrased (NatLang) 64.3% (26.1 points)
Theory's deepest proof5 23.5% · 0 60.6% (37.1 points)0 18.3% · 4 61.7% (43.4 points)
Theory length110+ 26.1% · 0-49 39.7% (13.6 points)50-79 36.8% · 110+ 53.7% (16.9 points)
Number of rules3-5 24.0% · 8+ 40.7% (16.8 points)8+ 32.0% · 3-5 61.2% (29.2 points)
Number of facts13+ 23.9% · 0-3 42.8% (18.9 points)0-3 32.9% · 8-12 53.6% (20.6 points)
Proof size2 0.0% · 0-1 74.7% (74.7 points)0-1 17.1% · 5+ 73.3% (56.2 points)

Exactly what GLiNER2.5-Decide and GLiNER2.5-Decide, without instructions were asked

One open-world problem at depth 0, as each model received it. The correct answer is unknown.

The bear is green. Green things are not young.

Statement: The bear is not big.
Jevright answered unknown (100.0%); correct answer unknown

POST https://api.typesafe.ai/v1/systemone

Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%. Its separate confidence: 100.0%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "jev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "jev-1.13.0",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 1,
      "probabilities": {
        "false": 0,
        "true": 0,
        "unknown": 1
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 460,
    "output_tokens": 38
  }
}
GLiNER2.5-Decideright answered unknown (92.1%); correct answer unknown

Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)

Fastino's hosted API for this model accepts a text and a list of answer labels, with no field for a question or for what each label means (the open-source package also accepts a short question as a prompt and a description per label; passed that way, our instructions made the model give the same answer to every problem). So the question and every option's meaning go into the text, after the problem, word for word as GPT-6 Luna received them, without Luna's line about replying in JSON. The labels are the three answers.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

Options:
- true: The statement follows from the facts and rules.
- false: The negation of the statement follows from the facts and rules.
- unknown: Neither the statement nor its negation follows from the facts and rules.

The answers it can give

true
false
unknown

Its probability for each answer: false 3.7%, true 4.2%, unknown 92.1%. Its separate confidence: 92.1%.

The full request and response, exactly as sent and recorded

Request

{
  "call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
  "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.",
  "tasks": {
    "Decision": [
      "true",
      "false",
      "unknown"
    ]
  }
}

Response (the saved record)

{
  "model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 0.9209111332893372,
      "probabilities": {
        "false": 0.036921169608831406,
        "true": 0.04216771200299263,
        "unknown": 0.9209111332893372
      },
      "type": "choice"
    }
  },
  "usage": null
}
GLiNER2.5-Decide, without instructionswrong answered false (49.2%); correct answer unknown

Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)

The same model, asked without the instructions: the text is only the problem (the facts and rules, then the statement), and the labels are the three answers. Nothing tells the model what true, false or unknown mean. This is the bare form Fastino's hosted API documents.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The answers it can give

true
false
unknown

Its probability for each answer: false 49.2%, true 31.3%, unknown 19.5%. Its separate confidence: 49.2%.

The full request and response, exactly as sent and recorded

Request

{
  "call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
  "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.",
  "tasks": {
    "Decision": [
      "true",
      "false",
      "unknown"
    ]
  }
}

Response (the saved record)

{
  "model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
  "answers": {
    "Decision": {
      "choice": "false",
      "confidence": 0.4924229383468628,
      "probabilities": {
        "false": 0.4924229383468628,
        "true": 0.3129580020904541,
        "unknown": 0.1946191042661667
      },
      "type": "choice"
    }
  },
  "usage": null
}

Consistency and calibration

GLiNER2.5-Decide

Asked every open world question a second time, GLiNER2.5-Decide gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 32.8% the first time and 32.8% the second.

Asked every closed world question a second time, GLiNER2.5-Decide gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 50.3% the first time and 50.3% the second.

Calibration on the open world task: GLiNER2.5-Decide's probability for its own answer averaged 93.0% against 32.8% accuracy; expected calibration error 60.2 points.

Calibration on the closed world task: GLiNER2.5-Decide's probability for its own answer averaged 64.6% against 50.3% accuracy; expected calibration error 14.3 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.51 on the open world task against Jev's 0.85; 0.53 on the closed world task against Jev's 0.86.

GLiNER2.5-Decide, without instructions

Asked every open world question a second time, GLiNER2.5-Decide, without instructions gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 45.5% the first time and 45.5% the second.

Asked every closed world question a second time, GLiNER2.5-Decide, without instructions gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 54.3% the first time and 54.3% the second.

Calibration on the open world task: GLiNER2.5-Decide, without instructions's probability for its own answer averaged 68.8% against 45.5% accuracy; expected calibration error 23.3 points.

Calibration on the closed world task: GLiNER2.5-Decide, without instructions's probability for its own answer averaged 87.3% against 54.3% accuracy; expected calibration error 33.1 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.60 on the open world task against Jev's 0.85; 0.55 on the closed world task against Jev's 0.86.

Size, hardware and speed

ModelKindParametersRan onMedian time per decisionMemory peak
Jevhosted decision modelundisclosedvendor's servers189 msnot applicable (hosted)
GLiNER2.5-Decideopen decision model340Mour laptop (Apple M1 Max, 32 GB)150 ms4.4 GB
GLiNER2.5-Decide, without instructionsopen decision model340Mour laptop (Apple M1 Max, 32 GB)89 ms4.5 GB

Times are open-world, one request at a time. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.

GLiNER2.5-Decide

Parameters
340M
Architecture
GLiNER2 schema classifier on a DeBERTa-v3-large encoder; answers in one pass ("does not reason", model card); asked with the full question and option descriptions in the message, task and labels as the answer form
Weights
1.95 GB (model.safetensors, fastino/GLiNER2.5-Decide@5a7adf7)
Precision
float32, as loaded by gliner2 (PyTorch)
Where it ran
this machine, gliner2 2.0.0 (local extra), PyTorch on MPS
Maker's stated hardware
"CPU or GPU"; 38.3 ms p50 on an NVIDIA V100, 167.3 ms on a 48-vCPU Xeon, short inputs (Fastino)
Memory measured
3.9 GB in use, 4.4 GB peak
Price
open weights (Apache 2.0); Fastino also hosts it at $0.03 or $0.15 per million input tokens (its pricing page and model catalog disagree)
Time per decision, open world
median 150 ms, 90th percentile 197 ms, on a laptop
Time per decision, closed world
median 141 ms, 90th percentile 183 ms, on a laptop
Source
huggingface.co/fastino/GLiNER2.5-Decide

GLiNER2.5-Decide, without instructions

Parameters
340M
Architecture
GLiNER2 schema classifier on a DeBERTa-v3-large encoder; answers in one pass ("does not reason", model card); asked in the bare hosted-API form (the text with task and labels only), without the question's instructions or option descriptions
Weights
1.95 GB (model.safetensors, fastino/GLiNER2.5-Decide@5a7adf7)
Precision
float32, as loaded by gliner2 (PyTorch)
Where it ran
this machine, gliner2 2.0.0 (local extra), PyTorch on MPS
Maker's stated hardware
"CPU or GPU"; 38.3 ms p50 on an NVIDIA V100, 167.3 ms on a 48-vCPU Xeon, short inputs (Fastino)
Memory measured
4.5 GB in use, 4.5 GB peak
Price
open weights (Apache 2.0); Fastino also hosts it at $0.03 or $0.15 per million input tokens (its pricing page and model catalog disagree)
Time per decision, open world
median 89 ms, 90th percentile 120 ms, on a laptop
Time per decision, closed world
median 87 ms, 90th percentile 116 ms, on a laptop
Source
huggingface.co/fastino/GLiNER2.5-Decide

The models

Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

GLiNER2.5-Decide GLiNER2.5-Decide is an open decision model made by Fastino Labs: a 340M-parameter schema classifier on a DeBERTa-v3-large encoder, released under Apache 2.0. It answers in one pass; its model card says it "does not reason." We ran it on our own laptop with Fastino's gliner2 package, PyTorch on Apple's GPU, with everything every other model gets: the problem, then the question's instructions and each option with its description, in the message, and the options as its answer labels. On the same inputs, our copy gave Fastino's hosted model's answers.

GLiNER2.5-Decide, without instructions The same GLiNER2.5-Decide checkpoint asked in the bare form Fastino's hosted API documents: the problem text, which ends with the statement, and the answer options as labels, without the question's instructions or the option descriptions. It is shown beside the full form to show what those add.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-16, from the harness's working tree at commit b324cd3 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.