HardDecisions

Comparison · GLiNER2.5-Decide vs Jev

Jev vs GLiNER2.5-Decide: a hosted decision model against Fastino's open one

GLiNER2.5-Decide is Fastino's open 340M-parameter decision model, released under Apache 2.0, that answers in one pass. We ran it on a laptop against Jev on the same 3,600 problems.

GLiNER2.5-Decide gave nearly the same answer to every problem: false to all 1,800 closed-world problems and unknown to 93% of the open-world ones, so its accuracy is what always giving the most common answer would score, at every depth. Its model card says it is a specialist for operational decisions such as intent, routing and moderation, and that it does not reason; this benchmark asks for chained reasoning.

Overall: Jev and GLiNER2.5-Decide

Open world: true, false or unknown

  • Jev82.2 to 85.5 (95%), 1,800 items83.8%
  • GLiNER2.5-Decide33.1 to 37.5 (95%), 1,800 items35.2%

Jev minus GLiNER2.5-Decide: +48.7 points on the same 1,800 items, 95% interval +45.8 to +51.7: Jev is clearly ahead.

Closed world: true or false

  • Jev87.8 to 90.7 (95%), 1,800 items89.3%
  • GLiNER2.5-Decide47.7 to 52.3 (95%), 1,800 items50.0%

Jev minus GLiNER2.5-Decide: +39.3 points on the same 1,800 items, 95% interval +36.5 to +42.0: Jev is clearly ahead.

Jev vs GLiNER2.5-Decide by proof depth

Each point is about 300 problems that need that many inference steps, with its 95% interval.

Open world: true, false or unknown
  • Jev
  • GLiNER2.5-Decide
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
  • GLiNER2.5-Decide
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Jev minus GLiNER2.5-Decide by depth, open world

depth 0300 items
+62.3
depth 1302 items
+53.0
depth 2303 items
+49.5
depth 3303 items
+43.6
depth 4303 items
+33.7
depth 5289 items
+50.2

Jev minus GLiNER2.5-Decide by depth, closed world

depth 0300 items
+49.3
depth 1300 items
+43.7
depth 2300 items
+42.0
depth 3300 items
+35.0
depth 4300 items
+26.3
depth 5300 items
+39.3

Paired differences: positive means Jev was more accurate on the same items.

Accuracy by depth, open world

By depth, Open world
Proof depthItemsJevGLiNER2.5-DecideBest constant
depth 030097.7% 96.0–99.335.3% 30.0–41.033.3%
depth 130287.7% 83.8–91.434.8% 29.1–40.433.4%
depth 230384.8% 80.2–88.435.3% 30.0–40.633.3%
depth 330379.9% 75.2–84.536.3% 30.7–41.633.3%
depth 430371.9% 66.7–77.238.3% 32.7–43.933.3%
depth 528981.0% 76.5–85.530.8% 25.6–36.034.9%

Accuracy by depth, closed world

By depth, Closed world
Proof depthItemsJevGLiNER2.5-DecideBest constant
depth 030099.3% 98.3–100.050.0% 44.3–55.750.0%
depth 130093.7% 91.0–96.350.0% 44.3–55.750.0%
depth 230092.0% 88.7–94.750.0% 44.3–55.750.0%
depth 330085.0% 81.3–89.350.0% 44.3–55.750.0%
depth 430076.3% 71.7–81.050.0% 44.3–55.750.0%
depth 530089.3% 85.7–92.350.0% 44.3–55.750.0%

What GLiNER2.5-Decide got right and wrong, depth by depth

At each proof depth, GLiNER2.5-Decide's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

35.3%
right on 106 of 300 problems wrong; chance 33% (dashed line)
wrong

The bald eagle visits the bear.

Correct answer: false · Model said: unknown (52.2% sure)

The problem and its proof

The bald eagle is big. The bald eagle is round. The bald eagle likes the bear. The bald eagle likes the cow. The bald eagle sees the bear. The bald eagle sees the cow. The bald eagle does not visit the bear. The bald eagle visits the cow. The bear is big. The bear is round. The bear likes the cow. The bear sees the bald eagle. The bear does not visit the bald eagle. The cow likes the bear. The cow does not see the bald eagle. The cow sees the bear. If someone visits the bald eagle and they like the bald eagle then the bald eagle likes the cow.

Stated in the text: "The bald eagle does not visit the bear."

right

The bald eagle does not need the bear.

Correct answer: unknown · Model said: unknown (51.5% sure)

The problem and its proof

The bald eagle chases the mouse. The bear chases the mouse. The bear eats the bald eagle. The bear is rough. The bear does not need the bald eagle. The mouse chases the bear. The mouse is not nice. The mouse is rough. The mouse needs the bald eagle. The mouse needs the bear. If someone needs the bear then they chase the mouse. If someone is young and they chase the mouse then the mouse does not chase the bear.

No proof: neither the statement nor its negation follows.

wrong

The bald eagle sees the rabbit.

Correct answer: false · Model said: unknown (59.0% sure)

The problem and its proof

The bald eagle does not see the rabbit. The mouse needs the rabbit. The rabbit needs the mouse. All big people are young.

Stated in the text: "The bald eagle does not see the rabbit."

wrong

Harry is round.

Correct answer: false · Model said: unknown (50.5% sure)

The problem and its proof

Erin is round. Erin is smart. Fiona is nice. Fiona is smart. Harry is blue. Harry is red. Harry is not round. If Harry is red and Harry is not round then Harry is rough. If something is smart then it is red.

Stated in the text: "Harry is not round."

Depth 1, open-world task

34.8%
right on 105 of 302 problems wrong; chance 33% (dashed line)
wrong

Dave is not quiet.

Correct answer: false · Model said: unknown (44.8% sure)

The problem and its proof

Dave is big. Dave is blue. Dave is furry. Dave is red. Erin is blue. Erin is not furry. Fiona is not quiet. If Dave is not furry then Dave is rough. All quiet people are not cold. Red, blue people are quiet.

The proof that the statement's negation follows:

  1. Dave is red. + Dave is blue.Red, blue people are quiet.Dave is quiet.
wrong

Charlie is not round.

Correct answer: false · Model said: unknown (50.0% sure)

The problem and its proof

Anne is cold. Anne is green. Charlie is big. Charlie is cold. Charlie is furry. Harry is green. Harry is nice. Big, cold things are round. All round things are young.

The proof that the statement's negation follows:

  1. Charlie is big. + Charlie is cold.Big, cold things are round.Charlie is round.
wrong

Fred is nice.

Correct answer: true · Model said: unknown (35.8% sure)

paraphrased
The problem and its proof

Dave is shaped big and round like a balloon. His red color shows how kind he is. Eric might be rough and red but he's actually very kind. Fred may be round, but he is also kind. Green, big and red are qualities which all describe Gary. People who are very nice and easily flush red are often wearing green concealer. If someone plays rough and is nice and round, they will be big. Anyone who is both red and green must be as cold as Christmas. Round people who are kind tend to be nice. Young people who are both nice and cold also tend to be rough. Why is it that every round person that is blue is so young. A kind person who looks blue because he is is cold is usually big in stature.

  1. Fred is round. + Fred is kind.If someone is round and kind then they are nice.Fred is nice.
right

Bob is not nice.

Correct answer: false · Model said: false (39.8% sure)

paraphrased
The problem and its proof

Bob, a kind young person is always feeling blue. Dave is nice and kind,though he's also rather cold. For being so cold, it's good Eric can remain nice. Gary is a kind person and he is also often cold. If you're truly nice, cold, and blue, you're kind, too. A blue colored person who is nice is a red person. If you come across a young, blue person you can count on them being nice to you. Big, kind folks are green ones. A person who is kind and rough and blue is young. Nice, young red people will also turn out to always be green. Cold, young people are also certain to be rough people.

The proof that the statement's negation follows:

  1. Bob is blue. + Bob is young.If someone is blue and young then they are nice.Bob is nice.

Depth 2, open-world task

35.3%
right on 107 of 303 problems wrong; chance 33% (dashed line)
right

Fred is rough.

Correct answer: unknown · Model said: unknown (46.1% sure)

paraphrased
The problem and its proof

Alan is cold to his ex-wife. She knows he is round and red. He is kind to everyone else. Bob is kind. He is also very cold and blue. Fred seems to be round. Gary is a big guy who is colored red from a sunburn. He also has a cold. Red people are almost always rough. Every time you meet someone kind and nice, they'll be green, too. Big people who are red in color are almost always young. It seems that a cold person who is round and nice will be green. People of young age are rough and cold but some are nice. Blue eyed people, green with sickness and rough around the edges quickly turn red and blush when stepping ashore. I bet anyone who is round and kind is also pretty young.

No proof: neither the statement nor its negation follows.

wrong

Bob is rough.

Correct answer: false · Model said: unknown (43.2% sure)

The problem and its proof

Bob is white. If someone is round and smart then they are furry. If someone is nice then they are furry. If Bob is white then Bob is furry. All furry people are not rough. If Bob is smart then Bob is not white. All rough people are not kind. Rough, round people are kind. If someone is white and not rough then they are not kind.

The proof that the statement's negation follows:

  1. Bob is white.If Bob is white then Bob is furry.Bob is furry.
  2. All furry people are not rough.Bob is not rough.
wrong

Alan is red.

Correct answer: true · Model said: unknown (44.0% sure)

paraphrased
The problem and its proof

Alan seems big and rough, but he's actually quite kind. Even though Charlie is round and big, he is very kind. Dave is green and cold too. That guy Eric sure is nice. Rough people who are red in color and big are usally round in shape. Big, kind folks are green ones. Kind red and round people often look green. When someone is blue, nice and young they will also be as big as the sun. When you run into someone who is rough and green at the same time, they will also be red.

  1. Alan is big. + Alan is kind.If someone is big and kind then they are green.Alan is green.
  2. + Alan is rough.If someone is rough and green then they are red.Alan is red.
wrong

Harry is not young.

Correct answer: false · Model said: unknown (38.2% sure)

paraphrased
The problem and its proof

After Charlie got wet in the rain, he feels cold. He also looks green but big. The young person who is always feeling cold is named Dave. Gary seems to be round. Rough and round Harry is nice and kind and he's also cold. Some cold people can be nice at times,and red at at other times. Round people who feel blue and are green in color are often young in age. A red hued and rough looking person is definitely a young person. Young people are so rough that they can hold their breath until they are blue. People of young age are rough and cold but some are nice.

The proof that the statement's negation follows:

  1. Harry is cold. + Harry is nice.If someone is cold and nice then they are red.Harry is red.
  2. + Harry is rough.If someone is red and rough then they are young.Harry is young.

Depth 3, open-world task

36.3%
right on 110 of 303 problems wrong; chance 33% (dashed line)
wrong

The mouse is not young.

Correct answer: false · Model said: unknown (47.9% sure)

The problem and its proof

The cat is not red. The cat needs the mouse. The cat needs the tiger. The cow eats the cat. The cow is young. The mouse visits the cat. The tiger eats the cat. The tiger does not eat the cow. The tiger is red. The tiger does not visit the cat. The tiger visits the cow. The tiger visits the mouse. If the tiger visits the mouse then the mouse eats the tiger. If someone is young and they need the mouse then they eat the mouse. If someone eats the tiger then they are green. If someone is green then they are young. If someone is young then they need the mouse.

The proof that the statement's negation follows:

  1. The tiger visits the mouse.If the tiger visits the mouse then the mouse eats the tiger.The mouse eats the tiger.
  2. If someone eats the tiger then they are green.The mouse is green.
  3. If someone is green then they are young.The mouse is young.
wrong

Fred is not big.

Correct answer: false · Model said: unknown (39.3% sure)

paraphrased
The problem and its proof

Dave may be round, but he is also kind. Fred is colored red and blue. He is also young, quite round, and cold. Green, big and red are qualities which all describe Harry. Folks who are rough and tumble yet gentle and kind are blue. When someone is kind yet can be cold and blue, they will also be very big. If you come across a young, blue person you can count on them being nice to you. An individual with the traits of being nice, red, and cold is also round. Young people are nice and cold are kind to people regardless.

The proof that the statement's negation follows:

  1. Fred is blue. + Fred is young.If someone is blue and young then they are nice.Fred is nice.
  2. + Fred is cold. + Fred is young.If someone is cold and nice and young then they are kind.Fred is kind.
  3. + Fred is cold. + Fred is blue.If someone is kind and cold and blue then they are big.Fred is big.
right

Charlie is not cold.

Correct answer: unknown · Model said: unknown (48.1% sure)

The problem and its proof

Charlie is green. Dave is red. Fiona is round. If something is green then it is not kind. Red things are furry. If something is red and green then it is round. If something is furry and cold then it is round. If something is kind then it is nice. All round things are nice. All nice things are cold. If something is red and not furry then it is not cold.

No proof: neither the statement nor its negation follows.

wrong

The bear is rough.

Correct answer: false · Model said: unknown (40.9% sure)

The problem and its proof

The bear does not see the rabbit. The bear visits the rabbit. The rabbit eats the bear. If someone visits the rabbit then they see the bear. If someone sees the bear then they are cold. If someone sees the bear then they are cold. Cold people are not rough. If someone is cold and they do not visit the rabbit then they do not visit the bear. If someone is young and they see the bear then they are not round. If someone eats the bear then they do not see the rabbit. Cold, round people are not young.

The proof that the statement's negation follows:

  1. The bear visits the rabbit.If someone visits the rabbit then they see the bear.The bear sees the bear.
  2. If someone sees the bear then they are cold.The bear is cold.
  3. Cold people are not rough.The bear is not rough.

Depth 4, open-world task

38.3%
right on 116 of 303 problems wrong; chance 33% (dashed line)
wrong

Fiona is not young.

Correct answer: false · Model said: unknown (53.3% sure)

The problem and its proof

Bob is kind. Bob is quiet. Bob is young. Charlie is not green. Dave is round. Fiona is rough. Fiona is round. Kind, blue people are quiet. Rough, round people are quiet. Round people are rough. If someone is blue then they are green. All rough, kind people are green. If someone is quiet then they are blue. All green, round people are young. All kind people are round. If Charlie is rough then Charlie is blue.

The proof that the statement's negation follows:

  1. Fiona is rough. + Fiona is round.Rough, round people are quiet.Fiona is quiet.
  2. If someone is quiet then they are blue.Fiona is blue.
  3. If someone is blue then they are green.Fiona is green.
  4. + Fiona is round.All green, round people are young.Fiona is young.
right

Bob is green.

Correct answer: unknown · Model said: unknown (62.5% sure)

The problem and its proof

Bob is round. Dave is big. Dave is young. Fiona is quiet. Gary is cold. Gary is red. Gary is young. All cold things are big. If something is quiet then it is young. Round, cold things are quiet. Young, big things are green. All big, green things are round. Young things are cold.

No proof: neither the statement nor its negation follows.

right

Fiona is not big.

Correct answer: unknown · Model said: unknown (54.3% sure)

The problem and its proof

Bob is green. Bob is rough. Bob is smart. Dave is big. Dave is blue. Dave is quiet. Dave is rough. Dave is smart. Fiona is rough. Fiona is smart. If someone is smart and big then they are blue. All smart people are furry. Rough, green people are big. All green, rough people are smart. If someone is blue and rough then they are quiet. If Bob is blue and Bob is smart then Bob is big. All quiet people are big. If someone is big then they are furry.

No proof: neither the statement nor its negation follows.

wrong

Charlie is not big.

Correct answer: false · Model said: unknown (42.6% sure)

paraphrased
The problem and its proof

The young person who is always feeling cold is named Bob. Charlie is nice and kind,though he's also rather cold. Gary is green and cold too. Some cold people can be nice at times,and red at at other times. An individual with the traits of being nice, red, and cold is also round. Kind red and round people often look green. Nice and green people are often found to be big as well. If a person is blue and red and big, then that person is cold.

The proof that the statement's negation follows:

  1. Charlie is cold. + Charlie is nice.If someone is cold and nice then they are red.Charlie is red.
  2. + Charlie is nice. + Charlie is cold.If someone is nice and red and cold then they are round.Charlie is round.
  3. + Charlie is kind.If someone is red and kind and round then they are green.Charlie is green.
  4. + Charlie is nice.If someone is nice and green then they are big.Charlie is big.

Depth 5, open-world task

30.8%
right on 89 of 289 problems wrong; chance 33% (dashed line)
wrong

Gary is kind.

Correct answer: false · Model said: unknown (46.2% sure)

The problem and its proof

Anne is not blue. Anne is cold. Anne is kind. Anne is nice. Anne is quiet. Erin is quiet. Gary is not nice. Gary is rough. Harry is not blue. Harry is cold. Harry is nice. Harry is quiet. If Gary is not nice then Gary is cold. Rough, cold things are blue. If something is kind then it is cold. If something is white and not nice then it is quiet. If something is cold and blue then it is white. All kind, blue things are not white. If something is quiet and rough then it is not kind.

The proof that the statement's negation follows:

  1. Gary is not nice.If Gary is not nice then Gary is cold.Gary is cold.
  2. + Gary is rough.Rough, cold things are blue.Gary is blue.
  3. If something is cold and blue then it is white.Gary is white.
  4. + Gary is not nice.If something is white and not nice then it is quiet.Gary is quiet.
  5. + Gary is rough.If something is quiet and rough then it is not kind.Gary is not kind.
wrong

The lion eats the lion.

Correct answer: true · Model said: unknown (54.0% sure)

The problem and its proof

The bear chases the lion. The bear chases the mouse. The bear is red. The bear is round. The bear needs the squirrel. The lion eats the bear. The lion is kind. The lion needs the bear. The mouse is round. The squirrel chases the bear. If something eats the bear and the bear eats the lion then the lion is round. If something chases the lion then the lion eats the squirrel. If something is red then it needs the lion. If something eats the squirrel and the squirrel eats the mouse then it chases the lion. If the bear is blue and the bear needs the mouse then the mouse eats the lion. If something eats the lion and it is round then the lion is red. If something needs the lion then it eats the lion.

  1. The bear is red.If something is red then it needs the lion.The bear needs the lion.
  2. If something needs the lion then it eats the lion.The bear eats the lion.
  3. + The bear is round.If something eats the lion and it is round then the lion is red.The lion is red.
  4. If something is red then it needs the lion.The lion needs the lion.
  5. If something needs the lion then it eats the lion.The lion eats the lion.
right

Gary is not green.

Correct answer: unknown · Model said: unknown (49.7% sure)

The problem and its proof

Bob is cold. Charlie is green. Charlie is quiet. Charlie is white. Charlie is young. Gary is quiet. Gary is white. If something is young then it is red. All white things are quiet. Quiet, young things are cold. If Gary is quiet and Gary is green then Gary is young. If something is red then it is green. If Gary is round then Gary is white. White, young things are green. If something is red then it is young.

No proof: neither the statement nor its negation follows.

wrong

The bear does not visit the cat.

Correct answer: false · Model said: unknown (49.6% sure)

The problem and its proof

The bear is green. The bear visits the cow. The cat is green. The cat is nice. The cow likes the cat. The cow visits the lion. The lion visits the cat. If someone visits the cow then the cow eats the lion. If someone eats the bear then the bear visits the cat. If someone is young then they eat the bear. If someone is blue then they eat the bear. If someone likes the lion then they are young. If the lion is nice then the lion eats the bear. If someone eats the lion then they like the lion. If someone is green then they eat the cat.

The proof that the statement's negation follows:

  1. The bear visits the cow.If someone visits the cow then the cow eats the lion.The cow eats the lion.
  2. If someone eats the lion then they like the lion.The cow likes the lion.
  3. If someone likes the lion then they are young.The cow is young.
  4. If someone is young then they eat the bear.The cow eats the bear.
  5. If someone eats the bear then the bear visits the cat.The bear visits the cat.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

50.0%
right on 150 of 300 problems wrong; chance 50% (dashed line)
wrong

The mouse is blue.

Correct answer: true · Model said: false (64.4% sure)

The problem and its proof

The mouse is blue. If someone is round and blue then they are cold. Round people are kind. If someone is blue then they are kind. Red people are kind. All cold people are kind. If someone is kind then they are round.

Stated in the text: "The mouse is blue."

right

The cow does not eat the squirrel.

Correct answer: false · Model said: false (65.2% sure)

The problem and its proof

The cow eats the squirrel. The cow likes the squirrel. The squirrel is cold. If something eats the cow then it chases the cow.

Stated in the text: "The cow eats the squirrel."

right

Fred is not cold.

Correct answer: false · Model said: false (70.1% sure)

paraphrased
The problem and its proof

Bob seems to be round. Dave may be young but he is nice, wears green shoes and is cold. For being so cold, it's good Fred can remain nice. The young person who is always feeling cold is named Gary. Young round people who are green are usually blue. If a person is described as being nice, cold and young it usually follow that they are red. One who is young, red and also cold will definitely be round, too. Someone that is cold rough and red is also considered to be kind. Every person I met that was cold and round was also red. A person who is green, blue, and red in color will be round shaped. A person who inherits genes that make them big and green will also express a blue color.

Stated in the text: "Fred is cold."

wrong

The tiger sees the cat.

Correct answer: true · Model said: false (62.0% sure)

The problem and its proof

The bear chases the tiger. The cat sees the tiger. The tiger sees the cat. If something sees the cat then the cat is cold.

Stated in the text: "The tiger sees the cat."

Depth 1, closed-world task

50.0%
right on 150 of 300 problems wrong; chance 50% (dashed line)
wrong

The squirrel sees the cat.

Correct answer: true · Model said: false (63.4% sure)

The problem and its proof

The bald eagle chases the cat. The bald eagle is blue. The bald eagle needs the cat. The bald eagle sees the squirrel. The cat is young. The cat needs the squirrel. The squirrel chases the cat. The squirrel is big. The squirrel is rough. The squirrel is young. The squirrel needs the bald eagle. The squirrel sees the bald eagle. If someone sees the bald eagle then they see the cat.

  1. The squirrel sees the bald eagle.If someone sees the bald eagle then they see the cat.The squirrel sees the cat.
right

The rabbit does not visit the cat.

Correct answer: false · Model said: false (67.2% sure)

The problem and its proof

The cat chases the cow. The cat chases the rabbit. The cat is nice. The cat needs the cow. The cat visits the rabbit. The cow is nice. The cow is red. The cow needs the cat. The cow visits the cat. The mouse chases the cow. The mouse chases the rabbit. The mouse is kind. The mouse needs the rabbit. The rabbit chases the cow. The rabbit needs the cow. If something visits the cow and it is nice then the cow chases the mouse. If something is nice and it chases the cow then it needs the cow. If the rabbit is red and the rabbit visits the mouse then the rabbit is kind. If something chases the cow then it visits the mouse. If something chases the cow then it visits the cat. If the mouse needs the rabbit and the rabbit visits the mouse then the rabbit is red.

  1. The rabbit chases the cow.If something chases the cow then it visits the cat.The rabbit visits the cat.
right

The bear does not chase the rabbit.

Correct answer: false · Model said: false (71.2% sure)

The problem and its proof

The bear likes the rabbit. The mouse chases the bear. The mouse chases the squirrel. The mouse needs the bear. The rabbit chases the mouse. The rabbit chases the squirrel. The squirrel is green. If something likes the bear and it needs the rabbit then it needs the squirrel. If something likes the rabbit and it chases the rabbit then it needs the bear. If the bear is blue and the bear is rough then the bear chases the mouse. If something chases the squirrel and it does not chase the bear then the bear chases the rabbit. If something needs the bear then it needs the mouse. Rough, blue things are green. If something likes the squirrel then it is rough. If something needs the bear and it needs the mouse then it likes the squirrel. If something is green and it does not like the squirrel then the squirrel is red.

  1. The rabbit chases the squirrel.If something chases the squirrel and it does not chase the bear then the bear chases the rabbit.The bear chases the rabbit.
wrong

Gary is not young.

Correct answer: true · Model said: false (71.8% sure)

paraphrased
The problem and its proof

Eric is round but he is also nice and kind. Fred is quite rough and very kind, but unfortunately cold and also blue. Gary seems to be round. That guy Harry sure is nice. If you meet someone with rough skin who is cold from being outside, you'll notice they are nice. Every time you meet someone kind and nice, they'll be green, too. Most big and cold people are also rough. Blue eyed people, green with sickness and rough around the edges quickly turn red and blush when stepping ashore. Is a kind person if feeling cold and looking red then they are usually young.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 2, closed-world task

50.0%
right on 150 of 300 problems wrong; chance 50% (dashed line)
wrong

The bear needs the bear.

Correct answer: true · Model said: false (59.6% sure)

The problem and its proof

The bear is blue. The rabbit chases the bear. The rabbit is blue. If someone chases the rabbit then they need the bear. If someone chases the bear and they are blue then the bear chases the rabbit.

  1. The rabbit chases the bear. + The rabbit is blue.If someone chases the bear and they are blue then the bear chases the rabbit.The bear chases the rabbit.
  2. If someone chases the rabbit then they need the bear.The bear needs the bear.
right

Fred is kind.

Correct answer: false · Model said: false (64.2% sure)

paraphrased
The problem and its proof

Round, nice Bob felt as blue and big as he had ever been. The young person who is always feeling cold is named Charlie. That guy Eric sure is nice. That guy Fred sure is nice. People here that are both big and kind while being cold are always red. A person feeling blue because of their big size will always be cold. You must be aware of big, red, young people being rough. Do not fear big cold people for they are often kind. People that are rough and red in the face, are usually big in size.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The bald eagle does not like the dog.

Correct answer: false · Model said: false (73.9% sure)

The problem and its proof

The bald eagle sees the dog. The dog is blue. The dog sees the bald eagle. If someone needs the bald eagle and they are kind then the bald eagle likes the dog. If someone sees the bald eagle then the bald eagle is kind. If someone is young and they see the dog then the dog likes the bald eagle. If someone sees the dog then they need the bald eagle. If someone likes the dog and they need the bald eagle then the dog needs the bald eagle. If someone is nice and young then they see the bald eagle. If someone needs the bald eagle and the bald eagle needs the dog then they like the dog. If the bald eagle sees the dog and the bald eagle does not like the dog then the bald eagle is blue.

  1. The bald eagle sees the dog.If someone sees the dog then they need the bald eagle.The bald eagle needs the bald eagle.
  2. + The dog sees the bald eagle.If someone sees the bald eagle then the bald eagle is kind.The bald eagle is kind.
  3. If someone needs the bald eagle and they are kind then the bald eagle likes the dog.The bald eagle likes the dog.
wrong

Erin is not green.

Correct answer: true · Model said: false (70.0% sure)

The problem and its proof

Erin is kind. Erin is quiet. Erin is young. Cold things are white. If something is quiet and red then it is white. If something is white and not red then it is green.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

50.0%
right on 150 of 300 problems wrong; chance 50% (dashed line)
wrong

The cow does not visit the cow.

Correct answer: true · Model said: false (69.9% sure)

The problem and its proof

The cat is big. The cat is blue. The cat needs the cow. The cow is nice. The cow needs the dog. The cow needs the rabbit. The cow visits the cat. The dog is red. The dog visits the rabbit. The rabbit needs the cat. The rabbit visits the cow. If someone sees the cow then the cow visits the cat. If someone is blue then they need the cow. Blue people are big. If the cat visits the dog then the dog needs the cat. If someone needs the cow then they visit the dog. If someone needs the cat then they are blue. If someone is big then they visit the cow. If someone sees the cow then the cow is blue. If someone needs the cow and they do not need the dog then they need the rabbit.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Eric is not cold.

Correct answer: false · Model said: false (72.9% sure)

paraphrased
The problem and its proof

Alan is green and cold too. Bob seems to be round. Eric who is young and round is also big and rough. When a person is nice and round and cold, they look blue. Even though young people are rough, they are still very red. A person feeling blue because of their big size will always be cold. Young people who feel cold and are still kind usually look rough. Red people, who are also round and young, are often quite blue as well. I bet anyone who is round and kind is also pretty young.

  1. Eric is rough. + Eric is young.If someone is rough and young then they are red.Eric is red.
  2. + Eric is round. + Eric is young.If someone is red and round and young then they are blue.Eric is blue.
  3. + Eric is big.If someone is blue and big then they are cold.Eric is cold.
wrong

Harry is not big.

Correct answer: true · Model said: false (69.8% sure)

The problem and its proof

Anne is big. Anne is round. Dave is nice. Dave is red. Fiona is red. Fiona is rough. Harry is round. If someone is green and big then they are round. Red people are rough. Rough people are green. Nice, round people are green. If Harry is green and Harry is big then Harry is kind. All nice people are red. If someone is rough and green then they are kind. Kind, rough people are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Harry is furry.

Correct answer: false · Model said: false (67.3% sure)

The problem and its proof

Anne is blue. Anne is nice. Erin is young. Fiona is big. Fiona is nice. Fiona is red. Harry is quiet. If someone is blue then they are red. If Erin is not big then Erin is furry. All young, blue people are furry. If someone is quiet then they are blue. If Erin is nice then Erin is quiet. Big, nice people are young. If Harry is blue then Harry is big. All young, furry people are nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 4, closed-world task

50.0%
right on 150 of 300 problems wrong; chance 50% (dashed line)
right

Fiona is big.

Correct answer: false · Model said: false (67.0% sure)

The problem and its proof

Anne is blue. Anne is furry. Erin is blue. Erin is cold. Erin is rough. Fiona is cold. Fiona is young. Big, red people are young. If someone is furry then they are blue. If someone is blue then they are red. If someone is blue and red then they are big. Cold, red people are furry. All blue people are red. If Anne is red then Anne is rough. All big people are cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The squirrel sees the squirrel.

Correct answer: false · Model said: false (65.1% sure)

The problem and its proof

The lion is nice. The lion is red. The squirrel chases the tiger. The squirrel likes the lion. The tiger chases the squirrel. The tiger likes the lion. The tiger sees the lion. If someone likes the lion then they see the lion. If the squirrel chases the lion and the squirrel sees the tiger then the squirrel chases the tiger. If someone sees the tiger then they see the squirrel. If someone chases the squirrel and they chase the lion then the squirrel is nice. If someone sees the squirrel and they are blue then they chase the tiger. If someone is young and nice then they chase the lion. If someone likes the tiger and the tiger is nice then they like the squirrel. If someone is nice then they see the tiger.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Dave is not white.

Correct answer: true · Model said: false (66.9% sure)

The problem and its proof

Anne is rough. Dave is quiet. If someone is rough then they are quiet. If Anne is red then Anne is green. If Anne is green then Anne is red. If someone is kind and not rough then they are green. If someone is white and not green then they are blue. Rough, quiet people are blue. If someone is blue and not red then they are white. If someone is green and not kind then they are white.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Bob is cold.

Correct answer: true · Model said: false (70.4% sure)

paraphrased
The problem and its proof

Bob is a big fellow, often blue and sad, but he is nice. Eric seems to be round. Gary is young and green. This makes him cold to others. A kind person who is down in the dumps and blue tends to have a rough side. Anyone who is both red and green must be as cold as Christmas. Round young people, red with loveliness, are very cold towards others. People who are said to be big and nice are round. Watch out for young people that are kind for they are rough. A nice person who feels blue and looks round is usually kind. When a person is blue, rough and kind then it means they also are cold.

  1. Bob is big. + Bob is nice.If someone is big and nice then they are round.Bob is round.
  2. + Bob is blue. + Bob is nice.If someone is round and blue and nice then they are kind.Bob is kind.
  3. + Bob is blue.If someone is blue and kind then they are rough.Bob is rough.
  4. + Bob is blue.If someone is blue and rough and kind then they are cold.Bob is cold.

Depth 5, closed-world task

50.0%
right on 150 of 300 problems wrong; chance 50% (dashed line)
wrong

Gary is not young.

Correct answer: true · Model said: false (70.5% sure)

The problem and its proof

Bob is big. Bob is blue. Dave is big. Dave is blue. Dave is furry. Dave is quiet. Dave is white. Gary is furry. Gary is red. Harry is big. Harry is furry. Harry is red. If Gary is quiet then Gary is young. All big things are young. Young, blue things are furry. All quiet things are white. All blue, white things are quiet. Blue, red things are quiet. White things are quiet. Furry things are red. Furry, quiet things are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Fiona is not young.

Correct answer: false · Model said: false (70.9% sure)

The problem and its proof

Anne is big. Anne is blue. Anne is round. Bob is green. Fiona is big. Gary is quiet. Gary is young. Big things are green. If something is round and quiet then it is young. All quiet things are round. If something is green and not red then it is blue. If something is round then it is blue. If something is young and not green then it is quiet. All blue, green things are quiet.

  1. Fiona is big.Big things are green.Fiona is green.
  2. If something is green and not red then it is blue.Fiona is blue.
  3. All blue, green things are quiet.Fiona is quiet.
  4. All quiet things are round.Fiona is round.
  5. If something is round and quiet then it is young.Fiona is young.
right

Gary is round.

Correct answer: false · Model said: false (65.8% sure)

The problem and its proof

Anne is big. Anne is young. Charlie is blue. Charlie is kind. Charlie is round. Charlie is young. Gary is big. Harry is big. Harry is blue. Harry is kind. Harry is nice. Harry is smart. All smart things are big. All kind, smart things are round. All young, kind things are blue. All nice, young things are smart. If something is big then it is kind. If something is blue then it is nice. If Anne is nice and Anne is kind then Anne is smart. If Gary is nice then Gary is young. If something is round and big then it is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Dave is not furry.

Correct answer: true · Model said: false (70.3% sure)

The problem and its proof

Charlie is big. Dave is red. Harry is smart. If Dave is green and Dave is smart then Dave is big. Red people are kind. If someone is red and blue then they are big. If someone is big then they are smart. If someone is furry and kind then they are smart. All blue, smart people are furry. If someone is smart then they are green. If someone is smart and big then they are blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

Where each one is better

Open world, Jev vs GLiNER2.5-Decide. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GLiNER2.5-Decide is clearly ahead on unknown answers (+19.7 points).

Closed world, Jev vs GLiNER2.5-Decide. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5 and true answers. GLiNER2.5-Decide is clearly ahead on false answers (+11.1 points).

A slice counts as clearly ahead when its paired 95% interval excludes zero.

True, false and unknown, open world

By correct answer, Open world
Correct answerItemsJevGLiNER2.5-DecideBest constant
false60588.3% 85.5–90.911.4% 8.9–13.9one answer
true60588.8% 86.1–91.11.7% 0.8–2.8one answer
unknown59074.2% 70.7–77.693.9% 92.0–95.8one answer

True, false and unknown, closed world

By correct answer, Closed world
Correct answerItemsJevGLiNER2.5-DecideBest constant
false90088.9% 86.9–90.9100.0% 100.0–100.0one answer
true90089.7% 87.7–91.60.0% 0.0–0.0one answer

Paraphrased vs templated rules, open world

By wording, Open world
WordingItemsJevGLiNER2.5-DecideBest constant
templated1,29886.1% 84.2–87.934.5% 32.0–37.034.5%
paraphrased (NatLang)50278.1% 74.5–82.136.9% 32.9–41.235.9%

Paraphrased vs templated rules, closed world

By wording, Closed world
WordingItemsJevGLiNER2.5-DecideBest constant
templated1,30694.0% 92.8–95.350.2% 47.5–52.850.2%
paraphrased (NatLang)49476.7% 73.1–80.449.4% 45.1–53.850.6%

Answer by answer: Jev and GLiNER2.5-Decide on the same problems

Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
GLiNER2.5-Decide, open-world task
Correct ↓ · said →truefalseunknown
true101594
false1069526
unknown828554

Jev said true on 37% of problems; the correct answer is true on 34%. GLiNER2.5-Decide said unknown on 93% of problems; the correct answer is unknown on 33%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
GLiNER2.5-Decide, closed-world task
Correct ↓ · said →truefalse
true0900
false0900

Jev said true on 50% of problems; the correct answer is true on 50%. GLiNER2.5-Decide said false on 100% of problems; the correct answer is false on 50%.

Where GLiNER2.5-Decide errs

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

GLiNER2.5-Decide, open-world task

GLiNER2.5-Decide, open-world task
Correct ↓ · said →truefalseunknownRecall
true1015941.7%
false106952611.4%
unknown82855493.9%

GLiNER2.5-Decide answered false 5.4%, true 1.6%, unknown 93.0% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

GLiNER2.5-Decide, closed-world task

GLiNER2.5-Decide, closed-world task
Correct ↓ · said →truefalseRecall
true09000.0%
false0900100.0%

GLiNER2.5-Decide answered false 100.0% of the time; the correct answers are false 50.0%, true 50.0%.

GLiNER2.5-Decide at depth 0 and depth 5

GLiNER2.5-Decide, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true00100
false21583
unknown2791
GLiNER2.5-Decide, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true00101
false0398
unknown0186

Open-world task. At depth 0, GLiNER2.5-Decide was right on 35% of problems (the green diagonal); at depth 5, on 31%. The answer whose share moved most is unknown: 91% of its answers at depth 0, 99% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
GLiNER2.5-Decide, closed-world task, depth 0
Correct ↓ · said →truefalse
true0150
false0150
GLiNER2.5-Decide, closed-world task, depth 5
Correct ↓ · said →truefalse
true0150
false0150

Closed-world task. At depth 0, GLiNER2.5-Decide was right on 50% of problems (the green diagonal); at depth 5, on 50%. The answer whose share moved most is true: 0% of its answers at depth 0, 0% at depth 5, against 50% and 50% of the correct answers.

GLiNER2.5-Decide across every breakdown

For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownWeakest · strongest
Proof depthdepth 5 30.8% · depth 4 38.3% (7.5 points)
Gold answertrue 1.7% · unknown 93.9% (92.2 points)
Theory kindrelation 31.3% · attribute 37.9% (6.6 points)
Negation in the theorywith negation 30.0% · without negation 40.2% (10.2 points)
Negated statementplain statement 33.1% · negated statement 37.2% (4.0 points)
Question strategyproof 1.7% · rconc 99.6% (97.9 points)
Paraphrased rulestemplated 34.5% · paraphrased (NatLang) 36.9% (2.3 points)
Theory's deepest proof5 24.2% · 0 60.6% (36.4 points)
Theory length110+ 29.9% · 0-49 39.7% (9.7 points)
Number of rules3-5 28.1% · 8+ 40.7% (12.7 points)
Number of facts13+ 23.9% · 0-3 42.8% (18.9 points)
Proof size2 0.9% · 0-1 72.0% (71.1 points)

Consistency and calibration

Asked every open world question a second time, GLiNER2.5-Decide gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 35.2% the first time and 35.2% the second.

Asked every closed world question a second time, GLiNER2.5-Decide gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 50.0% the first time and 50.0% the second.

Calibration on the open world task: GLiNER2.5-Decide's probability for its own answer averaged 48.8% against 35.2% accuracy; expected calibration error 13.7 points.

Calibration on the closed world task: GLiNER2.5-Decide's probability for its own answer averaged 68.7% against 50.0% accuracy; expected calibration error 18.7 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.51 on the open world task against Jev's 0.85; 0.57 on the closed world task against Jev's 0.86.

Consistency and calibration for every model, Jev included

Size, hardware and speed

ModelKindParametersRan onMedian time per decisionMemory peak
Jevhosted decision modelundisclosedvendor's servers189 msnot applicable (hosted)
GLiNER2.5-Decideopen decision model340Mour laptop (Apple M1 Max, 32 GB)153 ms4.5 GB

Times are open-world, one request at a time. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.

GLiNER2.5-Decide

Parameters
340M
Architecture
GLiNER2 schema classifier on a DeBERTa-v3-large encoder; answers in one pass ("does not reason", model card)
Weights
1.95 GB (model.safetensors, fastino/GLiNER2.5-Decide@5a7adf7)
Precision
float32, as loaded by gliner2 (PyTorch)
Where it ran
this machine, gliner2 2.0.0 (local extra), PyTorch on MPS
Maker's stated hardware
"CPU or GPU"; 38.3 ms p50 on an NVIDIA V100, 167.3 ms on a 48-vCPU Xeon, short inputs (Fastino)
Memory measured
4.5 GB in use, 4.5 GB peak
Price
open weights (Apache 2.0); Fastino also hosts it at $0.03 or $0.15 per million input tokens (its pricing page and model catalog disagree)
Time per decision, open world
median 153 ms, 90th percentile 203 ms, on a laptop
Time per decision, closed world
median 146 ms, 90th percentile 188 ms, on a laptop
Source
huggingface.co/fastino/GLiNER2.5-Decide

The models, and what we predicted

Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

GLiNER2.5-Decide GLiNER2.5-Decide is an open decision model made by Fastino Labs: a 340M-parameter schema classifier on a DeBERTa-v3-large encoder, released under Apache 2.0. It answers in one pass; its model card says it "does not reason." We ran it on our own laptop with Fastino's gliner2 package, PyTorch on Apple's GPU, giving it the question as its prompt and each option with its description.

Other comparisons: Jev vs GLiDE · Jev vs GPT-6 Luna · Kev vs Jev · Jev vs Laya · Every model