Skip to main content
HardDecisions

Comparison · Jev vs GLiDE

Jev vs GLiDE: two hosted decision models on deep reasoning

GLiDE is Fastino's hosted decision model, released on 1 October 2026, which Fastino calls the first thinking decision model. It takes the same request as Jev, so both got exactly the same text, question and options for the same 3,600 problems.

We ran GLiDE twice. The first run was on October 2, the day after launch. On October 3 Fastino's CEO asked us to re-evaluate GLiDE, saying Fastino had problems during the launch, so we ran exactly the same benchmark again with the same request. Both runs are shown side by side. Jev answers every decision in one parallel pass; GLiDE, by Fastino's own description, spends extra time and tokens on decisions it is unsure of. Fastino reports that GLiDE beats Jev on its Decision Index 0.2.1 (64.81 to 57.91); that is Fastino's benchmark, not this one.

Overall: Jev, GLiDE, Oct 3 rerun and GLiDE, Oct 2

Open world: true, false or unknown

  • Jev82.2 to 85.5 (95%), 1,800 items83.8%
  • GLiDE, Oct 3 rerun74.4 to 78.3 (95%), 1,800 items76.4%
  • GLiDE, Oct 281.1 to 84.6 (95%), 1,800 items82.9%

Jev minus GLiDE, Oct 3 rerun: +7.4 points on the same 1,800 items, 95% interval +5.2 to +9.9: Jev is clearly ahead.

Jev minus GLiDE, Oct 2: +0.9 points on the same 1,800 items, 95% interval −1.2 to +3.3: the interval includes zero.

Closed world: true or false

  • Jev87.8 to 90.7 (95%), 1,800 items89.3%
  • GLiDE, Oct 3 rerun66.7 to 71.0 (95%), 1,800 items68.8%
  • GLiDE, Oct 269.6 to 73.8 (95%), 1,800 items71.8%

Jev minus GLiDE, Oct 3 rerun: +20.4 points on the same 1,800 items, 95% interval +18.2 to +22.7: Jev is clearly ahead.

Jev minus GLiDE, Oct 2: +17.5 points on the same 1,800 items, 95% interval +15.4 to +19.6: Jev is clearly ahead.

Jev vs GLiDE, Oct 3 rerun and GLiDE, Oct 2 by proof depth

Each point is about 300 problems that need that many inference steps, with its 95% interval.

Open world: true, false or unknown
  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Jev minus GLiDE, Oct 3 rerun by depth, open world

depth 0300 items
0.0
depth 1302 items
+0.7
depth 2303 items
+5.0
depth 3303 items
+9.9
depth 4303 items
+10.6
depth 5289 items
+19.0

Jev minus GLiDE, Oct 3 rerun by depth, closed world

depth 0300 items
+0.3
depth 1300 items
+5.7
depth 2300 items
+23.7
depth 3300 items
+23.7
depth 4300 items
+24.0
depth 5300 items
+45.3

Jev minus GLiDE, Oct 2 by depth, open world

depth 0300 items
0.0
depth 1302 items
0.0
depth 2303 items
+0.7
depth 3303 items
+1.3
depth 4303 items
−2.0
depth 5289 items
+5.9

Jev minus GLiDE, Oct 2 by depth, closed world

depth 0300 items
0.0
depth 1300 items
+3.7
depth 2300 items
+20.0
depth 3300 items
+20.7
depth 4300 items
+20.7
depth 5300 items
+40.0

Paired differences: positive means Jev was more accurate on the same items.

Accuracy by depth, open world

By depth, Open world
Proof depthItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2Best constant
depth 030097.7% 96.0–99.397.7% 95.7–99.097.7% 95.7–99.033.3%
depth 130287.7% 83.8–91.487.1% 83.4–90.787.7% 84.1–91.433.4%
depth 230384.8% 80.2–88.479.9% 75.2–84.284.2% 80.2–88.133.3%
depth 330379.9% 75.2–84.570.0% 64.4–74.978.5% 73.6–83.233.3%
depth 430371.9% 66.7–77.261.4% 55.8–66.373.9% 69.3–78.533.3%
depth 528981.0% 76.5–85.561.9% 56.1–67.575.1% 69.6–79.934.9%

Accuracy by depth, closed world

By depth, Closed world
Proof depthItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2Best constant
depth 030099.3% 98.3–100.099.0% 97.7–100.099.3% 98.3–100.050.0%
depth 130093.7% 91.0–96.388.0% 84.3–91.790.0% 86.7–93.350.0%
depth 230092.0% 88.7–94.768.3% 63.0–73.772.0% 67.0–77.350.0%
depth 330085.0% 81.3–89.361.3% 55.7–66.764.3% 59.0–69.750.0%
depth 430076.3% 71.7–81.052.3% 47.0–58.055.7% 50.7–61.050.0%
depth 530089.3% 85.7–92.344.0% 38.3–49.349.3% 43.7–55.050.0%

What GLiDE, Oct 3 rerun got right and wrong, depth by depth

At each proof depth, GLiDE, Oct 3 rerun's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

97.7%
right on 293 of 300 problems wrong; chance 33% (dashed line)
right

Charlie is not kind.

Correct answer: unknown · Model said: unknown (97.6% sure)

paraphrased
The problem and its proof

Eric is feeling cold -- he is also feeling quite rough, red, nice and round. Rough and cold that is what they say about Blue Harry. When a person is nice and round and cold, they look blue. I feel that all round, cold and green people are that way because they are too young to know better. A person feeling blue because of their big size will always be cold. A big, round person who is kind is also red. Nice, young red people will also turn out to always be green. A red, nice person will definitely be a blue person. Why is it that every round person that is blue is so young.

No proof: neither the statement nor its negation follows.

right

Gary is furry.

Correct answer: unknown · Model said: unknown (90.3% sure)

The problem and its proof

Bob is red. Dave is big. Dave is not furry. Dave is red. Dave is young. Fiona is young. Gary is not red. If Bob is smart then Bob is red. If Fiona is rough then Fiona is furry. If Fiona is smart then Fiona is blue. Blue things are red. If Fiona is blue then Fiona is young. All red things are rough. Young things are smart.

No proof: neither the statement nor its negation follows.

right

The cat does not like the cat.

Correct answer: unknown · Model said: unknown (99.3% sure)

The problem and its proof

The cat is big. The cat is cold. The cat is red. The cat is rough. The cat is round. Rough things are red.

No proof: neither the statement nor its negation follows.

right

Dave is not young.

Correct answer: false · Model said: false (82.2% sure)

paraphrased
The problem and its proof

Bob is kind. He is also very cold and blue. Look, we know Dave is young and rough. We have to accept he's also red, nice, cold, and blue. After Harry got wet in the rain, he feels cold. He also looks green but big. Rough, cold people are blue. If a person happens to be big kind and red at the same time they are a young person. A kind person will certainly be rough as well. Do not fear big cold people for they are often kind. A person who is rough and young while being kind tends to be red.

Stated in the text: "Dave is young."

Depth 1, open-world task

87.1%
right on 263 of 302 problems wrong; chance 33% (dashed line)
wrong

Anne is not big.

Correct answer: unknown · Model said: true (71.2% sure)

The problem and its proof

Anne is not smart. Anne is young. Dave is white. Erin is furry. Erin is white. Gary is smart. Gary is white. Young things are red. All young things are red. White, red things are quiet. All furry things are quiet. White things are smart. If something is smart then it is big. All big, red things are furry. If something is smart then it is furry. Quiet things are young.

No proof: neither the statement nor its negation follows.

right

Bob is not blue.

Correct answer: unknown · Model said: unknown (69.7% sure)

The problem and its proof

Anne is big. Anne is blue. Anne is kind. Anne is quiet. Anne is young. Bob is big. Bob is kind. Bob is young. Harry is big. Harry is blue. Harry is cold. Harry is furry. Harry is kind. Harry is quiet. Harry is young. If Bob is quiet and Bob is kind then Bob is blue. If someone is quiet then they are young.

No proof: neither the statement nor its negation follows.

right

Harry is blue.

Correct answer: true · Model said: true (66.2% sure)

paraphrased
The problem and its proof

Alan seems to be round. Eric is a kind person and he is also often cold. Gary rather resembles the rainbow, as he is green, red and blue. Gary is also said to be very kind. Harry who frequently wears red clothes always feels very cold because of his young age on the team and his rough skin. A person who is kind, cold, and rough is big. Truth be told, every big, rough person out there that is green, will also be colored blue. Young people are so rough that they can hold their breath until they are blue. Blue, big and kind all at once, that person must be green too. Nice, young red people will also turn out to always be green. Someone that is cold rough and red is also considered to be kind.

  1. Harry is rough. + Harry is young.If someone is rough and young then they are blue.Harry is blue.
right

Eric is not red.

Correct answer: false · Model said: false (78.8% sure)

paraphrased
The problem and its proof

Bob is green and cold too. That guy Dave sure is nice. Round, nice Eric felt as blue and big as he had ever been. That guy Gary sure is nice. Folks who are rough and tumble yet gentle and kind are blue. Being young and cold also means you're green. All those who are nice and big are red. Nice, young red people will also turn out to always be green. A kind person will certainly be rough as well. A person who is blue, red and nice is usually considered to be a young person. A kind person will certainly be young.

The proof that the statement's negation follows:

  1. Eric is nice. + Eric is big.If someone is nice and big then they are red.Eric is red.

Depth 2, open-world task

79.9%
right on 242 of 303 problems wrong; chance 33% (dashed line)
right

Erin is green.

Correct answer: unknown · Model said: unknown (73.9% sure)

The problem and its proof

Charlie is big. Charlie is kind. Charlie is nice. Charlie is rough. Dave is green. Dave is kind. Dave is nice. Erin is cold. Erin is red. Harry is kind. Harry is nice. Harry is rough. If something is green then it is rough. If something is rough and nice then it is kind. If something is kind then it is green.

No proof: neither the statement nor its negation follows.

right

Gary is not cold.

Correct answer: false · Model said: false (72.1% sure)

paraphrased
The problem and its proof

Bob is a kind person and he is also often cold. Dave is a big fellow, often blue and sad, but he is nice. Eric is young and green. This makes him cold to others. Even though Gary is round and big, he is very kind. People here that are both big and kind while being cold are always red. Red people are almost always rough. Big, kind folks are green ones. A round shaped kind person who is colored green will be cold natured.

The proof that the statement's negation follows:

  1. Gary is big. + Gary is kind.If someone is big and kind then they are green.Gary is green.
  2. + Gary is round. + Gary is kind.If someone is round and green and kind then they are cold.Gary is cold.
wrong

The bald eagle is red.

Correct answer: false · Model said: unknown (89.6% sure)

The problem and its proof

The bald eagle chases the lion. The bald eagle chases the rabbit. The bald eagle is not big. The bald eagle is round. The bald eagle is young. The bald eagle needs the lion. The bald eagle needs the rabbit. The bald eagle visits the lion. The lion is round. The lion needs the bald eagle. The lion needs the rabbit. The lion visits the rabbit. The rabbit does not chase the bald eagle. The rabbit chases the lion. The rabbit is not blue. The rabbit visits the bald eagle. If someone visits the rabbit then they are not red. If someone chases the bald eagle and they are red then the bald eagle visits the rabbit. If someone needs the bald eagle then the bald eagle visits the rabbit. If the bald eagle is blue then the bald eagle visits the rabbit. If someone needs the rabbit then they need the bald eagle. If someone needs the bald eagle then the bald eagle visits the lion. If someone visits the rabbit then the rabbit needs the bald eagle. If someone chases the lion and they are not red then the lion does not visit the bald eagle.

The proof that the statement's negation follows:

  1. The lion needs the bald eagle.If someone needs the bald eagle then the bald eagle visits the rabbit.The bald eagle visits the rabbit.
  2. If someone visits the rabbit then they are not red.The bald eagle is not red.
right

Alan is not cold.

Correct answer: unknown · Model said: unknown (75.5% sure)

paraphrased
The problem and its proof

Green, big and red are qualities which all describe Alan. Bob seems to be round. Eric is cold but nice with red hair and dresses green. Harry may be round, but he is also kind. People,though kind and rough are also really nice. A kind person will certainly be rough as well. A person that is both nice and rough is someone who is also big. Anyone rough in texture and young in age is a cold person. I bet anyone who is round and kind is also pretty young.

No proof: neither the statement nor its negation follows.

Depth 3, open-world task

70.0%
right on 212 of 303 problems wrong; chance 33% (dashed line)
wrong

Charlie is not kind.

Correct answer: unknown · Model said: false (80.3% sure)

paraphrased
The problem and its proof

Bob is green and cold too. Charlie is a big fellow, often blue and sad, but he is nice. Gary seems to be round. Harry, a kind young person is always feeling blue. People,though kind and rough are also really nice. Kind, round people that are really feeling blue are going to always be big. All young and kind people that feel blue are described as red. Someone who is currently blue, big, and cold, is also kind. Young and red people look round. If a person is blue and red and big, then that person is cold.

No proof: neither the statement nor its negation follows.

right

The squirrel is not cold.

Correct answer: false · Model said: false (63.1% sure)

The problem and its proof

The bear is green. The bear is round. The bear likes the cow. The bear sees the lion. The cow eats the bear. The cow is rough. The cow is young. The cow likes the bear. The lion sees the squirrel. The squirrel eats the bear. The squirrel is rough. The squirrel sees the bear. If something eats the lion and it sees the lion then the lion likes the squirrel. If something is cold then it eats the lion. If something eats the cow and it eats the bear then the cow sees the lion. If something is green then it is cold. If something sees the squirrel then the squirrel sees the bear. If something sees the bear then it is young. If the cow likes the bear and the cow eats the lion then the bear is rough. If something is rough then it likes the lion. Young things are green.

The proof that the statement's negation follows:

  1. The squirrel sees the bear.If something sees the bear then it is young.The squirrel is young.
  2. Young things are green.The squirrel is green.
  3. If something is green then it is cold.The squirrel is cold.
right

Gary is rough.

Correct answer: true · Model said: true (78.6% sure)

paraphrased
The problem and its proof

Charlie is feeling cold -- he is also feeling quite rough, red, nice and round. Fred seems to be round. A bit overweight, Gary is big and round. He's a nice guy who is often cold. Folks who are rough and tumble yet gentle and kind are blue. A rather big person is a red person. A kind person will certainly be rough as well. Anyone having rough, cold and green qualities will also have a big quality. People who are young and blue are also red. People who are said to be big and nice are round. Cold and red people are always kind to others.

  1. Gary is big.If someone is big then they are red.Gary is red.
  2. + Gary is cold.If someone is cold and red then they are kind.Gary is kind.
  3. If someone is kind then they are rough.Gary is rough.
right

The squirrel is red.

Correct answer: true · Model said: true (69.9% sure)

The problem and its proof

The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.

  1. The squirrel is kind.All kind people are big.The squirrel is big.
  2. + The squirrel is kind.Big, kind people are green.The squirrel is green.
  3. Green, big people are red.The squirrel is red.

Depth 4, open-world task

61.4%
right on 186 of 303 problems wrong; chance 33% (dashed line)
right

Anne is nice.

Correct answer: unknown · Model said: unknown (86.8% sure)

The problem and its proof

Anne is young. Charlie is not young. Dave is red. Erin is big. All furry, white things are nice. If something is red then it is nice. If something is kind then it is big. All white things are kind. Nice things are furry. All big things are red.

No proof: neither the statement nor its negation follows.

right

Fiona is young.

Correct answer: unknown · Model said: unknown (79.4% sure)

The problem and its proof

Erin is green. Fiona is nice. Harry is kind. All young, green things are white. Kind things are green. All blue things are young. Kind things are not red. All green things are blue. If something is red then it is kind.

No proof: neither the statement nor its negation follows.

wrong

The cow visits the cow.

Correct answer: true · Model said: unknown (95.5% sure)

The problem and its proof

The bear is rough. The bear needs the dog. The bear visits the cat. The bear visits the dog. The cat is blue. The cat likes the cow. The cow is rough. The cow likes the bear. The dog needs the bear. The dog visits the cat. All blue people are big. If someone visits the cow then they need the cow. If someone is nice then they like the cow. If someone likes the cat and they are nice then they are big. If someone is blue then they visit the cow. If the cat needs the cow then the cow is blue.

  1. The cat is blue.If someone is blue then they visit the cow.The cat visits the cow.
  2. If someone visits the cow then they need the cow.The cat needs the cow.
  3. If the cat needs the cow then the cow is blue.The cow is blue.
  4. If someone is blue then they visit the cow.The cow visits the cow.
wrong

The bald eagle does not see the cat.

Correct answer: false · Model said: unknown (66.4% sure)

The problem and its proof

The bald eagle is young. The bald eagle likes the bear. The bear is blue. The cat is blue. The cat likes the bald eagle. The cat sees the cow. The cat visits the bald eagle. The cow is blue. The cow is nice. The cow is round. The cow does not like the bear. The cow sees the bear. If someone sees the bald eagle then the bald eagle sees the cow. If someone sees the cow then they see the cat. If someone sees the cat then they like the cow. If someone sees the cat then the cat sees the bald eagle. If someone likes the cow and the cow is round then the cow does not see the bald eagle.

The proof that the statement's negation follows:

  1. The cat sees the cow.If someone sees the cow then they see the cat.The cat sees the cat.
  2. If someone sees the cat then the cat sees the bald eagle.The cat sees the bald eagle.
  3. If someone sees the bald eagle then the bald eagle sees the cow.The bald eagle sees the cow.
  4. If someone sees the cow then they see the cat.The bald eagle sees the cat.

Depth 5, open-world task

61.9%
right on 179 of 289 problems wrong; chance 33% (dashed line)
wrong

The mouse is not round.

Correct answer: false · Model said: unknown (79.1% sure)

The problem and its proof

The bald eagle eats the tiger. The mouse visits the bald eagle. The squirrel is kind. The squirrel needs the bald eagle. The squirrel needs the tiger. The tiger eats the squirrel. The tiger is red. If someone is kind then they visit the tiger. If someone eats the mouse then the mouse visits the tiger. If the tiger eats the mouse then the tiger needs the mouse. If someone visits the tiger then they are round. If the squirrel is round then the squirrel eats the mouse. If the squirrel visits the tiger then the tiger is round. If someone is round then they visit the tiger.

The proof that the statement's negation follows:

  1. The squirrel is kind.If someone is kind then they visit the tiger.The squirrel visits the tiger.
  2. If someone visits the tiger then they are round.The squirrel is round.
  3. If the squirrel is round then the squirrel eats the mouse.The squirrel eats the mouse.
  4. If someone eats the mouse then the mouse visits the tiger.The mouse visits the tiger.
  5. If someone visits the tiger then they are round.The mouse is round.
wrong

Harry is white.

Correct answer: false · Model said: unknown (60.1% sure)

The problem and its proof

Charlie is not blue. Charlie is round. Dave is blue. Dave is smart. Gary is smart. Gary is white. Harry is round. If someone is round and smart then they are quiet. Quiet people are round. If someone is nice then they are red. If Harry is smart then Harry is blue. If someone is round then they are nice. If someone is white then they are nice. All red, round people are smart. Quiet, smart people are not white.

The proof that the statement's negation follows:

  1. Harry is round.If someone is round then they are nice.Harry is nice.
  2. If someone is nice then they are red.Harry is red.
  3. + Harry is round.All red, round people are smart.Harry is smart.
  4. + Harry is round.If someone is round and smart then they are quiet.Harry is quiet.
  5. Quiet, smart people are not white.Harry is not white.
right

Charlie is white.

Correct answer: unknown · Model said: unknown (82.5% sure)

The problem and its proof

Bob is green. Bob is kind. Charlie is nice. Erin is kind. Erin is nice. Erin is white. Gary is young. All young things are white. White things are young. All nice, green things are kind. If something is young then it is furry. All kind things are nice. All nice, green things are white. If something is cold and kind then it is green. If something is nice and young then it is furry. Furry things are cold.

No proof: neither the statement nor its negation follows.

right

Fiona is not red.

Correct answer: unknown · Model said: unknown (63.9% sure)

The problem and its proof

Erin is cold. Fiona is smart. If something is cold then it is rough. All rough things are red. Red things are kind. All cold things are blue. If Fiona is cold then Fiona is big. All rough, blue things are red. If something is blue and smart then it is red. Kind things are blue.

No proof: neither the statement nor its negation follows.

The closed-world task (true or false), depth by depth

Depth 0, closed-world task

99.0%
right on 297 of 300 problems wrong; chance 50% (dashed line)
right

The squirrel is not green.

Correct answer: true · Model said: true (95.0% sure)

The problem and its proof

The bear chases the lion. The bear eats the lion. The bear is kind. The bear is rough. The bear needs the lion. The bear needs the squirrel. The lion chases the bear. The lion is rough. The lion is young. The lion needs the squirrel. The squirrel chases the lion. The squirrel is young. If something is young then it is rough. If something is kind then it needs the bear. Rough things are kind.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Dave is not big.

Correct answer: false · Model said: false (80.8% sure)

The problem and its proof

Anne is blue. Anne is green. Charlie is blue. Charlie is nice. Dave is big. Dave is blue. Dave is cold. Dave is green. Dave is quiet. Gary is big. Gary is blue. Gary is cold. Gary is kind. Gary is nice. Big people are nice. If someone is kind then they are big. If Gary is kind then Gary is cold. Green people are quiet. If someone is nice then they are quiet. All blue, quiet people are kind. If someone is big then they are kind. All big people are cold. If someone is cold and quiet then they are green.

Stated in the text: "Dave is big."

right

Bob is not rough.

Correct answer: true · Model said: true (87.4% sure)

paraphrased
The problem and its proof

That guy Bob sure is nice. Dave is known for being big and rough, but few know he is also always red with cold. Rough and cold that is what they say about Blue Gary. Harry is somehow red and kind and also green. Kind people that are red and round are usually big. Not a single young person is not also kind. Nice people who are blue and round at the same time are always young. Big people with red hair are cold because they cannot find coats that fit. Most young kind people tend to be red too. If someone is cold and blue, you will find that they are also quite young. Young and red people look round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The mouse eats the rabbit.

Correct answer: true · Model said: true (95.8% sure)

The problem and its proof

The mouse chases the rabbit. The mouse eats the rabbit. The mouse is green. The mouse is nice. The mouse is round. The mouse is young. The mouse likes the rabbit. The rabbit chases the mouse. The rabbit eats the mouse. The rabbit likes the mouse. If the mouse is rough and the mouse does not chase the rabbit then the mouse eats the rabbit. If someone is nice then they chase the mouse. If someone is nice and they do not chase the mouse then they like the mouse.

Stated in the text: "The mouse eats the rabbit."

Depth 1, closed-world task

88.0%
right on 264 of 300 problems wrong; chance 50% (dashed line)
right

The cat needs the bear.

Correct answer: true · Model said: true (70.6% sure)

The problem and its proof

The bear needs the tiger. The bear sees the dog. The bear sees the tiger. The cat sees the tiger. The cat visits the tiger. The dog is red. The dog needs the bear. The dog sees the bear. The dog sees the cat. The dog visits the tiger. The tiger is big. If something visits the bear then the bear sees the cat. If something visits the tiger then it needs the bear. If something sees the tiger then the tiger visits the cat. If something visits the cat then it is rough. If something is rough and it does not need the cat then it needs the tiger. If something needs the tiger and the tiger is big then it is round. If something sees the dog then the dog needs the tiger. If something is round then it is cold.

  1. The cat visits the tiger.If something visits the tiger then it needs the bear.The cat needs the bear.
right

The cow does not see the lion.

Correct answer: true · Model said: true (78.8% sure)

The problem and its proof

The cow needs the lion. The lion eats the cow. The lion is rough. If something is nice and round then it needs the cow. If the cow needs the lion and the lion does not eat the cow then the cow sees the lion.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The dog likes the lion.

Correct answer: false · Model said: false (84.4% sure)

The problem and its proof

The cow visits the dog. The dog is red. The lion sees the cow. The squirrel is red. If something likes the dog and the dog likes the lion then the lion sees the dog. If something visits the dog then it likes the lion.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The bear is not big.

Correct answer: false · Model said: false (83.5% sure)

The problem and its proof

The bear is green. If someone is big then they are green. If the bear is green then the bear is big. If the bear is kind and the bear is red then the bear is big. Green people are big. Red people are big. Green, nice people are red. All nice people are red. All green, big people are kind.

  1. The bear is green.If the bear is green then the bear is big.The bear is big.

Depth 2, closed-world task

68.3%
right on 205 of 300 problems wrong; chance 50% (dashed line)
wrong

The lion does not see the cow.

Correct answer: false · Model said: true (90.5% sure)

The problem and its proof

The cat chases the cow. The cow chases the lion. The cow is round. The cow sees the lion. The lion chases the mouse. The lion is red. The mouse sees the cat. If someone chases the mouse then they chase the cat. If someone chases the cat then they see the cow. If someone is blue then they chase the cow. If someone chases the lion and they eat the mouse then the lion eats the cat. If someone sees the cow then the cow chases the mouse. If the cat sees the mouse and the cat is not red then the mouse eats the cat.

  1. The lion chases the mouse.If someone chases the mouse then they chase the cat.The lion chases the cat.
  2. If someone chases the cat then they see the cow.The lion sees the cow.
right

The cow eats the cow.

Correct answer: false · Model said: false (94.7% sure)

The problem and its proof

The bear eats the mouse. The bear likes the mouse. The cow eats the mouse. The cow is big. The dog eats the bear. The mouse eats the cow. The mouse is kind. If someone is rough then they need the cow. If someone needs the cow then they need the bear. If someone is kind then they eat the dog. If someone needs the dog then they eat the dog. If someone likes the cow and the cow eats the dog then the cow is green. If someone is green then they eat the cow. If someone eats the mouse and they need the cow then the cow is rough. If someone needs the cow then they eat the mouse. If the cow eats the mouse and the cow does not need the mouse then the mouse is rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Dave is not green.

Correct answer: true · Model said: true (91.0% sure)

paraphrased
The problem and its proof

Bob is kind. He is also very cold and blue. Fred is a round and rough around the edges, and he is also big. Gary has a round shape and is known to be cold and rough around the edges; however, he can also be kind. Every single big person is a little green in some areas. Rough, cold people are blue. A person who is blue, red and nice is usually considered to be a young person. Cold people that are big and red are usually nice. A kind person who looks blue because he is is cold is usually big in stature.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The lion is not green.

Correct answer: true · Model said: true (91.5% sure)

The problem and its proof

The lion eats the squirrel. The rabbit is kind. The squirrel sees the lion. If something eats the lion and it is kind then the lion sees the squirrel. If the lion is big then the lion sees the squirrel. If something is rough then it is green. If something sees the rabbit then the rabbit sees the lion. If something eats the squirrel then the squirrel sees the rabbit. If something eats the rabbit and it is kind then it is rough. If something is kind then it eats the rabbit. If something eats the squirrel and it is not green then the squirrel visits the lion.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 3, closed-world task

61.3%
right on 184 of 300 problems wrong; chance 50% (dashed line)
right

Harry is green.

Correct answer: true · Model said: true (75.8% sure)

The problem and its proof

Charlie is cold. Charlie is quiet. Charlie is young. Dave is furry. Dave is smart. Erin is furry. Harry is cold. If someone is red and furry then they are cold. Young, cold people are red. All quiet people are green. Cold people are smart. Green, young people are smart. All young, green people are smart. If Harry is green then Harry is young. All cold, smart people are furry. If someone is smart then they are quiet.

  1. Harry is cold.Cold people are smart.Harry is smart.
  2. If someone is smart then they are quiet.Harry is quiet.
  3. All quiet people are green.Harry is green.
right

Anne is not furry.

Correct answer: false · Model said: false (67.9% sure)

The problem and its proof

Anne is kind. Bob is green. Charlie is round. Harry is furry. All rough people are green. All round, kind people are rough. Rough, cold people are nice. All green people are furry. If someone is kind then they are rough. All furry, round people are rough.

  1. Anne is kind.If someone is kind then they are rough.Anne is rough.
  2. All rough people are green.Anne is green.
  3. All green people are furry.Anne is furry.
wrong

Gary is not green.

Correct answer: false · Model said: true (89.9% sure)

paraphrased
The problem and its proof

Round, nice Bob felt as blue and big as he had ever been. That guy Eric sure is nice. Gary who frequently wears red clothes always feels very cold because of his young age on the team and his rough skin. Harry is a kind person and he is also often cold. If you're truly nice, cold, and blue, you're kind, too. It's pretty typical of red, rough, and cold people to be round. A round person who is on the young side is also likely to treat people in a kind manner. Someone who is a young age and looks like they are round are also red. Jan is nice and rough, she can be blue in the winter which makes her cold. Kind red and round people often look green.

  1. Gary is red. + Gary is rough. + Gary is cold.If someone is red and rough and cold then they are round.Gary is round.
  2. + Gary is young.If someone is round and young then they are kind.Gary is kind.
  3. + Gary is red.If someone is red and kind and round then they are green.Gary is green.
wrong

The bear does not see the bear.

Correct answer: false · Model said: true (81.8% sure)

The problem and its proof

The bald eagle eats the bear. The bald eagle is blue. The bald eagle is nice. The bald eagle is young. The bald eagle likes the bear. The bear is nice. The bear likes the bald eagle. If something sees the bear then the bear is young. If something is young then it sees the bear. If something sees the bear and the bear likes the bald eagle then the bear eats the bald eagle.

  1. The bald eagle is young.If something is young then it sees the bear.The bald eagle sees the bear.
  2. If something sees the bear then the bear is young.The bear is young.
  3. If something is young then it sees the bear.The bear sees the bear.

Depth 4, closed-world task

52.3%
right on 157 of 300 problems wrong; chance 50% (dashed line)
wrong

Bob is round.

Correct answer: true · Model said: false (74.3% sure)

paraphrased
The problem and its proof

For being so cold, it's good Alan can remain nice. Bob is green, but often kind, even when he is blue and cold. Eric is a big guy who is colored red from a sunburn. He also has a cold. After Harry got wet in the rain, he feels cold. He also looks green but big. People here that are both big and kind while being cold are always red. A person who is kind, cold, and rough is big. Young, red people are usually quite blue too. A blue, cold natured person will be rough textured. I've noticed that nice people who have rough, green skin have a tendancy to be round. If someone is green and naive they may also have red, rough skin. It is a safe guess then that they are also round. Nice people who feel blue often look green when ill.

  1. Bob is blue. + Bob is cold.If someone is blue and cold then they are rough.Bob is rough.
  2. + Bob is kind. + Bob is cold.If someone is kind and cold and rough then they are big.Bob is big.
  3. + Bob is kind. + Bob is cold.If someone is big and kind and cold then they are red.Bob is red.
  4. + Bob is green.If someone is red and rough and green then they are round.Bob is round.
right

The bald eagle is kind.

Correct answer: true · Model said: true (77.7% sure)

The problem and its proof

The bald eagle visits the squirrel. The cow is rough. The cow is round. The cow likes the bald eagle. The cow likes the tiger. The cow needs the bald eagle. The cow needs the squirrel. The cow needs the tiger. The squirrel likes the bald eagle. The squirrel likes the cow. The squirrel visits the cow. The tiger is red. The tiger needs the squirrel. The tiger visits the squirrel. If something visits the bald eagle then the bald eagle is kind. If something is red and cold then it likes the tiger. If the cow visits the bald eagle and the cow is red then the bald eagle visits the squirrel. If something visits the cow and it is cold then it visits the bald eagle. If something needs the tiger then it is cold. If something likes the bald eagle and the bald eagle is kind then the bald eagle is cold. If something likes the squirrel and it visits the bald eagle then the squirrel needs the tiger. If something needs the cow then it is red. If something is cold then it visits the cow.

  1. The cow needs the tiger.If something needs the tiger then it is cold.The cow is cold.
  2. If something is cold then it visits the cow.The cow visits the cow.
  3. If something visits the cow and it is cold then it visits the bald eagle.The cow visits the bald eagle.
  4. If something visits the bald eagle then the bald eagle is kind.The bald eagle is kind.
right

Harry is white.

Correct answer: false · Model said: false (71.9% sure)

The problem and its proof

Erin is blue. Erin is furry. Harry is red. All kind people are quiet. Furry people are quiet. If someone is white and red then they are blue. All kind people are blue. If someone is quiet and blue then they are red. Blue, red people are kind. All blue people are white. If Harry is not kind and Harry is not quiet then Harry is young.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The rabbit does not visit the lion.

Correct answer: false · Model said: true (84.4% sure)

The problem and its proof

The bald eagle is blue. The bald eagle is red. The bald eagle likes the dog. The bald eagle visits the lion. The dog needs the lion. The lion needs the bald eagle. The rabbit needs the dog. If something needs the dog and the dog is red then it is red. If something visits the lion then the lion needs the bald eagle. If something is red then it visits the rabbit. If something is red and it does not visit the bald eagle then it visits the lion. All red things are green. If the lion needs the dog and the lion needs the rabbit then the dog likes the lion. If something visits the rabbit and the rabbit is red then it needs the bald eagle. If something visits the dog then the dog is red. If something needs the dog then it visits the dog.

  1. The rabbit needs the dog.If something needs the dog then it visits the dog.The rabbit visits the dog.
  2. If something visits the dog then the dog is red.The dog is red.
  3. + The rabbit needs the dog.If something needs the dog and the dog is red then it is red.The rabbit is red.
  4. If something is red and it does not visit the bald eagle then it visits the lion.The rabbit visits the lion.

Depth 5, closed-world task

44.0%
right on 132 of 300 problems wrong; chance 50% (dashed line)
right

Charlie is round.

Correct answer: true · Model said: true (76.6% sure)

The problem and its proof

Charlie is kind. Dave is rough. Dave is white. Erin is kind. Erin is red. Fiona is furry. Fiona is kind. Fiona is red. Fiona is rough. Fiona is white. All kind things are furry. All rough, kind things are white. If something is rough then it is smart. Furry things are rough. If Charlie is smart and Charlie is kind then Charlie is red. Round, smart things are red. Red things are round.

  1. Charlie is kind.All kind things are furry.Charlie is furry.
  2. Furry things are rough.Charlie is rough.
  3. If something is rough then it is smart.Charlie is smart.
  4. + Charlie is kind.If Charlie is smart and Charlie is kind then Charlie is red.Charlie is red.
  5. Red things are round.Charlie is round.
wrong

Harry is not kind.

Correct answer: false · Model said: true (81.8% sure)

paraphrased
The problem and its proof

Alan is kind and nice and looks green. That guy Gary sure is nice. Young Harry here is always rough but nice and yes he is big. When someone is kind yet can be cold and blue, they will also be very big. A nice young person who is big can be considered round. A red, nice person will definitely be a blue person. It's pretty typical of red, rough, and cold people to be round. Young people are nice and cold are kind to people regardless. Someone who is a young age and looks like they are round are also red. Any person who can be red and blue at the same time is cold.

  1. Harry is nice. + Harry is big. + Harry is young.If someone is nice and big and young then they are round.Harry is round.
  2. + Harry is young.If someone is young and round then they are red.Harry is red.
  3. + Harry is nice.If someone is red and nice then they are blue.Harry is blue.
  4. If someone is red and blue then they are cold.Harry is cold.
  5. + Harry is nice. + Harry is young.If someone is cold and nice and young then they are kind.Harry is kind.
wrong

Erin is rough.

Correct answer: false · Model said: true (73.1% sure)

The problem and its proof

Dave is blue. Dave is rough. Dave is young. Erin is blue. Erin is round. Erin is young. Fiona is rough. Gary is green. Gary is nice. Gary is young. Rough things are round. Green, nice things are quiet. Round, rough things are green. All quiet things are green. All green things are nice. All quiet things are rough. All rough, nice things are quiet. All quiet, green things are rough. If something is nice and quiet then it is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Charlie is furry.

Correct answer: false · Model said: false (90.5% sure)

The problem and its proof

Charlie is green. Erin is cold. Erin is kind. Erin is quiet. Fiona is cold. Fiona is furry. Fiona is green. Fiona is kind. Fiona is white. Gary is cold. Quiet people are furry. Green, furry people are white. All white people are quiet. If someone is furry then they are round. Kind people are cold. All cold people are white. Quiet, round people are green. Round, white people are cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

What GLiDE, Oct 2 got right and wrong, depth by depth

At each proof depth, GLiDE, Oct 2's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

97.7%
right on 293 of 300 problems wrong; chance 33% (dashed line)
right

Anne is round.

Correct answer: unknown · Model said: unknown (83.6% sure)

The problem and its proof

Anne is not kind. Anne is quiet. Dave is not green. Dave is rough. Dave is round. Dave is smart. Fiona is quiet. Fiona is round. Gary is smart. Gary is young. Round people are rough. If someone is smart then they are rough. All young, rough people are smart. If someone is rough then they are kind. Rough people are kind. If Fiona is smart and Fiona is kind then Fiona is green. If Anne is round then Anne is quiet. Round, rough people are quiet. All kind people are young.

No proof: neither the statement nor its negation follows.

right

Charlie is green.

Correct answer: true · Model said: true (98.7% sure)

paraphrased
The problem and its proof

Charlie is young and green. Even though he is round he is cold outside. Dave is a kind person and he is also often cold. Eric is kind and nice and looks green. Gary is a kind person and he is also often cold. If a kind person is green and a bit blue then they are also nice. If someone plays rough and is nice and round, they will be big. People of young age are rough and cold but some are nice. Most round and cold people are often rough. When you meet and big and nice man, they will be kind to the earth and live green lifestyle. Nice people who feel blue often look green when ill.

Stated in the text: "Charlie is green."

right

Harry is green.

Correct answer: false · Model said: false (97.8% sure)

The problem and its proof

Charlie is quiet. Gary is not quiet. Harry is not green. If someone is nice and not cold then they are white.

Stated in the text: "Harry is not green."

right

Eric is big.

Correct answer: true · Model said: true (98.6% sure)

paraphrased
The problem and its proof

Alan is green and cold too. Bob who is young and nice is also quite big and feels green. Eric is a big fellow, often blue and sad, but he is nice. Harry was proud of being round, yet rough. His red cheeks glowed. When a person is nice and round and cold, they look blue. Rough people who are red in color and big are usally round in shape. A rather big person is a red person. Kind people with rough skin are usually red because it's wind burn. Kind red and round people often look green. Big people who are blue and red are afflicted with rough skin. Rough people who look red are cold.

Stated in the text: "Eric is big."

Depth 1, open-world task

87.7%
right on 265 of 302 problems wrong; chance 33% (dashed line)
right

Bob is not rough.

Correct answer: true · Model said: true (99.2% sure)

The problem and its proof

Bob is young. Erin is rough. Harry is red. All red people are young. If someone is young then they are not rough.

  1. Bob is young.If someone is young then they are not rough.Bob is not rough.
right

Bob is not round.

Correct answer: true · Model said: true (92.1% sure)

The problem and its proof

Anne is cold. Anne is furry. Anne is not kind. Anne is round. Anne is not young. Bob is cold. Bob is furry. Bob is red. Bob is young. Harry is blue. Harry is cold. Harry is furry. Harry is kind. Harry is red. Harry is not round. Furry, young things are not round. If something is round and not furry then it is not red.

  1. Bob is furry. + Bob is young.Furry, young things are not round.Bob is not round.
right

Fred is not red.

Correct answer: unknown · Model said: unknown (79.6% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Bob woke up one day to find he had become round. He was still rough, red, and green, but now he was also big. Charlie may be young but he is nice, wears green shoes and is cold. That guy Fred sure is nice. Someone who has young, nice and cold qualities will possess a red quality, too. Interesting that all round people are cold. Cold and red people are always kind to others. Young people who feel cold and are still kind usually look rough.

No proof: neither the statement nor its negation follows.

right

The cat does not see the cow.

Correct answer: false · Model said: false (81.2% sure)

The problem and its proof

The cat needs the rabbit. The cat sees the rabbit. The cow chases the cat. The cow chases the rabbit. The cow is blue. The rabbit chases the cow. The rabbit is green. If someone sees the rabbit then they see the cow. If someone sees the cow then they are big.

The proof that the statement's negation follows:

  1. The cat sees the rabbit.If someone sees the rabbit then they see the cow.The cat sees the cow.

Depth 2, open-world task

84.2%
right on 255 of 303 problems wrong; chance 33% (dashed line)
right

The lion likes the cow.

Correct answer: unknown · Model said: unknown (83.7% sure)

The problem and its proof

The cow visits the squirrel. The lion is rough. The squirrel likes the tiger. The tiger visits the cow. If something is young and it chases the lion then it likes the squirrel. If something is cold then it likes the cow. If something likes the cow then it likes the tiger. If something is rough then it chases the squirrel. If something visits the cow then it is cold. If the cow does not like the tiger then the cow chases the tiger.

No proof: neither the statement nor its negation follows.

wrong

The bald eagle chases the lion.

Correct answer: true · Model said: unknown (89.3% sure)

The problem and its proof

The bald eagle chases the dog. The bald eagle chases the tiger. The bald eagle is red. The bald eagle is rough. The bald eagle needs the lion. The bald eagle sees the dog. The dog does not chase the lion. The dog is not rough. The dog sees the bald eagle. The dog sees the tiger. The lion is rough. The lion needs the bald eagle. The lion needs the dog. The lion needs the tiger. The lion sees the bald eagle. The tiger is red. If the lion sees the bald eagle and the bald eagle chases the tiger then the tiger is round. If someone sees the lion and the lion chases the dog then the dog needs the lion. If someone is kind and they chase the bald eagle then the bald eagle needs the dog. If someone sees the lion and the lion is rough then they chase the lion. If someone needs the dog and the dog is kind then the dog sees the bald eagle. If someone sees the bald eagle and the bald eagle chases the lion then the lion is red. If someone is red and rough then they see the lion. If someone needs the lion then the lion needs the dog. If someone is young then they are rough.

  1. The bald eagle is red. + The bald eagle is rough.If someone is red and rough then they see the lion.The bald eagle sees the lion.
  2. + The lion is rough.If someone sees the lion and the lion is rough then they chase the lion.The bald eagle chases the lion.
right

Bob is rough.

Correct answer: unknown · Model said: unknown (80.9% sure)

The problem and its proof

Anne is blue. Anne is round. Bob is smart. Erin is big. Erin is green. Harry is big. Harry is red. All round things are big. If something is big and smart then it is rough.

No proof: neither the statement nor its negation follows.

right

The tiger is not kind.

Correct answer: false · Model said: false (76.0% sure)

The problem and its proof

The squirrel is cold. The squirrel is kind. The squirrel is nice. The squirrel is young. The squirrel needs the tiger. The squirrel visits the tiger. The tiger is nice. The tiger needs the squirrel. The tiger sees the squirrel. The tiger visits the squirrel. If something is nice then it needs the squirrel. If the tiger needs the squirrel then the squirrel sees the tiger. If something needs the tiger then it sees the squirrel. If something is kind and it needs the squirrel then it sees the tiger. If something sees the squirrel and it sees the tiger then the tiger is kind. If something needs the squirrel then it is nice. If something visits the tiger then it is kind. If something sees the tiger then the tiger needs the squirrel.

The proof that the statement's negation follows:

  1. The squirrel needs the tiger.If something needs the tiger then it sees the squirrel.The squirrel sees the squirrel.
  2. + The tiger needs the squirrel.If the tiger needs the squirrel then the squirrel sees the tiger.The squirrel sees the tiger.
  3. If something sees the squirrel and it sees the tiger then the tiger is kind.The tiger is kind.

Depth 3, open-world task

78.5%
right on 238 of 303 problems wrong; chance 33% (dashed line)
right

Gary is rough.

Correct answer: unknown · Model said: unknown (77.1% sure)

paraphrased
The problem and its proof

Alan seems to be round. Young Bob while being blue because his is cold is round shaped while still maintaining a green thumb. The young person who is always feeling cold is named Gary. Kind red people are green on the inside. A person who is kind, cold, and rough is big. Young people who are cold to others and green with envy are actually nice. When green, young and round fits a person, you'll see that rough will also fit. Someone with rough and green feet is invariably kind. Even though a person is big and rough when they are cold they turn red. Most big and cold people are also rough.

No proof: neither the statement nor its negation follows.

wrong

The tiger is round.

Correct answer: false · Model said: unknown (76.6% sure)

The problem and its proof

The cow chases the squirrel. The lion chases the squirrel. The lion eats the squirrel. The lion does not eat the tiger. The lion does not need the squirrel. The squirrel chases the lion. The squirrel eats the lion. The tiger chases the squirrel. The tiger eats the lion. The tiger eats the squirrel. The tiger is cold. The tiger needs the squirrel. If something is green and rough then it chases the tiger. If something chases the squirrel then it is cold. If something eats the cow and it is green then it is not round. If something is cold and it eats the cow then the cow is green. If something is green then it eats the cow. If the tiger is cold then the tiger is green. If something eats the squirrel then the squirrel is green. If the cow needs the squirrel and the cow is not green then the squirrel does not need the tiger.

The proof that the statement's negation follows:

  1. The tiger is cold.If the tiger is cold then the tiger is green.The tiger is green.
  2. If something is green then it eats the cow.The tiger eats the cow.
  3. If something eats the cow and it is green then it is not round.The tiger is not round.
right

The bear needs the bear.

Correct answer: unknown · Model said: unknown (90.0% sure)

The problem and its proof

The bald eagle does not need the cat. The bear is rough. The cat needs the rabbit. The rabbit chases the bald eagle. If the bear eats the cat then the cat needs the rabbit. If something chases the bald eagle then it is big. If something chases the bear and the bear does not need the rabbit then it chases the bald eagle. If something chases the bear and it does not chase the cat then it eats the bear. If something chases the rabbit and the rabbit is big then the rabbit eats the bald eagle. If something is big then it needs the bear. If the rabbit needs the cat then the rabbit chases the bear. If something is rough and it does not chase the bear then it eats the rabbit.

No proof: neither the statement nor its negation follows.

right

The bear likes the squirrel.

Correct answer: true · Model said: true (65.3% sure)

The problem and its proof

The bald eagle likes the cow. The bald eagle visits the cow. The bear chases the bald eagle. The bear is rough. The cow is cold. The cow likes the bald eagle. The cow visits the bald eagle. The squirrel is young. The squirrel likes the bear. The squirrel visits the cow. If someone chases the squirrel then they like the squirrel. If someone is young and rough then they chase the squirrel. If someone likes the bear then they visit the bald eagle. If someone is cold then they chase the bald eagle. If someone likes the cow then they are blue. If someone visits the bald eagle and they like the squirrel then they like the bear. If someone is cold then they visit the bear. If someone chases the bald eagle then they are young. If someone is young then they chase the squirrel.

  1. The bear chases the bald eagle.If someone chases the bald eagle then they are young.The bear is young.
  2. If someone is young then they chase the squirrel.The bear chases the squirrel.
  3. If someone chases the squirrel then they like the squirrel.The bear likes the squirrel.

Depth 4, open-world task

73.9%
right on 224 of 303 problems wrong; chance 33% (dashed line)
right

Fiona is green.

Correct answer: unknown · Model said: unknown (77.5% sure)

The problem and its proof

Charlie is white. Fiona is smart. If something is blue then it is big. Big, young things are smart. Big things are green. All white things are blue. Round, big things are white. If Fiona is white and Fiona is blue then Fiona is young.

No proof: neither the statement nor its negation follows.

wrong

The mouse is big.

Correct answer: true · Model said: unknown (73.0% sure)

The problem and its proof

The dog is not round. The dog likes the lion. The dog needs the rabbit. The dog does not visit the lion. The lion is big. The lion does not like the mouse. The mouse is nice. The mouse likes the rabbit. The mouse does not need the lion. The mouse visits the rabbit. The rabbit is big. The rabbit is nice. The rabbit is not round. The rabbit likes the dog. The rabbit likes the lion. The rabbit does not visit the lion. If someone visits the mouse then they are big. If someone likes the mouse then the mouse likes the rabbit. If someone visits the mouse then the mouse needs the rabbit. If someone is nice then they need the mouse. If the dog likes the rabbit and the rabbit needs the lion then the rabbit likes the lion. If someone is big then they do not need the dog. If someone needs the rabbit then they visit the mouse.

  1. The dog needs the rabbit.If someone needs the rabbit then they visit the mouse.The dog visits the mouse.
  2. If someone visits the mouse then the mouse needs the rabbit.The mouse needs the rabbit.
  3. If someone needs the rabbit then they visit the mouse.The mouse visits the mouse.
  4. If someone visits the mouse then they are big.The mouse is big.
right

Harry is not cold.

Correct answer: false · Model said: false (56.9% sure)

paraphrased
The problem and its proof

Bob is a cold and round man who has red and green skin. No one knows Charlie like me and he is a kind guy, very round in the belly and green as grass. Fred seems to be round. Green, big and red are qualities which all describe Harry. A red hued and rough looking person is definitely a young person. A very big person who is green but also red is a rough person. A rather big person is a red person. A round person who is on the young side is also likely to treat people in a kind manner. Any person who can be red and blue at the same time is cold. People who are young are also blue.

The proof that the statement's negation follows:

  1. Harry is green. + Harry is big. + Harry is red.If someone is green and big and red then they are rough.Harry is rough.
  2. + Harry is red.If someone is red and rough then they are young.Harry is young.
  3. If someone is young then they are blue.Harry is blue.
  4. + Harry is red.If someone is red and blue then they are cold.Harry is cold.
right

The bald eagle visits the dog.

Correct answer: true · Model said: true (62.4% sure)

The problem and its proof

The bald eagle needs the dog. The bald eagle visits the bear. The bear is green. The bear is round. The dog is kind. The dog is round. The dog needs the bald eagle. The dog needs the bear. The dog sees the lion. The dog visits the lion. The lion is kind. If the bald eagle is round and the bald eagle visits the bear then the bald eagle sees the lion. If the bald eagle visits the dog then the dog sees the bear. If someone visits the dog then they see the dog. If the lion needs the dog and the dog sees the bald eagle then the dog needs the bear. If someone is kind and they visit the lion then they see the bald eagle. If someone is kind then they see the dog. If someone sees the lion then they visit the dog. If someone is kind and they see the bald eagle then the bald eagle is round.

  1. The dog is kind. + The dog visits the lion.If someone is kind and they visit the lion then they see the bald eagle.The dog sees the bald eagle.
  2. + The dog is kind.If someone is kind and they see the bald eagle then the bald eagle is round.The bald eagle is round.
  3. + The bald eagle visits the bear.If the bald eagle is round and the bald eagle visits the bear then the bald eagle sees the lion.The bald eagle sees the lion.
  4. If someone sees the lion then they visit the dog.The bald eagle visits the dog.

Depth 5, open-world task

75.1%
right on 217 of 289 problems wrong; chance 33% (dashed line)
right

Anne is not nice.

Correct answer: false · Model said: false (59.2% sure)

The problem and its proof

Anne is round. Bob is round. Bob is smart. Dave is red. Dave is round. Gary is furry. Gary is kind. Gary is nice. Gary is red. Gary is round. Furry people are kind. All kind people are round. If Anne is red then Anne is kind. If someone is round and smart then they are nice. Round people are red. Furry people are round. Round, furry people are smart. Red, kind people are furry.

The proof that the statement's negation follows:

  1. Anne is round.Round people are red.Anne is red.
  2. If Anne is red then Anne is kind.Anne is kind.
  3. Red, kind people are furry.Anne is furry.
  4. + Anne is round.Round, furry people are smart.Anne is smart.
  5. + Anne is round.If someone is round and smart then they are nice.Anne is nice.
wrong

The lion eats the lion.

Correct answer: true · Model said: unknown (71.6% sure)

The problem and its proof

The bear chases the lion. The bear chases the mouse. The bear is red. The bear is round. The bear needs the squirrel. The lion eats the bear. The lion is kind. The lion needs the bear. The mouse is round. The squirrel chases the bear. If something eats the bear and the bear eats the lion then the lion is round. If something chases the lion then the lion eats the squirrel. If something is red then it needs the lion. If something eats the squirrel and the squirrel eats the mouse then it chases the lion. If the bear is blue and the bear needs the mouse then the mouse eats the lion. If something eats the lion and it is round then the lion is red. If something needs the lion then it eats the lion.

  1. The bear is red.If something is red then it needs the lion.The bear needs the lion.
  2. If something needs the lion then it eats the lion.The bear eats the lion.
  3. + The bear is round.If something eats the lion and it is round then the lion is red.The lion is red.
  4. If something is red then it needs the lion.The lion needs the lion.
  5. If something needs the lion then it eats the lion.The lion eats the lion.
right

Charlie is not round.

Correct answer: unknown · Model said: unknown (82.2% sure)

The problem and its proof

Charlie is green. Erin is cold. Erin is kind. Erin is quiet. Fiona is cold. Fiona is furry. Fiona is green. Fiona is kind. Fiona is white. Gary is cold. Quiet people are furry. Green, furry people are white. All white people are quiet. If someone is furry then they are round. Kind people are cold. All cold people are white. Quiet, round people are green. Round, white people are cold.

No proof: neither the statement nor its negation follows.

right

Gary is young.

Correct answer: unknown · Model said: unknown (87.0% sure)

The problem and its proof

Bob is big. Bob is blue. Bob is nice. Bob is rough. Bob is white. Erin is blue. Gary is kind. Harry is kind. Harry is nice. Harry is white. Blue things are rough. All nice, white things are young. If something is big then it is blue. All blue, big things are nice. Big, rough things are kind. All white, rough things are big. If something is young then it is blue. If something is rough then it is white.

No proof: neither the statement nor its negation follows.

The closed-world task (true or false), depth by depth

Depth 0, closed-world task

99.3%
right on 298 of 300 problems wrong; chance 50% (dashed line)
right

The bald eagle is not young.

Correct answer: false · Model said: false (80.8% sure)

The problem and its proof

The bald eagle is young. Nice, red things are young.

Stated in the text: "The bald eagle is young."

right

Harry is not quiet.

Correct answer: false · Model said: false (80.8% sure)

The problem and its proof

Bob is nice. Bob is rough. Bob is round. Dave is nice. Erin is cold. Harry is green. Harry is quiet. All rough things are nice. Nice things are rough. All round, quiet things are cold. All big, round things are green. If something is big and quiet then it is round. If something is rough and quiet then it is green. Rough things are quiet. All green things are big.

Stated in the text: "Harry is quiet."

right

The cow does not eat the squirrel.

Correct answer: false · Model said: false (83.5% sure)

The problem and its proof

The cow eats the squirrel. The cow likes the squirrel. The squirrel is cold. If something eats the cow then it chases the cow.

Stated in the text: "The cow eats the squirrel."

right

Anne is nice.

Correct answer: false · Model said: false (92.8% sure)

The problem and its proof

Anne is cold. Bob is green. Dave is big. If someone is green and not nice then they are cold. If Anne is nice and Anne is white then Anne is green. All rough people are green. All cold people are rough. All rough people are kind. If Dave is rough and Dave is not white then Dave is kind. If someone is nice and not green then they are big. Kind people are big.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

90.0%
right on 270 of 300 problems wrong; chance 50% (dashed line)
right

Bob is rough.

Correct answer: false · Model said: false (97.8% sure)

paraphrased
The problem and its proof

Young Alan is kind and blue with green and cold traits that show how really big he is. Fred is a kind person and he is also often cold. Gary seems to be round. If someone you know is big, cold, and young, you know they are also kind. All those who are nice and big are red. Often a young, blue person is very kind and then will be nice to others. A young person with red hands is rough around the edges.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Harry is not big.

Correct answer: true · Model said: true (89.3% sure)

paraphrased
The problem and its proof

Alan is a kind person and he is also often cold. Charlie seems to be round. Eric is both young and red, but is nice and kind but he can be cold. That guy Harry sure is nice. Young, red people are usually quite blue too. If a person happens to be big kind and red at the same time they are a young person. When you meet and big and nice man, they will be kind to the earth and live green lifestyle. A kind person who looks blue because he is is cold is usually big in stature.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cat does not need the cow.

Correct answer: true · Model said: true (95.0% sure)

The problem and its proof

The cat likes the lion. The cow likes the rabbit. The cow needs the rabbit. The cow sees the cat. The lion likes the cow. The rabbit is rough. The rabbit is round. If someone needs the rabbit then they need the cow. If someone needs the cow and they do not like the cat then the cow sees the rabbit.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Erin is green.

Correct answer: true · Model said: true (91.5% sure)

The problem and its proof

Bob is quiet. Erin is red. Gary is rough. Harry is round. If something is green and cold then it is round. Kind things are quiet. If something is rough then it is kind. If something is green then it is kind. All red things are green. If something is green then it is rough. If Erin is kind then Erin is quiet. All quiet, green things are cold.

  1. Erin is red.All red things are green.Erin is green.

Depth 2, closed-world task

72.0%
right on 216 of 300 problems wrong; chance 50% (dashed line)
wrong

Fred is red.

Correct answer: true · Model said: false (75.5% sure)

paraphrased
The problem and its proof

Bob is kind and nice and looks green. Eric, who is relatively young, is also pretty big and tends to be cold. Fred is kind. He is also very cold and blue. That guy Harry sure is nice. People here that are both big and kind while being cold are always red. Kind red people are green on the inside. When someone is kind yet can be cold and blue, they will also be very big. Kind red and round people often look green. Is a kind person if feeling cold and looking red then they are usually young.

  1. Fred is kind. + Fred is cold. + Fred is blue.If someone is kind and cold and blue then they are big.Fred is big.
  2. + Fred is kind. + Fred is cold.If someone is big and kind and cold then they are red.Fred is red.
right

Bob is quiet.

Correct answer: false · Model said: false (94.0% sure)

The problem and its proof

Anne is young. Bob is young. Fiona is quiet. Gary is round. If something is round then it is quiet. All kind things are round.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cow eats the cow.

Correct answer: false · Model said: false (94.7% sure)

The problem and its proof

The bear eats the mouse. The bear likes the mouse. The cow eats the mouse. The cow is big. The dog eats the bear. The mouse eats the cow. The mouse is kind. If someone is rough then they need the cow. If someone needs the cow then they need the bear. If someone is kind then they eat the dog. If someone needs the dog then they eat the dog. If someone likes the cow and the cow eats the dog then the cow is green. If someone is green then they eat the cow. If someone eats the mouse and they need the cow then the cow is rough. If someone needs the cow then they eat the mouse. If the cow eats the mouse and the cow does not need the mouse then the mouse is rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Harry is not cold.

Correct answer: false · Model said: false (71.9% sure)

paraphrased
The problem and its proof

Alan rather resembles the rainbow, as he is green, red and blue. Alan is also said to be very kind. Charlie may be round, but he is also kind. Fred seems to be round. Harry is shaped big and round like a balloon. His red color shows how kind he is. A person with big, round, kind face appears to be rough around the edges because they are naive. Rough and big people are always also cold people. If you run across a young person with rough skin and a round figure, you can count on them being kind. Round young people, red with loveliness, are very cold towards others. Someone who is currently blue, big, and cold, is also kind. A person who is green, blue, and red in color will be round shaped. I bet anyone who is round and kind is also pretty young.

  1. Harry is big. + Harry is kind. + Harry is round.If someone is big and kind and round then they are rough.Harry is rough.
  2. + Harry is big.If someone is rough and big then they are cold.Harry is cold.

Depth 3, closed-world task

64.3%
right on 193 of 300 problems wrong; chance 50% (dashed line)
right

The dog is red.

Correct answer: false · Model said: false (84.0% sure)

The problem and its proof

The bald eagle chases the dog. The bald eagle is blue. The bald eagle is cold. The bald eagle is kind. The dog visits the bald eagle. The dog visits the tiger. The rabbit chases the tiger. The rabbit eats the dog. The rabbit is kind. The rabbit visits the bald eagle. The tiger eats the dog. The tiger is green. If someone is kind and red then they chase the dog. Green, blue people are red. If someone eats the bald eagle then they are blue. If someone chases the bald eagle then the bald eagle eats the dog. If someone visits the dog and they are red then the dog eats the bald eagle. If someone is blue then they visit the dog. If the bald eagle visits the dog and the bald eagle eats the dog then the dog is green. If the bald eagle eats the dog then the dog is red. If someone visits the tiger and they are not kind then the tiger is blue.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Dave is furry.

Correct answer: false · Model said: false (71.9% sure)

The problem and its proof

Anne is quiet. Dave is green. Dave is quiet. If something is kind and not rough then it is white. All quiet things are white. All rough things are white. Kind, white things are furry. All green, rough things are furry. Green, furry things are kind. If Anne is rough and Anne is white then Anne is furry. If something is white then it is green.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The lion does not see the cat.

Correct answer: false · Model said: false (65.4% sure)

The problem and its proof

The cat visits the mouse. The dog sees the cat. The lion visits the dog. The mouse likes the cat. If the lion likes the dog then the lion visits the cat. If someone visits the mouse then they like the dog. If someone likes the dog then the dog sees the lion. If the mouse is not rough then the mouse sees the dog. If someone likes the mouse then they are round. If someone sees the lion then the lion sees the cat.

  1. The cat visits the mouse.If someone visits the mouse then they like the dog.The cat likes the dog.
  2. If someone likes the dog then the dog sees the lion.The dog sees the lion.
  3. If someone sees the lion then the lion sees the cat.The lion sees the cat.
wrong

Dave is round.

Correct answer: true · Model said: false (75.5% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Bob may be round, but he is also kind. Green, big and red are qualities which all describe Dave. That guy Eric sure is nice. Anyone feeling cold is bound to act nice. Big people with red hair are cold because they cannot find coats that fit. People who are said to be big and nice are round. Having cold and red qualities means a person will also have a rough quality.

  1. Dave is red. + Dave is big.If someone is red and big then they are cold.Dave is cold.
  2. If someone is cold then they are nice.Dave is nice.
  3. + Dave is big.If someone is big and nice then they are round.Dave is round.

Depth 4, closed-world task

55.7%
right on 167 of 300 problems wrong; chance 50% (dashed line)
wrong

Charlie is green.

Correct answer: true · Model said: false (94.0% sure)

paraphrased
The problem and its proof

Charlie is both young and red, but is nice and kind but he can be cold. For being so cold, it's good Dave can remain nice. Eric, who is relatively young, is also pretty big and tends to be cold. The young person who is always feeling cold is named Gary. Folks who are rough and tumble yet gentle and kind are blue. Big, kind folks are green ones. People with big smiles and round eyes will have red hair. Any human who is rough, green, and kind, has a blue personality. Young people who feel cold and are still kind usually look rough. When someone is blue, nice and young they will also be as big as the sun. When you run into someone who is rough and green at the same time, they will also be red.

  1. Charlie is cold. + Charlie is kind. + Charlie is young.If someone is cold and kind and young then they are rough.Charlie is rough.
  2. + Charlie is kind.If someone is rough and kind then they are blue.Charlie is blue.
  3. + Charlie is young. + Charlie is nice.If someone is young and blue and nice then they are big.Charlie is big.
  4. + Charlie is kind.If someone is big and kind then they are green.Charlie is green.
right

Dave is not round.

Correct answer: true · Model said: true (86.7% sure)

The problem and its proof

Charlie is big. Charlie is round. Dave is green. All rough people are red. If someone is rough then they are smart. If Charlie is not green then Charlie is rough. If someone is red then they are round. If someone is rough then they are round. If someone is smart and not white then they are rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Gary is not cold.

Correct answer: true · Model said: true (81.8% sure)

The problem and its proof

Dave is red. Gary is green. All kind things are nice. If Gary is not kind then Gary is white. If Dave is white then Dave is cold. If Gary is green and Gary is not white then Gary is big. If something is big and white then it is red. If something is red then it is big. If something is big and not red then it is cold. If something is red and kind then it is cold.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Alan is red.

Correct answer: true · Model said: false (82.6% sure)

paraphrased
The problem and its proof

Alan seems big and rough, but he's actually quite kind. Dave is young and green. This makes him cold to others. Eric is green with envy, and rough with his kids, but overall is just a nice, round individual. People,though kind and rough are also really nice. A big, round person who is kind is also red. Do not fear big cold people for they are often kind. A person that is round and somewhat green while being nice tends to be red as well. When you meet and big and nice man, they will be kind to the earth and live green lifestyle. People who are young are also blue. Green folks who are nice and rough are a round shape.

  1. Alan is kind. + Alan is rough.If someone is kind and rough then they are nice.Alan is nice.
  2. + Alan is big. + Alan is kind.If someone is big and kind and nice then they are green.Alan is green.
  3. + Alan is rough.If someone is nice and rough and green then they are round.Alan is round.
  4. + Alan is big. + Alan is kind.If someone is big and round and kind then they are red.Alan is red.

Depth 5, closed-world task

49.3%
right on 148 of 300 problems wrong; chance 50% (dashed line)
right

The rabbit sees the lion.

Correct answer: true · Model said: true (66.7% sure)

The problem and its proof

The bear sees the tiger. The bear visits the lion. The bear visits the rabbit. The lion is big. The lion is nice. The lion needs the bear. The lion visits the tiger. The rabbit is big. The rabbit is nice. The rabbit needs the bear. The rabbit visits the tiger. The tiger is nice. The tiger needs the rabbit. The tiger visits the lion. If someone is nice and they see the bear then they see the lion. If someone visits the bear then they visit the rabbit. If the tiger visits the rabbit then the rabbit sees the bear. If someone sees the tiger then the tiger is blue. If the lion visits the rabbit and the lion is not blue then the rabbit needs the lion. If someone is blue then they visit the bear. If the rabbit does not need the lion then the lion sees the rabbit.

  1. The bear sees the tiger.If someone sees the tiger then the tiger is blue.The tiger is blue.
  2. If someone is blue then they visit the bear.The tiger visits the bear.
  3. If someone visits the bear then they visit the rabbit.The tiger visits the rabbit.
  4. If the tiger visits the rabbit then the rabbit sees the bear.The rabbit sees the bear.
  5. + The rabbit is nice.If someone is nice and they see the bear then they see the lion.The rabbit sees the lion.
wrong

Fiona is not big.

Correct answer: false · Model said: true (81.8% sure)

The problem and its proof

Charlie is green. Charlie is quiet. Charlie is round. Dave is cold. Fiona is round. Gary is big. Gary is round. Round, green people are white. If someone is blue then they are big. If someone is white and round then they are cold. If Fiona is round then Fiona is green. If someone is cold then they are blue.

  1. Fiona is round.If Fiona is round then Fiona is green.Fiona is green.
  2. + Fiona is round.Round, green people are white.Fiona is white.
  3. + Fiona is round.If someone is white and round then they are cold.Fiona is cold.
  4. If someone is cold then they are blue.Fiona is blue.
  5. If someone is blue then they are big.Fiona is big.
wrong

The rabbit does not need the rabbit.

Correct answer: false · Model said: true (94.0% sure)

The problem and its proof

The cat is blue. The cat is rough. The cat needs the dog. The cow is red. The cow likes the cat. The cow likes the dog. The dog is rough. The dog visits the rabbit. The rabbit is blue. The rabbit is rough. The rabbit needs the cow. The rabbit visits the cow. If someone visits the cat then they are cold. If someone visits the cat then they need the rabbit. If someone likes the dog then they visit the cat. If someone is rough then they like the rabbit. If someone needs the dog then they visit the cat. If someone needs the rabbit then the rabbit likes the dog. If the dog visits the rabbit and the rabbit visits the dog then the dog is nice.

  1. The cat needs the dog.If someone needs the dog then they visit the cat.The cat visits the cat.
  2. If someone visits the cat then they need the rabbit.The cat needs the rabbit.
  3. If someone needs the rabbit then the rabbit likes the dog.The rabbit likes the dog.
  4. If someone likes the dog then they visit the cat.The rabbit visits the cat.
  5. If someone visits the cat then they need the rabbit.The rabbit needs the rabbit.
right

Anne is rough.

Correct answer: false · Model said: false (76.6% sure)

The problem and its proof

Anne is blue. Bob is quiet. Bob is red. Dave is cold. Dave is quiet. Harry is blue. Harry is cold. Harry is nice. Harry is quiet. Harry is rough. If something is cold and furry then it is blue. Cold, red things are furry. Quiet things are red. If something is furry then it is nice. Red things are quiet. If something is nice then it is rough. If something is cold then it is red. If something is blue and red then it is nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

Where each one is better

Open world, Jev vs GLiDE, Oct 3 rerun. Jev is clearly ahead on depth 3, depth 4, depth 5, false answers and true answers. GLiDE, Oct 3 rerun is clearly ahead on unknown answers (+19.5 points). Too close to call: depth 0, depth 1 and depth 2.

Closed world, Jev vs GLiDE, Oct 3 rerun. Jev is clearly ahead on depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GLiDE, Oct 3 rerun is not clearly ahead on any depth or answer class. Too close to call: depth 0.

Open world, Jev vs GLiDE, Oct 2. Jev is clearly ahead on false answers and true answers. GLiDE, Oct 2 is clearly ahead on unknown answers (+21.7 points). Too close to call: depth 0, depth 1, depth 2, depth 3, depth 4 and depth 5.

Closed world, Jev vs GLiDE, Oct 2. Jev is clearly ahead on depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GLiDE, Oct 2 is not clearly ahead on any depth or answer class. Too close to call: depth 0.

A slice counts as clearly ahead when its paired 95% interval excludes zero.

True, false and unknown, open world

By correct answer, Open world
Correct answerItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2Best constant
false60588.3% 85.5–90.965.5% 61.5–69.376.0% 72.4–79.5one answer
true60588.8% 86.1–91.170.4% 66.6–73.977.0% 73.6–80.5one answer
unknown59074.2% 70.7–77.693.7% 91.7–95.695.9% 94.2–97.5one answer

True, false and unknown, closed world

By correct answer, Closed world
Correct answerItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2Best constant
false90088.9% 86.9–90.964.9% 61.9–68.167.9% 64.9–71.0one answer
true90089.7% 87.7–91.672.8% 69.8–75.775.7% 72.7–78.3one answer

Paraphrased vs templated rules, open world

By wording, Open world
WordingItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2Best constant
templated1,29886.1% 84.2–87.979.3% 77.1–81.486.0% 84.1–87.834.5%
paraphrased (NatLang)50278.1% 74.5–82.168.9% 65.1–73.174.9% 71.3–78.935.9%

Paraphrased vs templated rules, closed world

By wording, Closed world
WordingItemsJevGLiDE, Oct 3 rerunGLiDE, Oct 2Best constant
templated1,30694.0% 92.8–95.373.4% 71.1–75.976.5% 74.4–78.950.2%
paraphrased (NatLang)49476.7% 73.1–80.456.7% 52.4–61.159.3% 54.9–63.650.6%

Answer by answer: Jev, GLiDE, Oct 3 rerun and GLiDE, Oct 2 on the same problems

Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
GLiDE, Oct 3 rerun, open-world task
Correct ↓ · said →truefalseunknown
true4260179
false3396206
unknown2314553
GLiDE, Oct 2, open-world task
Correct ↓ · said →truefalseunknown
true4660139
false2460143
unknown177566

Jev said true on 37% of problems; the correct answer is true on 34%. GLiDE, Oct 3 rerun said unknown on 52% of problems; the correct answer is unknown on 33%. GLiDE, Oct 2 said unknown on 47% of problems; the correct answer is unknown on 33%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
GLiDE, Oct 3 rerun, closed-world task
Correct ↓ · said →truefalse
true655245
false316584
GLiDE, Oct 2, closed-world task
Correct ↓ · said →truefalse
true681219
false289611

Jev said true on 50% of problems; the correct answer is true on 50%. GLiDE, Oct 3 rerun said true on 54% of problems; the correct answer is true on 50%. GLiDE, Oct 2 said true on 54% of problems; the correct answer is true on 50%.

Where GLiDE, Oct 3 rerun and GLiDE, Oct 2 err

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

GLiDE, Oct 3 rerun, open-world task

GLiDE, Oct 3 rerun, open-world task
Correct ↓ · said →truefalseunknownRecall
true426017970.4%
false339620665.5%
unknown231455393.7%

GLiDE, Oct 3 rerun answered false 22.8%, true 25.1%, unknown 52.1% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

GLiDE, Oct 3 rerun, closed-world task

GLiDE, Oct 3 rerun, closed-world task
Correct ↓ · said →truefalseRecall
true65524572.8%
false31658464.9%

GLiDE, Oct 3 rerun answered false 46.1%, true 53.9% of the time; the correct answers are false 50.0%, true 50.0%.

GLiDE, Oct 3 rerun at depth 0 and depth 5

GLiDE, Oct 3 rerun, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9802
false0982
unknown2197
GLiDE, Oct 3 rerun, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true56045
false14654
unknown5577

Open-world task. At depth 0, GLiDE, Oct 3 rerun was right on 98% of problems (the green diagonal); at depth 5, on 62%. The answer whose share moved most is unknown: 34% of its answers at depth 0, 61% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
GLiDE, Oct 3 rerun, closed-world task, depth 0
Correct ↓ · said →truefalse
true1491
false2148
GLiDE, Oct 3 rerun, closed-world task, depth 5
Correct ↓ · said →truefalse
true7971
false9753

Closed-world task. At depth 0, GLiDE, Oct 3 rerun was right on 99% of problems (the green diagonal); at depth 5, on 44%. The answer whose share moved most is true: 50% of its answers at depth 0, 59% at depth 5, against 50% and 50% of the correct answers.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

GLiDE, Oct 2, open-world task

GLiDE, Oct 2, open-world task
Correct ↓ · said →truefalseunknownRecall
true466013977.0%
false246014376.0%
unknown17756695.9%

GLiDE, Oct 2 answered false 25.9%, true 26.9%, unknown 47.1% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

GLiDE, Oct 2, closed-world task

GLiDE, Oct 2, closed-world task
Correct ↓ · said →truefalseRecall
true68121975.7%
false28961167.9%

GLiDE, Oct 2 answered false 46.1%, true 53.9% of the time; the correct answers are false 50.0%, true 50.0%.

GLiDE, Oct 2 at depth 0 and depth 5

GLiDE, Oct 2, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true9802
false0982
unknown2197
GLiDE, Oct 2, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true68033
false06635
unknown3183

Open-world task. At depth 0, GLiDE, Oct 2 was right on 98% of problems (the green diagonal); at depth 5, on 75%. The answer whose share moved most is unknown: 34% of its answers at depth 0, 52% at depth 5, against 33% and 30% of the correct answers.

The closed-world task at depth 0 and depth 5
GLiDE, Oct 2, closed-world task, depth 0
Correct ↓ · said →truefalse
true1500
false2148
GLiDE, Oct 2, closed-world task, depth 5
Correct ↓ · said →truefalse
true8664
false8862

Closed-world task. At depth 0, GLiDE, Oct 2 was right on 99% of problems (the green diagonal); at depth 5, on 49%. The answer whose share moved most is false: 49% of its answers at depth 0, 42% at depth 5, against 50% and 50% of the correct answers.

GLiDE, Oct 3 rerun and GLiDE, Oct 2 across every breakdown

For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownGLiDE, Oct 3 rerunGLiDE, Oct 2
Proof depthdepth 4 61.4% · depth 0 97.7% (36.3 points)depth 4 73.9% · depth 0 97.7% (23.7 points)
Gold answerfalse 65.5% · unknown 93.7% (28.3 points)false 76.0% · unknown 95.9% (19.9 points)
Theory kindrelation 73.3% · attribute 78.6% (5.3 points)relation 80.0% · attribute 84.9% (4.9 points)
Negation in the theorywithout negation 73.6% · with negation 79.3% (5.8 points)without negation 80.1% · with negation 85.8% (5.7 points)
Negated statementplain statement 75.0% · negated statement 77.8% (2.8 points)plain statement 81.6% · negated statement 84.1% (2.5 points)
Question strategyinv-proof 65.5% · inv-random 98.0% (32.5 points)inv-proof 76.0% · inv-random 98.0% (22.0 points)
Paraphrased rulesparaphrased (NatLang) 68.9% · templated 79.3% (10.4 points)paraphrased (NatLang) 74.9% · templated 86.0% (11.1 points)
Theory's deepest proof4 56.8% · 0 98.6% (41.8 points)4 66.7% · 0 98.6% (31.9 points)
Theory length110+ 65.2% · 0-49 95.1% (29.9 points)110+ 74.3% · 0-49 95.5% (21.3 points)
Number of rules3-5 68.5% · 0-2 93.1% (24.6 points)3-5 75.1% · 0-2 94.1% (19.0 points)
Number of facts13+ 62.9% · 0-3 88.9% (26.0 points)13+ 74.2% · 0-3 92.2% (18.0 points)
Proof size5+ 50.4% · 0-1 94.8% (44.4 points)5+ 65.6% · 0-1 96.5% (30.9 points)

Exactly what GLiDE, Oct 3 rerun and GLiDE, Oct 2 were asked

One open-world problem at depth 0, as each model received it. The correct answer is unknown.

The bear is green. Green things are not young.

Statement: The bear is not big.
Jevright answered unknown (100.0%); correct answer unknown

POST https://api.typesafe.ai/v1/systemone

Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%. Its separate confidence: 100.0%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "jev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "jev-1.13.0",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 1,
      "probabilities": {
        "false": 0,
        "true": 0,
        "unknown": 1
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 460,
    "output_tokens": 38
  }
}
GLiDE, Oct 3 rerunright answered unknown (98.3%); correct answer unknown

POST https://api.fastino.ai/v1/systemone

The same request as GLiDE's first run, sent again on October 3 at Fastino's request.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.7%, true 1.0%, unknown 98.3%. Its separate confidence: 97.2%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "fastino/GLiDE",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "glide",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 0.9724408909000181,
      "probabilities": {
        "false": 0.007048396451747089,
        "true": 0.010255356324117408,
        "unknown": 0.9826962472241355
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 203,
    "output_tokens": 1
  }
}
GLiDE, Oct 2right answered unknown (98.3%); correct answer unknown

POST https://api.fastino.ai/v1/systemone

Fastino serves GLiDE on the same System One format as Jev, so this is Jev's request with a different address and model name.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.7%, true 1.0%, unknown 98.3%. Its separate confidence: 97.2%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "fastino/GLiDE",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "glide",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 0.9724408909000181,
      "probabilities": {
        "false": 0.007048396451747089,
        "true": 0.010255356324117408,
        "unknown": 0.9826962472241355
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 203,
    "output_tokens": 1
  }
}

Two runs: at launch, and again at Fastino's request

On October 3, Fastino's CEO asked us to re-evaluate GLiDE, saying Fastino had problems during the launch. We sent the same 3,600 problems again with exactly the same request and settings, eight requests at a time as before. Fastino's API reports the model as "glide" both times, so the responses do not show what changed.

Open world: true, false or unknown

2026-10-02, at launch2026-10-03, rerun
Accuracy82.9%76.4%
Accuracy minus Jev's (95% interval)−0.9 (−3.1 to +1.2)−7.4 (−9.9 to −5.1)
By depth, 0 to 598 · 88 · 84 · 79 · 74 · 7598 · 87 · 80 · 70 · 61 · 62
Median time per decision1.15 s0.80 s
Slowest 10% start at10 s8.73 s
Decisions taking 10 seconds or more185156
Input tokens billed per decision927531
List-price cost per million decisions$278$159
How well its probability separates right from wrong (AUROC; 0.5 is chance)0.560.66

141 of 1,800 answers changed between the runs: 129 went from right to wrong and 12 from wrong to right, most of them at depth 3 and deeper.

Closed world: true or false

2026-10-02, at launch2026-10-03, rerun
Accuracy71.8%68.8%
Accuracy minus Jev's (95% interval)−17.5 (−19.6 to −15.4)−20.4 (−22.6 to −18.3)
By depth, 0 to 599 · 90 · 72 · 64 · 56 · 4999 · 88 · 68 · 61 · 52 · 44
Median time per decision0.92 s0.69 s
Slowest 10% start at5.37 s1.01 s
Decisions taking 10 seconds or more5824
Input tokens billed per decision495342
List-price cost per million decisions$148$103
How well its probability separates right from wrong (AUROC; 0.5 is chance)0.520.56

61 of 1,800 answers changed between the runs: 57 went from right to wrong and 4 from wrong to right, most of them at depth 3 and deeper.

Time and money

GLiDE, Oct 3 rerun's median open-world decision took 798 ms against Jev's 189 ms, and 180 of its 3,600 decisions took ten seconds or more (the slowest 149 s). At list price GLiDE, Oct 3 rerun cost $131 per million decisions against Jev's $23: 5.7 times as much per decision and 6.8 times as much per correct decision.GLiDE, Oct 2's median open-world decision took 1.2 s against Jev's 189 ms, and 243 of its 3,600 decisions took ten seconds or more (the slowest 135 s). At list price GLiDE, Oct 2 cost $213 per million decisions against Jev's $23: 9.3 times as much per decision and 10.4 times as much per correct decision.

Fastino is upfront that GLiDE works this way. It writes that "traditional decision models score a set of options in a single fixed pass," while GLiDE "first produces a fast probability distribution, then allocates additional reasoning when the leading result is uncertain," so it can "spend more computation on difficult decisions while keeping straightforward ones fast." Fastino bills that reasoning as tokens, and its API documentation recommends a read timeout of at least 300 seconds (Fastino). TypeSafe describes Jev, a System One model, as generating all outputs "in a single query," in parallel, in 70 to 500 milliseconds, so every decision takes about the same time and money (TypeSafe).

Every decision's time

  • Jev
  • GLiDE, Oct 3 rerun
  • GLiDE, Oct 2
05001,0001,5002,0000.1 s0.3 s1 s3 s10 s30 s100 stime per decision (log scale)180 GLiDE, Oct 3 rerun decisions ≥ 10 s243 GLiDE, Oct 2 decisions ≥ 10 sJev median 179 msGLiDE, Oct 3 rerun median 752 msGLiDE, Oct 2 median 979 ms

Average time per decision, by proof depth

Open world: true, false or unknown

0 s2 s4 s6 s8 s012345proof depth (inference steps)207ms202ms209ms203ms207ms209ms1.0s2.1s4.0s4.6s4.5s3.5s1.1s3.4s5.3s6.7s6.6s4.4s

Closed world: true or false

0 s2 s4 s6 s8 s012345proof depth (inference steps)177ms177ms175ms174ms174ms178ms733ms1.1s1.1s1.2s1.2s1.1s921ms2.4s2.5s2.5s2.2s2.2s
ModelTaskDepth 0Depth 1Depth 2Depth 3Depth 4Depth 5
JevOpen world207 ms202 ms209 ms203 ms207 ms209 ms
JevClosed world177 ms177 ms175 ms174 ms174 ms178 ms
GLiDE, Oct 3 rerunOpen world1.0 s2.1 s4.0 s4.6 s4.5 s3.5 s
GLiDE, Oct 3 rerunClosed world733 ms1.1 s1.1 s1.2 s1.2 s1.1 s
GLiDE, Oct 2Open world1.1 s3.4 s5.3 s6.7 s6.6 s4.4 s
GLiDE, Oct 2Closed world921 ms2.4 s2.5 s2.5 s2.2 s2.2 s

List-price cost per million decisions, by proof depth

Open world: true, false or unknown

$0$100$200$300$400012345proof depth (inference steps)$23$23$23$24$24$24$91$122$165$182$208$190$97$213$288$374$391$305

Closed world: true or false

$0$100$200$300$400012345proof depth (inference steps)$22$22$22$23$23$23$82$100$101$108$115$111$84$157$166$167$159$157
ModelTaskDepth 0Depth 1Depth 2Depth 3Depth 4Depth 5
JevOpen world$23$23$23$24$24$24
JevClosed world$22$22$22$23$23$23
GLiDE, Oct 3 rerunOpen world$91$122$165$182$208$190
GLiDE, Oct 3 rerunClosed world$82$100$101$108$115$111
GLiDE, Oct 2Open world$97$213$288$374$391$305
GLiDE, Oct 2Closed world$84$157$166$167$159$157

GLiDE, Oct 3 rerun's fast and slow answers

Split at 2 seconds. The slow answers are the ones where GLiDE, Oct 3 rerun spent extra computation; each column says what that cost and what it bought.

TaskAnswersCountAverage timeInput tokens billed, averageRightStated probability, average
Open worldUnder 2 s1,407740 ms30372.6%80.6%
Open worldOver 2 s39312 s1,35189.8%72.3%
Closed worldUnder 2 s1,711700 ms28467.4%84.6%
Closed worldOver 2 s898.1 s1,45895.5%75.3%

When GLiDE, Oct 3 rerun took longer, it was more often right, and it stated lower probabilities for those answers. It decides when to spend the extra time from its first probability, so where that probability is high and wrong, it answers quickly and wrong.

GLiDE, Oct 2's fast and slow answers

Split at 2 seconds. The slow answers are the ones where GLiDE, Oct 2 spent extra computation; each column says what that cost and what it bought.

TaskAnswersCountAverage timeInput tokens billed, averageRightStated probability, average
Open worldUnder 2 s1,157967 ms29978.5%84.8%
Open worldOver 2 s64311 s2,05790.8%72.2%
Closed worldUnder 2 s1,559873 ms28568.2%86.4%
Closed worldOver 2 s24110 s1,85295.0%76.1%

When GLiDE, Oct 2 took longer, it was more often right, and it stated lower probabilities for those answers. It decides when to spend the extra time from its first probability, so where that probability is high and wrong, it answers quickly and wrong.

Times are wall time per request; GLiDE, Oct 3 rerun and GLiDE, Oct 2 timed in their scored run with 8 requests in flight, the others one request at a time. Costs are list prices times the tokens each API reported. Cost per decision for every hosted model · Speed, size and memory.

Consistency and calibration

GLiDE, Oct 3 rerun

GLiDE, Oct 3 rerun was not asked a second time: at its speed a rerun of every item would have taken hours, so its consistency is not measured.

Calibration on the open world task: GLiDE, Oct 3 rerun's probability for its own answer averaged 78.8% against 76.4% accuracy; expected calibration error 3.6 points.

Calibration on the closed world task: GLiDE, Oct 3 rerun's probability for its own answer averaged 84.1% against 68.8% accuracy; expected calibration error 15.3 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.66 on the open world task against Jev's 0.85; 0.56 on the closed world task against Jev's 0.86.

GLiDE, Oct 2

GLiDE, Oct 2 was not asked a second time: at its speed a rerun of every item would have taken hours, so its consistency is not measured.

Calibration on the open world task: GLiDE, Oct 2's probability for its own answer averaged 80.3% against 82.9% accuracy; expected calibration error 6.7 points.

Calibration on the closed world task: GLiDE, Oct 2's probability for its own answer averaged 85.0% against 71.8% accuracy; expected calibration error 15.0 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.56 on the open world task against Jev's 0.85; 0.52 on the closed world task against Jev's 0.86.

Size, hardware and speed

ModelKindParametersRan onMedian time per decisionMemory peak
Jevhosted decision modelundisclosedvendor's servers189 msnot applicable (hosted)
GLiDE, Oct 3 rerunhosted decision modelundisclosedvendor's servers798 msnot applicable (hosted)
GLiDE, Oct 2hosted decision modelundisclosedvendor's servers1,153 msnot applicable (hosted)

Times are open-world, one request at a time except GLiDE, Oct 3 rerun and GLiDE, Oct 2, timed in its scored run with 8 requests in flight. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.

GLiDE, Oct 3 rerun

Parameters
undisclosed
Architecture
undisclosed ("thinking" decision model that adds reasoning when its first distribution is uncertain)
Weights
not available (hosted)
Where it ran
Fastino API, POST /v1/systemone, model fastino/GLiDE (same System One request as Jev); rerun on 2026-10-03 at the request of Fastino's CEO, who said Fastino had problems during the launch
Maker's stated hardware
not applicable (hosted)
Price
$0.30 per million input tokens; output free
Time per decision, open world
median 798 ms, 90th percentile 8,726 ms, over the network
Time per decision, closed world
median 690 ms, 90th percentile 1,005 ms, over the network
Source
docs.fastino.ai/concepts/glide

GLiDE, Oct 2

Parameters
undisclosed
Architecture
undisclosed ("thinking" decision model that adds reasoning when its first distribution is uncertain)
Weights
not available (hosted)
Where it ran
Fastino API, POST /v1/systemone, model fastino/GLiDE (same System One request as Jev)
Maker's stated hardware
not applicable (hosted)
Price
$0.30 per million input tokens; output free
Time per decision, open world
median 1,153 ms, 90th percentile 10,152 ms, over the network
Time per decision, closed world
median 923 ms, 90th percentile 5,367 ms, over the network
Source
docs.fastino.ai/concepts/glide

The models

Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

GLiDE, Oct 3 rerun The same GLiDE, run again on October 3 with exactly the same request and settings, because Fastino's CEO asked us to re-evaluate it: he said Fastino had problems during the launch. The October 2 run is kept and shown beside it. Fastino's API reports the model as "glide" both times, so the responses do not show what changed.

GLiDE, Oct 2 GLiDE is a hosted decision model made by Fastino Labs, which describes it as a "thinking" decision model: it forms a probability over the options and spends more computation when the leading option is uncertain. Fastino serves it on the same System One request format as Jev, so it got exactly Jev's request: the text, the question and the options, one request per item. Its size and architecture are not published.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-16, from the harness's working tree at commit b324cd3 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.