Jev vs GLiDE: two hosted decision models on deep reasoning
GLiDE is Fastino's hosted decision model, released on 1 October 2026, which Fastino calls the first thinking decision model. It takes the same request as Jev, so both got exactly the same text, question and options for the same 3,600 problems.
Jev answers every decision in one parallel pass. GLiDE, by Fastino's own description, spends extra time and tokens on decisions it is unsure of, so it is slower and costs more, most of all on the problems it finds hard (see Time and money). Fastino reports that GLiDE beats Jev on its Decision Index 0.2.1 (64.81 to 57.91); that is Fastino's benchmark, not this one, and we preregistered no prediction about which would win here.
Jev minus GLiDE: +17.5 points on the same 1,800 items, 95% interval +15.4 to +19.6: Jev is clearly ahead.
Jev vs GLiDE by proof depth
Each point is about 300 problems that need that many inference steps, with its 95% interval.
Open world: true, false or unknown
Jev
GLiDE
Closed world: true or false
Jev
GLiDE
Jev minus GLiDE by depth, open world
−100+13
depth 0300 items
0.0
depth 1302 items
0.0
depth 2303 items
+0.7
depth 3303 items
+1.3
depth 4303 items
−2.0
depth 5289 items
+5.9
Jev minus GLiDE by depth, closed world
−100+46
depth 0300 items
0.0
depth 1300 items
+3.7
depth 2300 items
+20.0
depth 3300 items
+20.7
depth 4300 items
+20.7
depth 5300 items
+40.0
Paired differences: positive means Jev was more accurate on the same items.
Accuracy by depth, open world
By depth, Open world
Proof depth
Items
Jev
GLiDE
Best constant
depth 0
300
97.7%96.0–99.3
97.7%95.7–99.0
33.3%
depth 1
302
87.7%83.8–91.4
87.7%84.1–91.4
33.4%
depth 2
303
84.8%80.2–88.4
84.2%80.2–88.1
33.3%
depth 3
303
79.9%75.2–84.5
78.5%73.6–83.2
33.3%
depth 4
303
71.9%66.7–77.2
73.9%69.3–78.5
33.3%
depth 5
289
81.0%76.5–85.5
75.1%69.6–79.9
34.9%
Accuracy by depth, closed world
By depth, Closed world
Proof depth
Items
Jev
GLiDE
Best constant
depth 0
300
99.3%98.3–100.0
99.3%98.3–100.0
50.0%
depth 1
300
93.7%91.0–96.3
90.0%86.7–93.3
50.0%
depth 2
300
92.0%88.7–94.7
72.0%67.0–77.3
50.0%
depth 3
300
85.0%81.3–89.3
64.3%59.0–69.7
50.0%
depth 4
300
76.3%71.7–81.0
55.7%50.7–61.0
50.0%
depth 5
300
89.3%85.7–92.3
49.3%43.7–55.0
50.0%
What GLiDE got right and wrong, depth by depth
At each proof depth, GLiDE's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.
How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)
Depth 0, open-world task
97.7%
right on 293 of 300 problems wrong; chance 33% (dashed line)
Anne is round.
Correct answer: unknown · Model said: unknown (83.6% sure)
The problem and its proof
Anne is not kind. Anne is quiet. Dave is not green. Dave is rough. Dave is round. Dave is smart. Fiona is quiet. Fiona is round. Gary is smart. Gary is young. Round people are rough. If someone is smart then they are rough. All young, rough people are smart. If someone is rough then they are kind. Rough people are kind. If Fiona is smart and Fiona is kind then Fiona is green. If Anne is round then Anne is quiet. Round, rough people are quiet. All kind people are young.
No proof: neither the statement nor its negation follows.
Charlie is green.
Correct answer: true · Model said: true (98.7% sure)
paraphrasedThe problem and its proof
Charlie is young and green. Even though he is round he is cold outside. Dave is a kind person and he is also often cold. Eric is kind and nice and looks green. Gary is a kind person and he is also often cold. If a kind person is green and a bit blue then they are also nice. If someone plays rough and is nice and round, they will be big. People of young age are rough and cold but some are nice. Most round and cold people are often rough. When you meet and big and nice man, they will be kind to the earth and live green lifestyle. Nice people who feel blue often look green when ill.
Stated in the text: "Charlie is green."
Harry is green.
Correct answer: false · Model said: false (97.8% sure)
The problem and its proof
Charlie is quiet. Gary is not quiet. Harry is not green. If someone is nice and not cold then they are white.
Stated in the text: "Harry is not green."
Eric is big.
Correct answer: true · Model said: true (98.6% sure)
paraphrasedThe problem and its proof
Alan is green and cold too. Bob who is young and nice is also quite big and feels green. Eric is a big fellow, often blue and sad, but he is nice. Harry was proud of being round, yet rough. His red cheeks glowed. When a person is nice and round and cold, they look blue. Rough people who are red in color and big are usally round in shape. A rather big person is a red person. Kind people with rough skin are usually red because it's wind burn. Kind red and round people often look green. Big people who are blue and red are afflicted with rough skin. Rough people who look red are cold.
Stated in the text: "Eric is big."
Depth 1, open-world task
87.7%
right on 265 of 302 problems wrong; chance 33% (dashed line)
Bob is not rough.
Correct answer: true · Model said: true (99.2% sure)
The problem and its proof
Bob is young. Erin is rough. Harry is red. All red people are young. If someone is young then they are not rough.
Bob is young.→If someone is young then they are not rough.→Bob is not rough.
Bob is not round.
Correct answer: true · Model said: true (92.1% sure)
The problem and its proof
Anne is cold. Anne is furry. Anne is not kind. Anne is round. Anne is not young. Bob is cold. Bob is furry. Bob is red. Bob is young. Harry is blue. Harry is cold. Harry is furry. Harry is kind. Harry is red. Harry is not round. Furry, young things are not round. If something is round and not furry then it is not red.
Bob is furry. + Bob is young.→Furry, young things are not round.→Bob is not round.
Fred is not red.
Correct answer: unknown · Model said: unknown (79.6% sure)
paraphrasedThe problem and its proof
That guy Alan sure is nice. Bob woke up one day to find he had become round. He was still rough, red, and green, but now he was also big. Charlie may be young but he is nice, wears green shoes and is cold. That guy Fred sure is nice. Someone who has young, nice and cold qualities will possess a red quality, too. Interesting that all round people are cold. Cold and red people are always kind to others. Young people who feel cold and are still kind usually look rough.
No proof: neither the statement nor its negation follows.
The cat does not see the cow.
Correct answer: false · Model said: false (81.2% sure)
The problem and its proof
The cat needs the rabbit. The cat sees the rabbit. The cow chases the cat. The cow chases the rabbit. The cow is blue. The rabbit chases the cow. The rabbit is green. If someone sees the rabbit then they see the cow. If someone sees the cow then they are big.
The proof that the statement's negation follows:
The cat sees the rabbit.→If someone sees the rabbit then they see the cow.→The cat sees the cow.
Depth 2, open-world task
84.2%
right on 255 of 303 problems wrong; chance 33% (dashed line)
The lion likes the cow.
Correct answer: unknown · Model said: unknown (83.7% sure)
The problem and its proof
The cow visits the squirrel. The lion is rough. The squirrel likes the tiger. The tiger visits the cow. If something is young and it chases the lion then it likes the squirrel. If something is cold then it likes the cow. If something likes the cow then it likes the tiger. If something is rough then it chases the squirrel. If something visits the cow then it is cold. If the cow does not like the tiger then the cow chases the tiger.
No proof: neither the statement nor its negation follows.
The bald eagle chases the lion.
Correct answer: true · Model said: unknown (89.3% sure)
The problem and its proof
The bald eagle chases the dog. The bald eagle chases the tiger. The bald eagle is red. The bald eagle is rough. The bald eagle needs the lion. The bald eagle sees the dog. The dog does not chase the lion. The dog is not rough. The dog sees the bald eagle. The dog sees the tiger. The lion is rough. The lion needs the bald eagle. The lion needs the dog. The lion needs the tiger. The lion sees the bald eagle. The tiger is red. If the lion sees the bald eagle and the bald eagle chases the tiger then the tiger is round. If someone sees the lion and the lion chases the dog then the dog needs the lion. If someone is kind and they chase the bald eagle then the bald eagle needs the dog. If someone sees the lion and the lion is rough then they chase the lion. If someone needs the dog and the dog is kind then the dog sees the bald eagle. If someone sees the bald eagle and the bald eagle chases the lion then the lion is red. If someone is red and rough then they see the lion. If someone needs the lion then the lion needs the dog. If someone is young then they are rough.
The bald eagle is red. + The bald eagle is rough.→If someone is red and rough then they see the lion.→The bald eagle sees the lion.
+ The lion is rough.→If someone sees the lion and the lion is rough then they chase the lion.→The bald eagle chases the lion.
Bob is rough.
Correct answer: unknown · Model said: unknown (80.9% sure)
The problem and its proof
Anne is blue. Anne is round. Bob is smart. Erin is big. Erin is green. Harry is big. Harry is red. All round things are big. If something is big and smart then it is rough.
No proof: neither the statement nor its negation follows.
The tiger is not kind.
Correct answer: false · Model said: false (76.0% sure)
The problem and its proof
The squirrel is cold. The squirrel is kind. The squirrel is nice. The squirrel is young. The squirrel needs the tiger. The squirrel visits the tiger. The tiger is nice. The tiger needs the squirrel. The tiger sees the squirrel. The tiger visits the squirrel. If something is nice then it needs the squirrel. If the tiger needs the squirrel then the squirrel sees the tiger. If something needs the tiger then it sees the squirrel. If something is kind and it needs the squirrel then it sees the tiger. If something sees the squirrel and it sees the tiger then the tiger is kind. If something needs the squirrel then it is nice. If something visits the tiger then it is kind. If something sees the tiger then the tiger needs the squirrel.
The proof that the statement's negation follows:
The squirrel needs the tiger.→If something needs the tiger then it sees the squirrel.→The squirrel sees the squirrel.
+ The tiger needs the squirrel.→If the tiger needs the squirrel then the squirrel sees the tiger.→The squirrel sees the tiger.
→If something sees the squirrel and it sees the tiger then the tiger is kind.→The tiger is kind.
Depth 3, open-world task
78.5%
right on 238 of 303 problems wrong; chance 33% (dashed line)
Gary is rough.
Correct answer: unknown · Model said: unknown (77.1% sure)
paraphrasedThe problem and its proof
Alan seems to be round. Young Bob while being blue because his is cold is round shaped while still maintaining a green thumb. The young person who is always feeling cold is named Gary. Kind red people are green on the inside. A person who is kind, cold, and rough is big. Young people who are cold to others and green with envy are actually nice. When green, young and round fits a person, you'll see that rough will also fit. Someone with rough and green feet is invariably kind. Even though a person is big and rough when they are cold they turn red. Most big and cold people are also rough.
No proof: neither the statement nor its negation follows.
The tiger is round.
Correct answer: false · Model said: unknown (76.6% sure)
The problem and its proof
The cow chases the squirrel. The lion chases the squirrel. The lion eats the squirrel. The lion does not eat the tiger. The lion does not need the squirrel. The squirrel chases the lion. The squirrel eats the lion. The tiger chases the squirrel. The tiger eats the lion. The tiger eats the squirrel. The tiger is cold. The tiger needs the squirrel. If something is green and rough then it chases the tiger. If something chases the squirrel then it is cold. If something eats the cow and it is green then it is not round. If something is cold and it eats the cow then the cow is green. If something is green then it eats the cow. If the tiger is cold then the tiger is green. If something eats the squirrel then the squirrel is green. If the cow needs the squirrel and the cow is not green then the squirrel does not need the tiger.
The proof that the statement's negation follows:
The tiger is cold.→If the tiger is cold then the tiger is green.→The tiger is green.
→If something is green then it eats the cow.→The tiger eats the cow.
→If something eats the cow and it is green then it is not round.→The tiger is not round.
The bear needs the bear.
Correct answer: unknown · Model said: unknown (90.0% sure)
The problem and its proof
The bald eagle does not need the cat. The bear is rough. The cat needs the rabbit. The rabbit chases the bald eagle. If the bear eats the cat then the cat needs the rabbit. If something chases the bald eagle then it is big. If something chases the bear and the bear does not need the rabbit then it chases the bald eagle. If something chases the bear and it does not chase the cat then it eats the bear. If something chases the rabbit and the rabbit is big then the rabbit eats the bald eagle. If something is big then it needs the bear. If the rabbit needs the cat then the rabbit chases the bear. If something is rough and it does not chase the bear then it eats the rabbit.
No proof: neither the statement nor its negation follows.
The bear likes the squirrel.
Correct answer: true · Model said: true (65.3% sure)
The problem and its proof
The bald eagle likes the cow. The bald eagle visits the cow. The bear chases the bald eagle. The bear is rough. The cow is cold. The cow likes the bald eagle. The cow visits the bald eagle. The squirrel is young. The squirrel likes the bear. The squirrel visits the cow. If someone chases the squirrel then they like the squirrel. If someone is young and rough then they chase the squirrel. If someone likes the bear then they visit the bald eagle. If someone is cold then they chase the bald eagle. If someone likes the cow then they are blue. If someone visits the bald eagle and they like the squirrel then they like the bear. If someone is cold then they visit the bear. If someone chases the bald eagle then they are young. If someone is young then they chase the squirrel.
The bear chases the bald eagle.→If someone chases the bald eagle then they are young.→The bear is young.
→If someone is young then they chase the squirrel.→The bear chases the squirrel.
→If someone chases the squirrel then they like the squirrel.→The bear likes the squirrel.
Depth 4, open-world task
73.9%
right on 224 of 303 problems wrong; chance 33% (dashed line)
Fiona is green.
Correct answer: unknown · Model said: unknown (77.5% sure)
The problem and its proof
Charlie is white. Fiona is smart. If something is blue then it is big. Big, young things are smart. Big things are green. All white things are blue. Round, big things are white. If Fiona is white and Fiona is blue then Fiona is young.
No proof: neither the statement nor its negation follows.
The mouse is big.
Correct answer: true · Model said: unknown (73.0% sure)
The problem and its proof
The dog is not round. The dog likes the lion. The dog needs the rabbit. The dog does not visit the lion. The lion is big. The lion does not like the mouse. The mouse is nice. The mouse likes the rabbit. The mouse does not need the lion. The mouse visits the rabbit. The rabbit is big. The rabbit is nice. The rabbit is not round. The rabbit likes the dog. The rabbit likes the lion. The rabbit does not visit the lion. If someone visits the mouse then they are big. If someone likes the mouse then the mouse likes the rabbit. If someone visits the mouse then the mouse needs the rabbit. If someone is nice then they need the mouse. If the dog likes the rabbit and the rabbit needs the lion then the rabbit likes the lion. If someone is big then they do not need the dog. If someone needs the rabbit then they visit the mouse.
The dog needs the rabbit.→If someone needs the rabbit then they visit the mouse.→The dog visits the mouse.
→If someone visits the mouse then the mouse needs the rabbit.→The mouse needs the rabbit.
→If someone needs the rabbit then they visit the mouse.→The mouse visits the mouse.
→If someone visits the mouse then they are big.→The mouse is big.
Harry is not cold.
Correct answer: false · Model said: false (56.9% sure)
paraphrasedThe problem and its proof
Bob is a cold and round man who has red and green skin. No one knows Charlie like me and he is a kind guy, very round in the belly and green as grass. Fred seems to be round. Green, big and red are qualities which all describe Harry. A red hued and rough looking person is definitely a young person. A very big person who is green but also red is a rough person. A rather big person is a red person. A round person who is on the young side is also likely to treat people in a kind manner. Any person who can be red and blue at the same time is cold. People who are young are also blue.
The proof that the statement's negation follows:
Harry is green. + Harry is big. + Harry is red.→If someone is green and big and red then they are rough.→Harry is rough.
+ Harry is red.→If someone is red and rough then they are young.→Harry is young.
→If someone is young then they are blue.→Harry is blue.
+ Harry is red.→If someone is red and blue then they are cold.→Harry is cold.
The bald eagle visits the dog.
Correct answer: true · Model said: true (62.4% sure)
The problem and its proof
The bald eagle needs the dog. The bald eagle visits the bear. The bear is green. The bear is round. The dog is kind. The dog is round. The dog needs the bald eagle. The dog needs the bear. The dog sees the lion. The dog visits the lion. The lion is kind. If the bald eagle is round and the bald eagle visits the bear then the bald eagle sees the lion. If the bald eagle visits the dog then the dog sees the bear. If someone visits the dog then they see the dog. If the lion needs the dog and the dog sees the bald eagle then the dog needs the bear. If someone is kind and they visit the lion then they see the bald eagle. If someone is kind then they see the dog. If someone sees the lion then they visit the dog. If someone is kind and they see the bald eagle then the bald eagle is round.
The dog is kind. + The dog visits the lion.→If someone is kind and they visit the lion then they see the bald eagle.→The dog sees the bald eagle.
+ The dog is kind.→If someone is kind and they see the bald eagle then the bald eagle is round.→The bald eagle is round.
+ The bald eagle visits the bear.→If the bald eagle is round and the bald eagle visits the bear then the bald eagle sees the lion.→The bald eagle sees the lion.
→If someone sees the lion then they visit the dog.→The bald eagle visits the dog.
Depth 5, open-world task
75.1%
right on 217 of 289 problems wrong; chance 33% (dashed line)
Anne is not nice.
Correct answer: false · Model said: false (59.2% sure)
The problem and its proof
Anne is round. Bob is round. Bob is smart. Dave is red. Dave is round. Gary is furry. Gary is kind. Gary is nice. Gary is red. Gary is round. Furry people are kind. All kind people are round. If Anne is red then Anne is kind. If someone is round and smart then they are nice. Round people are red. Furry people are round. Round, furry people are smart. Red, kind people are furry.
The proof that the statement's negation follows:
Anne is round.→Round people are red.→Anne is red.
→If Anne is red then Anne is kind.→Anne is kind.
→Red, kind people are furry.→Anne is furry.
+ Anne is round.→Round, furry people are smart.→Anne is smart.
+ Anne is round.→If someone is round and smart then they are nice.→Anne is nice.
The lion eats the lion.
Correct answer: true · Model said: unknown (71.6% sure)
The problem and its proof
The bear chases the lion. The bear chases the mouse. The bear is red. The bear is round. The bear needs the squirrel. The lion eats the bear. The lion is kind. The lion needs the bear. The mouse is round. The squirrel chases the bear. If something eats the bear and the bear eats the lion then the lion is round. If something chases the lion then the lion eats the squirrel. If something is red then it needs the lion. If something eats the squirrel and the squirrel eats the mouse then it chases the lion. If the bear is blue and the bear needs the mouse then the mouse eats the lion. If something eats the lion and it is round then the lion is red. If something needs the lion then it eats the lion.
The bear is red.→If something is red then it needs the lion.→The bear needs the lion.
→If something needs the lion then it eats the lion.→The bear eats the lion.
+ The bear is round.→If something eats the lion and it is round then the lion is red.→The lion is red.
→If something is red then it needs the lion.→The lion needs the lion.
→If something needs the lion then it eats the lion.→The lion eats the lion.
Charlie is not round.
Correct answer: unknown · Model said: unknown (82.2% sure)
The problem and its proof
Charlie is green. Erin is cold. Erin is kind. Erin is quiet. Fiona is cold. Fiona is furry. Fiona is green. Fiona is kind. Fiona is white. Gary is cold. Quiet people are furry. Green, furry people are white. All white people are quiet. If someone is furry then they are round. Kind people are cold. All cold people are white. Quiet, round people are green. Round, white people are cold.
No proof: neither the statement nor its negation follows.
Gary is young.
Correct answer: unknown · Model said: unknown (87.0% sure)
The problem and its proof
Bob is big. Bob is blue. Bob is nice. Bob is rough. Bob is white. Erin is blue. Gary is kind. Harry is kind. Harry is nice. Harry is white. Blue things are rough. All nice, white things are young. If something is big then it is blue. All blue, big things are nice. Big, rough things are kind. All white, rough things are big. If something is young then it is blue. If something is rough then it is white.
No proof: neither the statement nor its negation follows.
The closed-world task (true or false), depth by depth
Depth 0, closed-world task
99.3%
right on 298 of 300 problems wrong; chance 50% (dashed line)
The bald eagle is not young.
Correct answer: false · Model said: false (80.8% sure)
The problem and its proof
The bald eagle is young. Nice, red things are young.
Stated in the text: "The bald eagle is young."
Harry is not quiet.
Correct answer: false · Model said: false (80.8% sure)
The problem and its proof
Bob is nice. Bob is rough. Bob is round. Dave is nice. Erin is cold. Harry is green. Harry is quiet. All rough things are nice. Nice things are rough. All round, quiet things are cold. All big, round things are green. If something is big and quiet then it is round. If something is rough and quiet then it is green. Rough things are quiet. All green things are big.
Stated in the text: "Harry is quiet."
The cow does not eat the squirrel.
Correct answer: false · Model said: false (83.5% sure)
The problem and its proof
The cow eats the squirrel. The cow likes the squirrel. The squirrel is cold. If something eats the cow then it chases the cow.
Stated in the text: "The cow eats the squirrel."
Anne is nice.
Correct answer: false · Model said: false (92.8% sure)
The problem and its proof
Anne is cold. Bob is green. Dave is big. If someone is green and not nice then they are cold. If Anne is nice and Anne is white then Anne is green. All rough people are green. All cold people are rough. All rough people are kind. If Dave is rough and Dave is not white then Dave is kind. If someone is nice and not green then they are big. Kind people are big.
No proof to show: under the closed world this answer holds because something cannot be derived.
Depth 1, closed-world task
90.0%
right on 270 of 300 problems wrong; chance 50% (dashed line)
Bob is rough.
Correct answer: false · Model said: false (97.8% sure)
paraphrasedThe problem and its proof
Young Alan is kind and blue with green and cold traits that show how really big he is. Fred is a kind person and he is also often cold. Gary seems to be round. If someone you know is big, cold, and young, you know they are also kind. All those who are nice and big are red. Often a young, blue person is very kind and then will be nice to others. A young person with red hands is rough around the edges.
No proof to show: under the closed world this answer holds because something cannot be derived.
Harry is not big.
Correct answer: true · Model said: true (89.3% sure)
paraphrasedThe problem and its proof
Alan is a kind person and he is also often cold. Charlie seems to be round. Eric is both young and red, but is nice and kind but he can be cold. That guy Harry sure is nice. Young, red people are usually quite blue too. If a person happens to be big kind and red at the same time they are a young person. When you meet and big and nice man, they will be kind to the earth and live green lifestyle. A kind person who looks blue because he is is cold is usually big in stature.
No proof to show: under the closed world this answer holds because something cannot be derived.
The cat does not need the cow.
Correct answer: true · Model said: true (95.0% sure)
The problem and its proof
The cat likes the lion. The cow likes the rabbit. The cow needs the rabbit. The cow sees the cat. The lion likes the cow. The rabbit is rough. The rabbit is round. If someone needs the rabbit then they need the cow. If someone needs the cow and they do not like the cat then the cow sees the rabbit.
No proof to show: under the closed world this answer holds because something cannot be derived.
Erin is green.
Correct answer: true · Model said: true (91.5% sure)
The problem and its proof
Bob is quiet. Erin is red. Gary is rough. Harry is round. If something is green and cold then it is round. Kind things are quiet. If something is rough then it is kind. If something is green then it is kind. All red things are green. If something is green then it is rough. If Erin is kind then Erin is quiet. All quiet, green things are cold.
Erin is red.→All red things are green.→Erin is green.
Depth 2, closed-world task
72.0%
right on 216 of 300 problems wrong; chance 50% (dashed line)
Fred is red.
Correct answer: true · Model said: false (75.5% sure)
paraphrasedThe problem and its proof
Bob is kind and nice and looks green. Eric, who is relatively young, is also pretty big and tends to be cold. Fred is kind. He is also very cold and blue. That guy Harry sure is nice. People here that are both big and kind while being cold are always red. Kind red people are green on the inside. When someone is kind yet can be cold and blue, they will also be very big. Kind red and round people often look green. Is a kind person if feeling cold and looking red then they are usually young.
Fred is kind. + Fred is cold. + Fred is blue.→If someone is kind and cold and blue then they are big.→Fred is big.
+ Fred is kind. + Fred is cold.→If someone is big and kind and cold then they are red.→Fred is red.
Bob is quiet.
Correct answer: false · Model said: false (94.0% sure)
The problem and its proof
Anne is young. Bob is young. Fiona is quiet. Gary is round. If something is round then it is quiet. All kind things are round.
No proof to show: under the closed world this answer holds because something cannot be derived.
The cow eats the cow.
Correct answer: false · Model said: false (94.7% sure)
The problem and its proof
The bear eats the mouse. The bear likes the mouse. The cow eats the mouse. The cow is big. The dog eats the bear. The mouse eats the cow. The mouse is kind. If someone is rough then they need the cow. If someone needs the cow then they need the bear. If someone is kind then they eat the dog. If someone needs the dog then they eat the dog. If someone likes the cow and the cow eats the dog then the cow is green. If someone is green then they eat the cow. If someone eats the mouse and they need the cow then the cow is rough. If someone needs the cow then they eat the mouse. If the cow eats the mouse and the cow does not need the mouse then the mouse is rough.
No proof to show: under the closed world this answer holds because something cannot be derived.
Harry is not cold.
Correct answer: false · Model said: false (71.9% sure)
paraphrasedThe problem and its proof
Alan rather resembles the rainbow, as he is green, red and blue. Alan is also said to be very kind. Charlie may be round, but he is also kind. Fred seems to be round. Harry is shaped big and round like a balloon. His red color shows how kind he is. A person with big, round, kind face appears to be rough around the edges because they are naive. Rough and big people are always also cold people. If you run across a young person with rough skin and a round figure, you can count on them being kind. Round young people, red with loveliness, are very cold towards others. Someone who is currently blue, big, and cold, is also kind. A person who is green, blue, and red in color will be round shaped. I bet anyone who is round and kind is also pretty young.
Harry is big. + Harry is kind. + Harry is round.→If someone is big and kind and round then they are rough.→Harry is rough.
+ Harry is big.→If someone is rough and big then they are cold.→Harry is cold.
Depth 3, closed-world task
64.3%
right on 193 of 300 problems wrong; chance 50% (dashed line)
The dog is red.
Correct answer: false · Model said: false (84.0% sure)
The problem and its proof
The bald eagle chases the dog. The bald eagle is blue. The bald eagle is cold. The bald eagle is kind. The dog visits the bald eagle. The dog visits the tiger. The rabbit chases the tiger. The rabbit eats the dog. The rabbit is kind. The rabbit visits the bald eagle. The tiger eats the dog. The tiger is green. If someone is kind and red then they chase the dog. Green, blue people are red. If someone eats the bald eagle then they are blue. If someone chases the bald eagle then the bald eagle eats the dog. If someone visits the dog and they are red then the dog eats the bald eagle. If someone is blue then they visit the dog. If the bald eagle visits the dog and the bald eagle eats the dog then the dog is green. If the bald eagle eats the dog then the dog is red. If someone visits the tiger and they are not kind then the tiger is blue.
No proof to show: under the closed world this answer holds because something cannot be derived.
Dave is furry.
Correct answer: false · Model said: false (71.9% sure)
The problem and its proof
Anne is quiet. Dave is green. Dave is quiet. If something is kind and not rough then it is white. All quiet things are white. All rough things are white. Kind, white things are furry. All green, rough things are furry. Green, furry things are kind. If Anne is rough and Anne is white then Anne is furry. If something is white then it is green.
No proof to show: under the closed world this answer holds because something cannot be derived.
The lion does not see the cat.
Correct answer: false · Model said: false (65.4% sure)
The problem and its proof
The cat visits the mouse. The dog sees the cat. The lion visits the dog. The mouse likes the cat. If the lion likes the dog then the lion visits the cat. If someone visits the mouse then they like the dog. If someone likes the dog then the dog sees the lion. If the mouse is not rough then the mouse sees the dog. If someone likes the mouse then they are round. If someone sees the lion then the lion sees the cat.
The cat visits the mouse.→If someone visits the mouse then they like the dog.→The cat likes the dog.
→If someone likes the dog then the dog sees the lion.→The dog sees the lion.
→If someone sees the lion then the lion sees the cat.→The lion sees the cat.
Dave is round.
Correct answer: true · Model said: false (75.5% sure)
paraphrasedThe problem and its proof
That guy Alan sure is nice. Bob may be round, but he is also kind. Green, big and red are qualities which all describe Dave. That guy Eric sure is nice. Anyone feeling cold is bound to act nice. Big people with red hair are cold because they cannot find coats that fit. People who are said to be big and nice are round. Having cold and red qualities means a person will also have a rough quality.
Dave is red. + Dave is big.→If someone is red and big then they are cold.→Dave is cold.
→If someone is cold then they are nice.→Dave is nice.
+ Dave is big.→If someone is big and nice then they are round.→Dave is round.
Depth 4, closed-world task
55.7%
right on 167 of 300 problems wrong; chance 50% (dashed line)
Charlie is green.
Correct answer: true · Model said: false (94.0% sure)
paraphrasedThe problem and its proof
Charlie is both young and red, but is nice and kind but he can be cold. For being so cold, it's good Dave can remain nice. Eric, who is relatively young, is also pretty big and tends to be cold. The young person who is always feeling cold is named Gary. Folks who are rough and tumble yet gentle and kind are blue. Big, kind folks are green ones. People with big smiles and round eyes will have red hair. Any human who is rough, green, and kind, has a blue personality. Young people who feel cold and are still kind usually look rough. When someone is blue, nice and young they will also be as big as the sun. When you run into someone who is rough and green at the same time, they will also be red.
Charlie is cold. + Charlie is kind. + Charlie is young.→If someone is cold and kind and young then they are rough.→Charlie is rough.
+ Charlie is kind.→If someone is rough and kind then they are blue.→Charlie is blue.
+ Charlie is young. + Charlie is nice.→If someone is young and blue and nice then they are big.→Charlie is big.
+ Charlie is kind.→If someone is big and kind then they are green.→Charlie is green.
Dave is not round.
Correct answer: true · Model said: true (86.7% sure)
The problem and its proof
Charlie is big. Charlie is round. Dave is green. All rough people are red. If someone is rough then they are smart. If Charlie is not green then Charlie is rough. If someone is red then they are round. If someone is rough then they are round. If someone is smart and not white then they are rough.
No proof to show: under the closed world this answer holds because something cannot be derived.
Gary is not cold.
Correct answer: true · Model said: true (81.8% sure)
The problem and its proof
Dave is red. Gary is green. All kind things are nice. If Gary is not kind then Gary is white. If Dave is white then Dave is cold. If Gary is green and Gary is not white then Gary is big. If something is big and white then it is red. If something is red then it is big. If something is big and not red then it is cold. If something is red and kind then it is cold.
No proof to show: under the closed world this answer holds because something cannot be derived.
Alan is red.
Correct answer: true · Model said: false (82.6% sure)
paraphrasedThe problem and its proof
Alan seems big and rough, but he's actually quite kind. Dave is young and green. This makes him cold to others. Eric is green with envy, and rough with his kids, but overall is just a nice, round individual. People,though kind and rough are also really nice. A big, round person who is kind is also red. Do not fear big cold people for they are often kind. A person that is round and somewhat green while being nice tends to be red as well. When you meet and big and nice man, they will be kind to the earth and live green lifestyle. People who are young are also blue. Green folks who are nice and rough are a round shape.
Alan is kind. + Alan is rough.→If someone is kind and rough then they are nice.→Alan is nice.
+ Alan is big. + Alan is kind.→If someone is big and kind and nice then they are green.→Alan is green.
+ Alan is rough.→If someone is nice and rough and green then they are round.→Alan is round.
+ Alan is big. + Alan is kind.→If someone is big and round and kind then they are red.→Alan is red.
Depth 5, closed-world task
49.3%
right on 148 of 300 problems wrong; chance 50% (dashed line)
The rabbit sees the lion.
Correct answer: true · Model said: true (66.7% sure)
The problem and its proof
The bear sees the tiger. The bear visits the lion. The bear visits the rabbit. The lion is big. The lion is nice. The lion needs the bear. The lion visits the tiger. The rabbit is big. The rabbit is nice. The rabbit needs the bear. The rabbit visits the tiger. The tiger is nice. The tiger needs the rabbit. The tiger visits the lion. If someone is nice and they see the bear then they see the lion. If someone visits the bear then they visit the rabbit. If the tiger visits the rabbit then the rabbit sees the bear. If someone sees the tiger then the tiger is blue. If the lion visits the rabbit and the lion is not blue then the rabbit needs the lion. If someone is blue then they visit the bear. If the rabbit does not need the lion then the lion sees the rabbit.
The bear sees the tiger.→If someone sees the tiger then the tiger is blue.→The tiger is blue.
→If someone is blue then they visit the bear.→The tiger visits the bear.
→If someone visits the bear then they visit the rabbit.→The tiger visits the rabbit.
→If the tiger visits the rabbit then the rabbit sees the bear.→The rabbit sees the bear.
+ The rabbit is nice.→If someone is nice and they see the bear then they see the lion.→The rabbit sees the lion.
Fiona is not big.
Correct answer: false · Model said: true (81.8% sure)
The problem and its proof
Charlie is green. Charlie is quiet. Charlie is round. Dave is cold. Fiona is round. Gary is big. Gary is round. Round, green people are white. If someone is blue then they are big. If someone is white and round then they are cold. If Fiona is round then Fiona is green. If someone is cold then they are blue.
Fiona is round.→If Fiona is round then Fiona is green.→Fiona is green.
+ Fiona is round.→Round, green people are white.→Fiona is white.
+ Fiona is round.→If someone is white and round then they are cold.→Fiona is cold.
→If someone is cold then they are blue.→Fiona is blue.
→If someone is blue then they are big.→Fiona is big.
The rabbit does not need the rabbit.
Correct answer: false · Model said: true (94.0% sure)
The problem and its proof
The cat is blue. The cat is rough. The cat needs the dog. The cow is red. The cow likes the cat. The cow likes the dog. The dog is rough. The dog visits the rabbit. The rabbit is blue. The rabbit is rough. The rabbit needs the cow. The rabbit visits the cow. If someone visits the cat then they are cold. If someone visits the cat then they need the rabbit. If someone likes the dog then they visit the cat. If someone is rough then they like the rabbit. If someone needs the dog then they visit the cat. If someone needs the rabbit then the rabbit likes the dog. If the dog visits the rabbit and the rabbit visits the dog then the dog is nice.
The cat needs the dog.→If someone needs the dog then they visit the cat.→The cat visits the cat.
→If someone visits the cat then they need the rabbit.→The cat needs the rabbit.
→If someone needs the rabbit then the rabbit likes the dog.→The rabbit likes the dog.
→If someone likes the dog then they visit the cat.→The rabbit visits the cat.
→If someone visits the cat then they need the rabbit.→The rabbit needs the rabbit.
Anne is rough.
Correct answer: false · Model said: false (76.6% sure)
The problem and its proof
Anne is blue. Bob is quiet. Bob is red. Dave is cold. Dave is quiet. Harry is blue. Harry is cold. Harry is nice. Harry is quiet. Harry is rough. If something is cold and furry then it is blue. Cold, red things are furry. Quiet things are red. If something is furry then it is nice. Red things are quiet. If something is nice then it is rough. If something is cold then it is red. If something is blue and red then it is nice.
No proof to show: under the closed world this answer holds because something cannot be derived.
Where each one is better
Open world, Jev vs GLiDE. Jev is clearly ahead on false answers and true answers. GLiDE is clearly ahead on unknown answers (+21.7 points). Too close to call: depth 0, depth 1, depth 2, depth 3, depth 4 and depth 5.
Closed world, Jev vs GLiDE. Jev is clearly ahead on depth 1, depth 2, depth 3, depth 4, depth 5, false answers and true answers. GLiDE is not clearly ahead on any depth or answer class. Too close to call: depth 0.
A slice counts as clearly ahead when its paired 95% interval excludes zero.
True, false and unknown, open world
By correct answer, Open world
Correct answer
Items
Jev
GLiDE
Best constant
false
605
88.3%85.5–90.9
76.0%72.4–79.5
one answer
true
605
88.8%86.1–91.1
77.0%73.6–80.5
one answer
unknown
590
74.2%70.7–77.6
95.9%94.2–97.5
one answer
True, false and unknown, closed world
By correct answer, Closed world
Correct answer
Items
Jev
GLiDE
Best constant
false
900
88.9%86.9–90.9
67.9%64.9–71.0
one answer
true
900
89.7%87.7–91.6
75.7%72.7–78.3
one answer
Paraphrased vs templated rules, open world
By wording, Open world
Wording
Items
Jev
GLiDE
Best constant
templated
1,298
86.1%84.2–87.9
86.0%84.1–87.8
34.5%
paraphrased (NatLang)
502
78.1%74.5–82.1
74.9%71.3–78.9
35.9%
Paraphrased vs templated rules, closed world
By wording, Closed world
Wording
Items
Jev
GLiDE
Best constant
templated
1,306
94.0%92.8–95.3
76.5%74.4–78.9
50.2%
paraphrased (NatLang)
494
76.7%73.1–80.4
59.3%54.9–63.6
50.6%
Answer by answer: Jev and GLiDE on the same problems
Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.
right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.
Open world: true, false or unknown
Jev, open-world task
Correct ↓ · said →
true
false
unknown
true
537
5
63
false
10
534
61
unknown
124
28
438
GLiDE, open-world task
Correct ↓ · said →
true
false
unknown
true
466
0
139
false
2
460
143
unknown
17
7
566
Jev said true on 37% of problems; the correct answer is true on 34%. GLiDE said unknown on 47% of problems; the correct answer is unknown on 33%.
Closed world: true or false
Jev, closed-world task
Correct ↓ · said →
true
false
true
807
93
false
100
800
GLiDE, closed-world task
Correct ↓ · said →
true
false
true
681
219
false
289
611
Jev said true on 50% of problems; the correct answer is true on 50%. GLiDE said true on 54% of problems; the correct answer is true on 50%.
Where GLiDE errs
Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.
right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.
GLiDE, open-world task
GLiDE, open-world task
Correct ↓ · said →
true
false
unknown
Recall
true
466
0
139
77.0%
false
2
460
143
76.0%
unknown
17
7
566
95.9%
GLiDE answered false 25.9%, true 26.9%, unknown 47.1% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.
GLiDE, closed-world task
GLiDE, closed-world task
Correct ↓ · said →
true
false
Recall
true
681
219
75.7%
false
289
611
67.9%
GLiDE answered false 46.1%, true 53.9% of the time; the correct answers are false 50.0%, true 50.0%.
GLiDE at depth 0 and depth 5
GLiDE, open-world task, depth 0
Correct ↓ · said →
true
false
unknown
true
98
0
2
false
0
98
2
unknown
2
1
97
GLiDE, open-world task, depth 5
Correct ↓ · said →
true
false
unknown
true
68
0
33
false
0
66
35
unknown
3
1
83
Open-world task. At depth 0, GLiDE was right on 98% of problems (the green diagonal); at depth 5, on 75%. The answer whose share moved most is unknown: 34% of its answers at depth 0, 52% at depth 5, against 33% and 30% of the correct answers.
The closed-world task at depth 0 and depth 5
GLiDE, closed-world task, depth 0
Correct ↓ · said →
true
false
true
150
0
false
2
148
GLiDE, closed-world task, depth 5
Correct ↓ · said →
true
false
true
86
64
false
88
62
Closed-world task. At depth 0, GLiDE was right on 99% of problems (the green diagonal); at depth 5, on 49%. The answer whose share moved most is false: 49% of its answers at depth 0, 42% at depth 5, against 50% and 50% of the correct answers.
GLiDE across every breakdown
For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.
GLiDE's median open-world decision took 1.2 s against Jev's 189 ms, and 243 of its 3,600 decisions took ten seconds or more (the slowest 135 s). At list price GLiDE cost $213 per million decisions against Jev's $23: 9.3 times as much per decision and 10.4 times as much per correct decision.
Fastino is upfront that GLiDE works this way. It writes that "traditional decision models score a set of options in a single fixed pass," while GLiDE "first produces a fast probability distribution, then allocates additional reasoning when the leading result is uncertain," so it can "spend more computation on difficult decisions while keeping straightforward ones fast." Fastino bills that reasoning as tokens, and its API documentation recommends a read timeout of at least 300 seconds (Fastino). TypeSafe describes Jev, a System One model, as generating all outputs "in a single query," in parallel, in 70 to 500 milliseconds, so every decision takes about the same time and money (TypeSafe).
Every decision's time
Jev
GLiDE
Average time per decision, by proof depth
Open world: true, false or unknown
Closed world: true or false
Model
Task
Depth 0
Depth 1
Depth 2
Depth 3
Depth 4
Depth 5
Jev
Open world
207 ms
202 ms
209 ms
203 ms
207 ms
209 ms
Jev
Closed world
177 ms
177 ms
175 ms
174 ms
174 ms
178 ms
GLiDE
Open world
1.1 s
3.4 s
5.3 s
6.7 s
6.6 s
4.4 s
GLiDE
Closed world
921 ms
2.4 s
2.5 s
2.5 s
2.2 s
2.2 s
List-price cost per million decisions, by proof depth
Open world: true, false or unknown
Closed world: true or false
Model
Task
Depth 0
Depth 1
Depth 2
Depth 3
Depth 4
Depth 5
Jev
Open world
$23
$23
$23
$24
$24
$24
Jev
Closed world
$22
$22
$22
$23
$23
$23
GLiDE
Open world
$97
$213
$288
$374
$391
$305
GLiDE
Closed world
$84
$157
$166
$167
$159
$157
GLiDE's fast and slow answers
Split at 2 seconds. The slow answers are the ones where GLiDE spent extra computation; each column says what that cost and what it bought.
Task
Answers
Count
Average time
Input tokens billed, average
Right
Stated probability, average
Open world
Under 2 s
1,157
967 ms
299
78.5%
84.8%
Open world
Over 2 s
643
11 s
2,057
90.8%
72.2%
Closed world
Under 2 s
1,559
873 ms
285
68.2%
86.4%
Closed world
Over 2 s
241
10 s
1,852
95.0%
76.1%
When GLiDE took longer, it was more often right, and it stated lower probabilities for those answers. It decides when to spend the extra time from its first probability, so where that probability is high and wrong, it answers quickly and wrong.
Times are wall time per request; GLiDE timed in its scored run with 8 requests in flight, the others one request at a time. Costs are list prices times the tokens each API reported. Cost per decision for every hosted model · Speed, size and memory.
Consistency and calibration
GLiDE was not asked a second time: at its speed a rerun of every item would have taken hours, so its consistency is not measured.
Calibration on the open world task: GLiDE's probability for its own answer averaged 80.3% against 82.9% accuracy; expected calibration error 6.7 points.
Calibration on the closed world task: GLiDE's probability for its own answer averaged 85.0% against 71.8% accuracy; expected calibration error 15.0 points.
Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.56 on the open world task against Jev's 0.85; 0.52 on the closed world task against Jev's 0.86.
Times are open-world, one request at a time except GLiDE, timed in its scored run with 8 requests in flight. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.
GLiDE
Parameters
undisclosed
Architecture
undisclosed ("thinking" decision model that adds reasoning when its first distribution is uncertain)
Weights
not available (hosted)
Where it ran
Fastino API, POST /v1/systemone, model fastino/GLiDE (same System One request as Jev)
Maker's stated hardware
not applicable (hosted)
Price
$0.30 per million input tokens; output free
Time per decision, open world
median 1,153 ms, 90th percentile 10,152 ms, over the network
Time per decision, closed world
median 923 ms, 90th percentile 5,367 ms, over the network
Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.
GLiDE GLiDE is a hosted decision model made by Fastino Labs, which describes it as a "thinking" decision model: it forms a probability over the options and spends more computation when the leading option is uncertain. Fastino serves it on the same System One request format as Jev, so it got exactly Jev's request: the text, the question and the options, one request per item. Its size and architecture are not published.