Skip to main content
HardDecisions

Comparison · Jev vs Laya

Jev vs Laya: hosted and open decision models on deep reasoning

Laya is a small open decision model built on an encoder. Both it and Jev answered the same problems.

Overall: Jev and Laya

Open world: true, false or unknown

  • Jev82.2 to 85.5 (95%), 1,800 items83.8%
  • Laya40.3 to 44.5 (95%), 1,800 items42.4%

Jev minus Laya: +41.4 points on the same 1,800 items, 95% interval +38.8 to +44.0: Jev is clearly ahead.

Closed world: true or false

  • Jev87.8 to 90.7 (95%), 1,800 items89.3%
  • Laya53.3 to 58.1 (95%), 1,800 items55.6%

Jev minus Laya: +33.7 points on the same 1,800 items, 95% interval +30.7 to +36.5: Jev is clearly ahead.

Jev vs Laya by proof depth

Each point is about 300 problems that need that many inference steps, with its 95% interval.

Open world: true, false or unknown
  • Jev
  • Laya
0%25%50%75%100%012345proof depth (inference steps)chance 33%
Closed world: true or false
  • Jev
  • Laya
0%25%50%75%100%012345proof depth (inference steps)chance 50%

Jev minus Laya by depth, open world

depth 0300 items
+35.0
depth 1302 items
+45.7
depth 2303 items
+43.6
depth 3303 items
+41.6
depth 4303 items
+36.0
depth 5289 items
+46.7

Jev minus Laya by depth, closed world

depth 0300 items
+27.3
depth 1300 items
+42.7
depth 2300 items
+37.3
depth 3300 items
+33.7
depth 4300 items
+17.7
depth 5300 items
+43.3

Paired differences: positive means Jev was more accurate on the same items.

Accuracy by depth, open world

By depth, Open world
Proof depthItemsJevLayaBest constant
depth 030097.7% 96.0–99.362.7% 57.0–68.333.3%
depth 130287.7% 83.8–91.442.1% 36.8–47.733.4%
depth 230384.8% 80.2–88.441.3% 36.0–46.933.3%
depth 330379.9% 75.2–84.538.3% 33.0–44.633.3%
depth 430371.9% 66.7–77.236.0% 31.0–41.633.3%
depth 528981.0% 76.5–85.534.3% 29.1–39.834.9%

Accuracy by depth, closed world

By depth, Closed world
Proof depthItemsJevLayaBest constant
depth 030099.3% 98.3–100.072.0% 66.7–76.750.0%
depth 130093.7% 91.0–96.351.0% 45.0–56.050.0%
depth 230092.0% 88.7–94.754.7% 48.7–59.750.0%
depth 330085.0% 81.3–89.351.3% 45.7–56.750.0%
depth 430076.3% 71.7–81.058.7% 53.0–64.350.0%
depth 530089.3% 85.7–92.346.0% 40.3–51.050.0%

What Laya got right and wrong, depth by depth

At each proof depth, Laya's accuracy on every problem at that depth, then four of those problems. The share of right and wrong among the four matches the accuracy, rounded to the nearest quarter, so they show what its success and failure look like, not a separate score. What proof depth means.

How the four are chosen: For each scored engine, task and depth (0-5): the model's accuracy on all sampled problems at that depth, and four example problems. The number of wrong examples is 4 x (1 - accuracy), rounded half up (so 49.5% accuracy shows 2 right and 2 wrong); the rest are right. Examples are drawn at random from the model's right and wrong answers at that depth, with a fixed seed derived from engine, task and depth, so the draw is reproducible and blind to anything but correctness. Each example carries the exact text the model saw, the statement, the gold answer, the model's answer and its stated probability (GPT-6 Luna's from the log-probability probe, when the probe returned the same answer), and the dataset's gold proof as steps when one exists (true and false answers). (GPT-6 Luna ran with reasoning off, under the same one-request protocol as every model.)

Depth 0, open-world task

62.7%
right on 188 of 300 problems wrong; chance 33% (dashed line)
right

Charlie is kind.

Correct answer: true · Model said: true (70.5% sure)

paraphrased
The problem and its proof

Bob is a cold and round man who has red and green skin. No one knows Charlie like me and he is a kind guy, very round in the belly and green as grass. Fred seems to be round. Green, big and red are qualities which all describe Harry. A red hued and rough looking person is definitely a young person. A very big person who is green but also red is a rough person. A rather big person is a red person. A round person who is on the young side is also likely to treat people in a kind manner. Any person who can be red and blue at the same time is cold. People who are young are also blue.

Stated in the text: "Charlie is kind."

wrong

The bear is not round.

Correct answer: true · Model said: false (59.6% sure)

The problem and its proof

The bear chases the cat. The bear chases the rabbit. The bear eats the cat. The bear is not big. The bear is green. The bear is not round. The cat chases the mouse. The cat eats the bear. The cat is not cold. The cat visits the mouse. The mouse chases the bear. The mouse eats the bear. The mouse is round. The rabbit chases the cat. The rabbit does not eat the cat. The rabbit is young. If something chases the bear then it is green. If something visits the mouse then it is not young. If something visits the mouse then it is big.

Stated in the text: "The bear is not round."

right

The dog visits the cat.

Correct answer: false · Model said: false (73.2% sure)

The problem and its proof

The cat needs the cow. The cat needs the lion. The cow is nice. The cow visits the lion. The dog does not visit the cat. The dog visits the lion. The lion sees the cow. If something sees the cow and the cow sees the lion then it is nice. If something is young and nice then it sees the cat. If something sees the dog and it sees the cow then it is green. If something needs the cow and it does not need the lion then the lion is blue. If something sees the lion and it visits the lion then the lion visits the cow. All nice things are young. If something sees the cat and the cat needs the cow then the cat is nice. If the cat is not nice then the cat is not blue.

Stated in the text: "The dog does not visit the cat."

right

The cat is red.

Correct answer: false · Model said: false (69.6% sure)

The problem and its proof

The cat chases the rabbit. The cat is not big. The cat is blue. The cat is not cold. The cat is not red. The cat is not rough. The cat does not like the rabbit. The cat does not see the rabbit. The rabbit does not chase the cat. The rabbit is big. The rabbit is blue. The rabbit is cold. The rabbit is red. The rabbit is rough. The rabbit does not like the cat. The rabbit sees the cat. If someone is red then they like the rabbit. If someone is blue and not big then they do not like the cat. If someone likes the rabbit and they are cold then they see the rabbit.

Stated in the text: "The cat is not red."

Depth 1, open-world task

42.1%
right on 127 of 302 problems wrong; chance 33% (dashed line)
right

The cat is kind.

Correct answer: false · Model said: false (73.3% sure)

The problem and its proof

The bear chases the cat. The bear visits the tiger. The cat chases the dog. The cat is not young. The dog does not chase the bear. The dog eats the bear. The tiger visits the cat. If something chases the cat then the cat is not kind. If something visits the tiger then the tiger chases the cat. If the cat eats the dog and the dog chases the tiger then the tiger eats the bear. If something chases the cat then it is kind. If something visits the tiger and it does not eat the tiger then the tiger visits the bear. If something is kind then it visits the tiger.

The proof that the statement's negation follows:

  1. The bear chases the cat.If something chases the cat then the cat is not kind.The cat is not kind.
right

Gary is white.

Correct answer: false · Model said: false (94.6% sure)

The problem and its proof

Erin is rough. Gary is blue. Gary is nice. Blue things are kind. Blue, kind things are not rough. If something is nice then it is smart. Smart, rough things are cold. If something is smart and not rough then it is not cold. All kind things are nice. All kind, cold things are nice. Blue, nice things are not white.

The proof that the statement's negation follows:

  1. Gary is blue. + Gary is nice.Blue, nice things are not white.Gary is not white.
wrong

Eric is red.

Correct answer: true · Model said: false (58.6% sure)

paraphrased
The problem and its proof

Bob is green and cold too. That guy Dave sure is nice. Round, nice Eric felt as blue and big as he had ever been. That guy Gary sure is nice. Folks who are rough and tumble yet gentle and kind are blue. Being young and cold also means you're green. All those who are nice and big are red. Nice, young red people will also turn out to always be green. A kind person will certainly be rough as well. A person who is blue, red and nice is usually considered to be a young person. A kind person will certainly be young.

  1. Eric is nice. + Eric is big.If someone is nice and big then they are red.Eric is red.
wrong

Gary is green.

Correct answer: true · Model said: false (55.4% sure)

paraphrased
The problem and its proof

Bob is a kind person and he is also often cold. Dave is a big fellow, often blue and sad, but he is nice. Eric is young and green. This makes him cold to others. Even though Gary is round and big, he is very kind. People here that are both big and kind while being cold are always red. Red people are almost always rough. Big, kind folks are green ones. A round shaped kind person who is colored green will be cold natured.

  1. Gary is big. + Gary is kind.If someone is big and kind then they are green.Gary is green.

Depth 2, open-world task

41.3%
right on 125 of 303 problems wrong; chance 33% (dashed line)
right

The cow visits the cow.

Correct answer: false · Model said: false (64.8% sure)

The problem and its proof

The bear chases the cat. The bear is young. The bear visits the cat. The bear visits the cow. The cat chases the cow. The cat is cold. The cat is red. The cat is not round. The cat visits the cow. The cow chases the bear. The cow chases the cat. The cow is not big. The cow is red. The cow is young. The cow likes the bear. If the bear visits the cow then the cow is round. If something visits the bear then the bear is big. If something is round then it does not visit the cow.

The proof that the statement's negation follows:

  1. The bear visits the cow.If the bear visits the cow then the cow is round.The cow is round.
  2. If something is round then it does not visit the cow.The cow does not visit the cow.
wrong

The mouse does not need the mouse.

Correct answer: true · Model said: false (83.3% sure)

The problem and its proof

The cat is young. The cow is young. The cow visits the mouse. The mouse eats the cat. The mouse needs the cat. The mouse needs the rabbit. The mouse does not visit the cow. The mouse visits the rabbit. The rabbit is round. The rabbit does not need the cat. The rabbit visits the mouse. If someone eats the cat and they eat the cow then they do not need the mouse. If someone visits the cat then the cat eats the cow. All nice people are red. If the mouse needs the rabbit and the mouse eats the cat then the mouse eats the cow. If someone is red and they visit the mouse then the mouse needs the cow. If someone needs the cow then the cow visits the cat. All round people are nice. If someone needs the cat and the cat visits the rabbit then they do not eat the rabbit.

  1. The mouse needs the rabbit. + The mouse eats the cat.If the mouse needs the rabbit and the mouse eats the cat then the mouse eats the cow.The mouse eats the cow.
  2. + The mouse eats the cat.If someone eats the cat and they eat the cow then they do not need the mouse.The mouse does not need the mouse.
wrong

The bald eagle likes the dog.

Correct answer: unknown · Model said: false (85.8% sure)

The problem and its proof

The bald eagle chases the dog. The bald eagle eats the dog. The bald eagle is big. The bald eagle likes the bear. The bear eats the dog. The bear is kind. The dog chases the tiger. The dog eats the bald eagle. The dog is kind. The tiger chases the bear. The tiger eats the bear. The tiger is big. The tiger is round. The tiger is young. If someone is kind then they chase the bald eagle. If someone chases the bear and the bear likes the tiger then they eat the bald eagle. If someone is kind then they eat the bald eagle. If someone likes the dog then they like the tiger. If someone likes the tiger then they are young. If someone eats the dog then the dog is big. If someone eats the bald eagle then they like the dog. If someone likes the dog and they do not chase the tiger then the dog chases the bald eagle.

No proof: neither the statement nor its negation follows.

right

Bob is rough.

Correct answer: false · Model said: false (65.7% sure)

The problem and its proof

Bob is white. If someone is round and smart then they are furry. If someone is nice then they are furry. If Bob is white then Bob is furry. All furry people are not rough. If Bob is smart then Bob is not white. All rough people are not kind. Rough, round people are kind. If someone is white and not rough then they are not kind.

The proof that the statement's negation follows:

  1. Bob is white.If Bob is white then Bob is furry.Bob is furry.
  2. All furry people are not rough.Bob is not rough.

Depth 3, open-world task

38.3%
right on 116 of 303 problems wrong; chance 33% (dashed line)
right

Fred is big.

Correct answer: true · Model said: true (80.9% sure)

paraphrased
The problem and its proof

Bob may be young but he is nice, wears green shoes and is cold. Eric was proud of being round, yet rough. His red cheeks glowed. Fred is feeling cold -- he is also feeling quite rough, red, nice and round. Folks who are rough and tumble yet gentle and kind are blue. When green, young and round fits a person, you'll see that rough will also fit. Cold and red people are always kind to others. If you know someone who is nice, green, and big, you know a young person. A person who is cold and red and blue is big.

  1. Fred is cold. + Fred is red.If someone is cold and red then they are kind.Fred is kind.
  2. + Fred is rough.If someone is rough and kind then they are blue.Fred is blue.
  3. + Fred is red. + Fred is cold.If someone is red and blue and cold then they are big.Fred is big.
right

The dog sees the rabbit.

Correct answer: false · Model said: false (66.3% sure)

The problem and its proof

The cow does not chase the dog. The cow sees the squirrel. The dog sees the cow. The rabbit is red. The rabbit sees the squirrel. The squirrel eats the cow. The squirrel does not see the dog. If someone sees the cow then they are big. If someone chases the cow and they do not see the cow then the cow is big. If someone eats the cow and they see the rabbit then the cow is big. If someone is cold and big then they do not see the rabbit. If someone eats the cow then the cow is kind. All big people are cold.

The proof that the statement's negation follows:

  1. The dog sees the cow.If someone sees the cow then they are big.The dog is big.
  2. All big people are cold.The dog is cold.
  3. If someone is cold and big then they do not see the rabbit.The dog does not see the rabbit.
wrong

Bob is red.

Correct answer: true · Model said: false (59.0% sure)

paraphrased
The problem and its proof

For being so cold, it's good Alan can remain nice. Bob is green, but often kind, even when he is blue and cold. Eric is a big guy who is colored red from a sunburn. He also has a cold. After Harry got wet in the rain, he feels cold. He also looks green but big. People here that are both big and kind while being cold are always red. A person who is kind, cold, and rough is big. Young, red people are usually quite blue too. A blue, cold natured person will be rough textured. I've noticed that nice people who have rough, green skin have a tendancy to be round. If someone is green and naive they may also have red, rough skin. It is a safe guess then that they are also round. Nice people who feel blue often look green when ill.

  1. Bob is blue. + Bob is cold.If someone is blue and cold then they are rough.Bob is rough.
  2. + Bob is kind. + Bob is cold.If someone is kind and cold and rough then they are big.Bob is big.
  3. + Bob is kind. + Bob is cold.If someone is big and kind and cold then they are red.Bob is red.
wrong

Fred is young.

Correct answer: true · Model said: false (77.3% sure)

paraphrased
The problem and its proof

Alan seems to be round. The young person who is always feeling cold is named Charlie. Eric was proud of being round, yet rough. His red cheeks glowed. Fred is nice and kind even though people make fun of his red and green skin. Fred always feels cold. Why is it that every round person that is blue is so young. A young person who is big and rough and big is also usually round. A nice person is inevitably round as well. People who are young are also blue. People who are round and behave in a cold way are surely blue.

  1. Fred is nice.If someone is nice then they are round.Fred is round.
  2. + Fred is cold.If someone is round and cold then they are blue.Fred is blue.
  3. If someone is blue and round then they are young.Fred is young.

Depth 4, open-world task

36.0%
right on 109 of 303 problems wrong; chance 33% (dashed line)
right

Gary is furry.

Correct answer: false · Model said: false (73.5% sure)

The problem and its proof

Bob is kind. Bob is red. Bob is round. Charlie is kind. Fiona is not green. Fiona is smart. Gary is round. All round people are quiet. All smart people are round. If someone is furry and not green then they are smart. Quiet people are smart. If someone is kind and quiet then they are not furry. If someone is smart then they are kind. If someone is kind and not furry then they are red.

The proof that the statement's negation follows:

  1. Gary is round.All round people are quiet.Gary is quiet.
  2. Quiet people are smart.Gary is smart.
  3. If someone is smart then they are kind.Gary is kind.
  4. If someone is kind and quiet then they are not furry.Gary is not furry.
wrong

Charlie is young.

Correct answer: unknown · Model said: false (85.7% sure)

The problem and its proof

Anne is green. Anne is red. Charlie is red. Erin is cold. Erin is green. Erin is nice. Erin is quiet. Erin is red. Erin is white. Erin is young. Cold, red people are green. All quiet people are green. If someone is nice then they are young. If someone is green and white then they are young. All red, white people are nice. If someone is green then they are white.

No proof: neither the statement nor its negation follows.

wrong

Alan is not nice.

Correct answer: false · Model said: true (66.1% sure)

paraphrased
The problem and its proof

Alan has big, red cheeks with a rough texture. He always wear green sweaters because he is easily cold. For being so cold, it's good Bob can remain nice. That guy Gary sure is nice. Harry is a kind person and he is also often cold. People,though kind and rough are also really nice. Big people who are red in color are almost always young. A big, round person who is kind is also red. A big and round individual is sure to be a kind individual, too. A young person who is big and rough and big is also usually round. If a person is both cold and round, that person is also someone who is red. When somebody is red, blue and round, you can bet they are also rough.

The proof that the statement's negation follows:

  1. Alan is big. + Alan is red.If someone is big and red then they are young.Alan is young.
  2. + Alan is rough. + Alan is big.If someone is rough and big and young then they are round.Alan is round.
  3. + Alan is big.If someone is big and round then they are kind.Alan is kind.
  4. + Alan is rough.If someone is kind and rough then they are nice.Alan is nice.
wrong

The cow visits the cow.

Correct answer: true · Model said: false (91.9% sure)

The problem and its proof

The bear is rough. The bear needs the dog. The bear visits the cat. The bear visits the dog. The cat is blue. The cat likes the cow. The cow is rough. The cow likes the bear. The dog needs the bear. The dog visits the cat. All blue people are big. If someone visits the cow then they need the cow. If someone is nice then they like the cow. If someone likes the cat and they are nice then they are big. If someone is blue then they visit the cow. If the cat needs the cow then the cow is blue.

  1. The cat is blue.If someone is blue then they visit the cow.The cat visits the cow.
  2. If someone visits the cow then they need the cow.The cat needs the cow.
  3. If the cat needs the cow then the cow is blue.The cow is blue.
  4. If someone is blue then they visit the cow.The cow visits the cow.

Depth 5, open-world task

34.3%
right on 99 of 289 problems wrong; chance 33% (dashed line)
wrong

Gary is not quiet.

Correct answer: true · Model said: false (55.1% sure)

The problem and its proof

Bob is not furry. Bob is quiet. Charlie is furry. Charlie is kind. Dave is green. Dave is kind. Dave is round. Dave is white. Dave is young. Gary is furry. Gary is green. If something is white then it is not quiet. All young, green things are round. All green, white things are kind. If something is young and not furry then it is kind. All round, green things are kind. If Charlie is white then Charlie is green. Kind, green things are white. All furry things are young. If Dave is quiet and Dave is furry then Dave is not kind.

  1. Gary is furry.All furry things are young.Gary is young.
  2. + Gary is green.All young, green things are round.Gary is round.
  3. + Gary is green.All round, green things are kind.Gary is kind.
  4. + Gary is green.Kind, green things are white.Gary is white.
  5. If something is white then it is not quiet.Gary is not quiet.
wrong

Anne is not nice.

Correct answer: unknown · Model said: false (85.7% sure)

The problem and its proof

Anne is blue. Erin is quiet. Fiona is big. Fiona is furry. Fiona is nice. Harry is furry. Harry is not rough. Round things are blue. If something is round and not big then it is not nice. Quiet things are nice. All blue, round things are nice. Nice, rough things are round. If something is furry then it is round. Furry, blue things are quiet. Nice things are furry. If something is round and blue then it is not rough.

No proof: neither the statement nor its negation follows.

right

Charlie is white.

Correct answer: false · Model said: false (68.8% sure)

The problem and its proof

Charlie is cold. Charlie is kind. Erin is not cold. Erin is kind. Erin is nice. Erin is smart. Erin is white. Gary is kind. Harry is cold. Harry is rough. If Charlie is rough and Charlie is white then Charlie is not smart. Cold, smart things are quiet. Rough things are smart. If something is cold then it is rough. Quiet, kind things are nice. If something is quiet and nice then it is not white.

The proof that the statement's negation follows:

  1. Charlie is cold.If something is cold then it is rough.Charlie is rough.
  2. Rough things are smart.Charlie is smart.
  3. + Charlie is cold.Cold, smart things are quiet.Charlie is quiet.
  4. + Charlie is kind.Quiet, kind things are nice.Charlie is nice.
  5. If something is quiet and nice then it is not white.Charlie is not white.
wrong

The rabbit is blue.

Correct answer: true · Model said: false (84.3% sure)

The problem and its proof

The bald eagle is round. The bald eagle likes the dog. The bald eagle likes the rabbit. The bald eagle sees the squirrel. The dog is cold. The dog likes the bald eagle. The dog likes the rabbit. The dog visits the bald eagle. The rabbit is green. The rabbit visits the squirrel. The squirrel is cold. The squirrel is rough. If something sees the bald eagle and the bald eagle likes the rabbit then the rabbit likes the dog. If the squirrel sees the bald eagle and the squirrel visits the rabbit then the rabbit visits the bald eagle. If the dog is blue then the dog sees the bald eagle. If something likes the dog then it is cold. If something is cold then it is blue.

  1. The dog is cold.If something is cold then it is blue.The dog is blue.
  2. If the dog is blue then the dog sees the bald eagle.The dog sees the bald eagle.
  3. + The bald eagle likes the rabbit.If something sees the bald eagle and the bald eagle likes the rabbit then the rabbit likes the dog.The rabbit likes the dog.
  4. If something likes the dog then it is cold.The rabbit is cold.
  5. If something is cold then it is blue.The rabbit is blue.
The closed-world task (true or false), depth by depth

Depth 0, closed-world task

72.0%
right on 216 of 300 problems wrong; chance 50% (dashed line)
right

The mouse chases the rabbit.

Correct answer: false · Model said: false (96.2% sure)

The problem and its proof

The cow is green. The mouse chases the tiger. The rabbit needs the cow. The tiger chases the rabbit. If something chases the mouse then the mouse is blue. If something is green and it does not visit the rabbit then it chases the mouse.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The bald eagle is nice.

Correct answer: false · Model said: false (94.0% sure)

The problem and its proof

The bald eagle is big. The bear eats the cow. The cow sees the mouse. The mouse likes the bald eagle. If something likes the bald eagle then the bald eagle sees the mouse.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Bob is not blue.

Correct answer: false · Model said: false (91.3% sure)

The problem and its proof

Bob is blue. Bob is green. Charlie is nice. Charlie is white. Charlie is young. Fiona is nice. Fiona is white. Gary is big. Gary is blue. Gary is smart. Gary is white. Green things are white. All blue, big things are smart. If Charlie is blue and Charlie is nice then Charlie is green. Young things are big. Green, young things are big. If Bob is green and Bob is nice then Bob is young. Young, smart things are nice. If Bob is white then Bob is nice.

Stated in the text: "Bob is blue."

wrong

Dave is rough.

Correct answer: false · Model said: true (89.3% sure)

paraphrased
The problem and its proof

The young person who is always feeling cold is named Dave. Thankfully Eric is nice. People have realized that he is big and rough. Since he is young he has tendency to become red when they mention it. Gary seems to be round. Harry seems to be round. If someone plays rough and is nice and round, they will be big. A big round young person is often blue. Nice people who feel blue often look green when ill. A nice person is inevitably round as well. If a person is both cold and round, that person is also someone who is red. If you know someone who is nice, green, and big, you know a young person. A person who is cold and red and blue is big.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 1, closed-world task

51.0%
right on 153 of 300 problems wrong; chance 50% (dashed line)
wrong

Dave is red.

Correct answer: true · Model said: false (55.2% sure)

paraphrased
The problem and its proof

Bob seems to be round. Dave may be young but he is nice, wears green shoes and is cold. Fred seems to be round. Young Gary has rough, green skin that is always cold. Gary is known for being a nice guy. If a person is described as being nice, cold and young it usually follow that they are red. When someone is kind yet can be cold and blue, they will also be very big. Even though young people are rough, they are still very red. It's pretty typical of red, rough, and cold people to be round. Most young kind people tend to be red too. Every single blue and red person who acts sort of rough tends to be green in places. People who are round and behave in a cold way are surely blue.

  1. Dave is nice. + Dave is cold. + Dave is young.If someone is nice and cold and young then they are red.Dave is red.
right

The bald eagle is not blue.

Correct answer: false · Model said: false (94.4% sure)

The problem and its proof

The bald eagle eats the dog. The dog eats the bald eagle. The dog is blue. The dog needs the tiger. The rabbit eats the bald eagle. The rabbit eats the dog. The rabbit is round. The rabbit is young. The rabbit needs the bald eagle. The rabbit needs the dog. The tiger eats the bald eagle. The tiger is blue. The tiger needs the bald eagle. The tiger needs the rabbit. The tiger visits the bald eagle. The tiger visits the rabbit. If something eats the rabbit then the rabbit visits the dog. If something needs the dog then the dog needs the rabbit. All blue things are round. If the bald eagle eats the tiger then the tiger eats the rabbit. If something needs the dog then the dog eats the tiger. If something is young then it eats the tiger. If the bald eagle is round then the bald eagle needs the rabbit. If something visits the rabbit and it visits the bald eagle then the bald eagle is blue. If something is round then it is young.

  1. The tiger visits the rabbit. + The tiger visits the bald eagle.If something visits the rabbit and it visits the bald eagle then the bald eagle is blue.The bald eagle is blue.
wrong

The squirrel is green.

Correct answer: false · Model said: true (81.2% sure)

The problem and its proof

The cat chases the squirrel. The cat is red. The cat is young. The cat sees the squirrel. The mouse chases the squirrel. The mouse is green. The mouse visits the cat. The mouse visits the squirrel. The squirrel chases the mouse. The squirrel visits the cat. If something is red then it sees the mouse. If something sees the squirrel and it visits the cat then the squirrel is green.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The mouse likes the bear.

Correct answer: false · Model said: false (92.4% sure)

The problem and its proof

The bald eagle is blue. The bear needs the mouse. The cat needs the bear. The mouse likes the cat. If something visits the bald eagle and the bald eagle visits the cat then the bald eagle needs the mouse. If something is blue then it likes the bear.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 2, closed-world task

54.7%
right on 164 of 300 problems wrong; chance 50% (dashed line)
wrong

Alan is not red.

Correct answer: true · Model said: false (96.4% sure)

paraphrased
The problem and its proof

That guy Alan sure is nice. Bob is round but he is also nice and kind. People that know Charlie will tell you he is rough and cold. But he can also be kind and red despite being blue because he is so young. That guy Harry sure is nice. People who are very nice and easily flush red are often wearing green concealer. If you're truly nice, cold, and blue, you're kind, too. Big, kind folks are green ones. Round people who are kind tend to be nice. Young and red people look round. A nice and green man or woman is also red in color.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Charlie is not white.

Correct answer: true · Model said: true (83.1% sure)

The problem and its proof

Charlie is nice. Dave is nice. Dave is rough. All rough people are white. Kind, smart people are white. All nice people are smart. If Charlie is quiet then Charlie is nice. All kind people are rough. If someone is nice then they are young.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Gary is young.

Correct answer: false · Model said: false (95.8% sure)

paraphrased
The problem and its proof

Alan is red, but he has turned blue as he is cold. He is nice but can be rough. Bob may be young but he is nice, wears green shoes and is cold. That guy Gary sure is nice. Big people who are red in color are almost always young. If a person happens to be big kind and red at the same time they are a young person. A person that is both nice and rough is someone who is also big. One who is young, red and also cold will definitely be round, too. Most young kind people tend to be red too. A nice person with cold skin is going to be rough.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

The bear likes the lion.

Correct answer: true · Model said: false (91.1% sure)

The problem and its proof

The bear eats the cat. The bear eats the lion. The bear is kind. The bear visits the cat. The cat eats the lion. The cat is cold. The lion visits the cat. If something eats the cat and the cat is cold then it is nice. If something is nice then it likes the lion.

  1. The bear eats the cat. + The cat is cold.If something eats the cat and the cat is cold then it is nice.The bear is nice.
  2. If something is nice then it likes the lion.The bear likes the lion.

Depth 3, closed-world task

51.3%
right on 154 of 300 problems wrong; chance 50% (dashed line)
right

Bob is not big.

Correct answer: false · Model said: false (88.0% sure)

paraphrased
The problem and its proof

Bob is young and green. Even though he is round he is cold outside. That guy Eric sure is nice. That person with the super big round body wearing red and green clothes who is acting quite nice towards you while being cold and feeling blue is named Harry. If someone plays rough and is nice and round, they will be big. Any person that's blue, young and green will turn out to be a nice person, too. Someone who is currently blue, big, and cold, is also kind. If you ever meet someone who is both green and blue while being nice, then they'll be big, too. People who are young are also blue.

  1. Bob is young.If someone is young then they are blue.Bob is blue.
  2. + Bob is young. + Bob is green.If someone is blue and young and green then they are nice.Bob is nice.
  3. + Bob is green.If someone is green and blue and nice then they are big.Bob is big.
right

The bear is not young.

Correct answer: false · Model said: false (92.9% sure)

The problem and its proof

The bear is nice. The cow is big. The cow is kind. The cow is young. The squirrel is rough. The squirrel is young. The squirrel likes the tiger. The squirrel needs the tiger. The squirrel sees the cow. The tiger is young. The tiger sees the squirrel. If something likes the bear then the bear is young. If something is nice and rough then it sees the bear. If something is young then it sees the bear. If the tiger is kind and the tiger is big then the tiger likes the bear. If something needs the tiger then the tiger likes the cow. If something likes the squirrel then it likes the bear. If something sees the bear then it likes the bear. If something is young and it needs the bear then the bear likes the squirrel. If something needs the bear then the bear is kind.

  1. The tiger is young.If something is young then it sees the bear.The tiger sees the bear.
  2. If something sees the bear then it likes the bear.The tiger likes the bear.
  3. If something likes the bear then the bear is young.The bear is young.
wrong

The bald eagle is rough.

Correct answer: true · Model said: false (96.6% sure)

The problem and its proof

The bald eagle is blue. If someone is green and blue then they are cold. If someone is nice then they are blue. If someone is cold then they are rough. All nice people are rough. If someone is cold then they are green. If someone is blue and not nice then they are green.

  1. The bald eagle is blue.If someone is blue and not nice then they are green.The bald eagle is green.
  2. + The bald eagle is blue.If someone is green and blue then they are cold.The bald eagle is cold.
  3. If someone is cold then they are rough.The bald eagle is rough.
wrong

The dog is not young.

Correct answer: true · Model said: false (90.2% sure)

The problem and its proof

The bald eagle chases the dog. The bald eagle is big. The bald eagle is kind. The bald eagle is nice. The bear eats the bald eagle. The bear eats the cow. The bear is cold. The cow chases the bald eagle. The cow chases the dog. The cow eats the bear. The dog chases the bear. If something chases the dog then it eats the dog. If something eats the dog then it is young. If something chases the bald eagle and the bald eagle chases the cow then it is young. If the bear eats the bald eagle then the bear chases the dog. If something is young then it needs the bear. If the bear chases the bald eagle and the bald eagle is kind then the bald eagle is cold. If something needs the dog and the dog needs the bear then it is kind. If something needs the bear then it chases the cow. If something eats the dog and the dog is not big then it chases the dog.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 4, closed-world task

58.7%
right on 176 of 300 problems wrong; chance 50% (dashed line)
wrong

The bear does not see the lion.

Correct answer: false · Model said: true (82.3% sure)

The problem and its proof

The bear chases the tiger. The bear is blue. The bear is cold. The lion chases the mouse. The lion chases the tiger. The lion is big. The lion is kind. The lion likes the mouse. The lion sees the tiger. The mouse likes the lion. The mouse sees the lion. The tiger chases the mouse. The tiger likes the mouse. The tiger sees the mouse. If the lion likes the tiger and the lion sees the mouse then the tiger chases the mouse. If someone chases the tiger and they see the lion then they chase the bear. If someone chases the bear then the bear is kind. If someone chases the lion then the lion is cold. If someone is kind then they see the lion.

  1. The lion is kind.If someone is kind then they see the lion.The lion sees the lion.
  2. + The lion chases the tiger.If someone chases the tiger and they see the lion then they chase the bear.The lion chases the bear.
  3. If someone chases the bear then the bear is kind.The bear is kind.
  4. If someone is kind then they see the lion.The bear sees the lion.
right

Charlie is white.

Correct answer: false · Model said: false (98.7% sure)

The problem and its proof

Charlie is green. Charlie is young. Dave is blue. Dave is smart. Dave is white. Harry is green. Harry is smart. All green, quiet things are cold. All blue, white things are cold. If something is cold and not blue then it is white. If something is smart and not blue then it is white. All white things are quiet. If Dave is green then Dave is quiet.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

The cow is not blue.

Correct answer: false · Model said: false (78.5% sure)

The problem and its proof

The cat is green. The cat is rough. The cat sees the cow. The cat visits the mouse. The cow is rough. The cow visits the cat. The mouse is young. The mouse needs the cat. The mouse needs the rabbit. The mouse sees the cat. The mouse sees the cow. The rabbit needs the cow. The rabbit sees the cat. The rabbit visits the mouse. If something sees the cat then it is blue. If something visits the rabbit then the rabbit sees the mouse. All cold things are young. If something sees the cow then it needs the cat. If something is young and blue then it needs the cat. If the mouse does not need the rabbit then the rabbit is rough. If something visits the cow then the cow sees the cat. If something is blue then it visits the cow. If something needs the cow then the cow needs the mouse.

  1. The mouse sees the cat.If something sees the cat then it is blue.The mouse is blue.
  2. If something is blue then it visits the cow.The mouse visits the cow.
  3. If something visits the cow then the cow sees the cat.The cow sees the cat.
  4. If something sees the cat then it is blue.The cow is blue.
wrong

Charlie is not red.

Correct answer: true · Model said: false (95.3% sure)

The problem and its proof

Charlie is nice. Erin is blue. Erin is red. Erin is white. Fiona is blue. Fiona is cold. Fiona is nice. If Charlie is furry and Charlie is not cold then Charlie is red. If someone is cold then they are furry. If Charlie is white then Charlie is furry. Furry people are red. All red, white people are young. Young, furry people are white. All young people are blue. If Charlie is white then Charlie is young.

No proof to show: under the closed world this answer holds because something cannot be derived.

Depth 5, closed-world task

46.0%
right on 138 of 300 problems wrong; chance 50% (dashed line)
right

Harry is furry.

Correct answer: false · Model said: false (72.4% sure)

The problem and its proof

Anne is blue. Anne is furry. Anne is kind. Anne is quiet. Anne is rough. Anne is round. Anne is young. Erin is blue. Erin is quiet. Fiona is kind. Fiona is rough. Fiona is round. Fiona is young. Harry is blue. Harry is round. If someone is rough and furry then they are kind. Furry people are rough. All quiet people are kind. If someone is kind and rough then they are round. If Harry is blue then Harry is young. If someone is quiet then they are furry. All rough people are quiet. All kind people are quiet.

No proof to show: under the closed world this answer holds because something cannot be derived.

wrong

Harry is not young.

Correct answer: true · Model said: false (50.1% sure)

The problem and its proof

Anne is kind. Anne is nice. Anne is round. Anne is smart. Erin is young. Gary is blue. Gary is cold. Gary is kind. Gary is nice. Gary is round. Gary is smart. Gary is young. Harry is blue. Harry is kind. All round, nice things are young. All cold, blue things are young. All cold, kind things are round. All round things are cold. If something is young then it is cold. All kind, round things are smart. If Erin is round and Erin is young then Erin is blue. All cold things are kind. Blue, cold things are nice.

No proof to show: under the closed world this answer holds because something cannot be derived.

right

Anne is not cold.

Correct answer: false · Model said: false (65.1% sure)

The problem and its proof

Anne is red. Charlie is blue. Charlie is nice. Charlie is red. Charlie is round. Charlie is white. Erin is cold. Harry is blue. Harry is nice. Harry is round. Harry is white. White, red things are blue. Red things are white. Blue things are round. If something is blue then it is nice. If something is green and white then it is cold. All nice things are green. If something is white and green then it is red. If Erin is white and Erin is nice then Erin is green.

  1. Anne is red.Red things are white.Anne is white.
  2. + Anne is red.White, red things are blue.Anne is blue.
  3. If something is blue then it is nice.Anne is nice.
  4. All nice things are green.Anne is green.
  5. If something is green and white then it is cold.Anne is cold.
wrong

The cow is round.

Correct answer: true · Model said: false (93.0% sure)

The problem and its proof

The bear chases the cow. The bear visits the cow. The bear visits the squirrel. The cow chases the squirrel. The cow chases the tiger. The cow is nice. The cow visits the squirrel. The squirrel chases the tiger. The squirrel eats the tiger. The squirrel visits the bear. The squirrel visits the tiger. The tiger visits the bear. If the cow chases the squirrel then the squirrel eats the bear. If something visits the squirrel and the squirrel eats the bear then it eats the tiger. If the bear visits the squirrel then the bear is round. If something eats the tiger and it visits the squirrel then it is nice. If something is nice and it eats the tiger then the tiger visits the cow. If something is kind then it is round. If the cow does not eat the tiger then the tiger visits the bear. If something visits the cow and it is not nice then the cow is kind.

  1. The cow chases the squirrel.If the cow chases the squirrel then the squirrel eats the bear.The squirrel eats the bear.
  2. + The cow visits the squirrel.If something visits the squirrel and the squirrel eats the bear then it eats the tiger.The cow eats the tiger.
  3. + The cow is nice.If something is nice and it eats the tiger then the tiger visits the cow.The tiger visits the cow.
  4. If something visits the cow and it is not nice then the cow is kind.The cow is kind.
  5. If something is kind then it is round.The cow is round.

Where each one is better

Open world, Jev vs Laya. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5, true answers and unknown answers. Laya is not clearly ahead on any depth or answer class. Too close to call: false answers.

Closed world, Jev vs Laya. Jev is clearly ahead on depth 0, depth 1, depth 2, depth 3, depth 4, depth 5 and true answers. Laya is not clearly ahead on any depth or answer class. Too close to call: false answers.

A slice counts as clearly ahead when its paired 95% interval excludes zero.

True, false and unknown, open world

By correct answer, Open world
Correct answerItemsJevLayaBest constant
false60588.3% 85.5–90.985.1% 82.1–87.9one answer
true60588.8% 86.1–91.140.2% 36.2–44.1one answer
unknown59074.2% 70.7–77.61.0% 0.3–1.9one answer

True, false and unknown, closed world

By correct answer, Closed world
Correct answerItemsJevLayaBest constant
false90088.9% 86.9–90.986.4% 84.1–88.6one answer
true90089.7% 87.7–91.624.8% 21.9–27.6one answer

Paraphrased vs templated rules, open world

By wording, Open world
WordingItemsJevLayaBest constant
templated1,29886.1% 84.2–87.941.7% 39.2–44.334.5%
paraphrased (NatLang)50278.1% 74.5–82.144.4% 40.2–48.835.9%

Paraphrased vs templated rules, closed world

By wording, Closed world
WordingItemsJevLayaBest constant
templated1,30694.0% 92.8–95.356.1% 53.4–58.950.2%
paraphrased (NatLang)49476.7% 73.1–80.454.3% 49.8–58.950.6%

Answer by answer: Jev and Laya on the same problems

Each matrix counts every problem by its correct answer (rows) and what the model said (columns), on exactly the same items.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Open world: true, false or unknown

Jev, open-world task
Correct ↓ · said →truefalseunknown
true537563
false1053461
unknown12428438
Laya, open-world task
Correct ↓ · said →truefalseunknown
true2433548
false895151
unknown1274576

Jev said true on 37% of problems; the correct answer is true on 34%. Laya said false on 74% of problems; the correct answer is false on 34%.

Closed world: true or false

Jev, closed-world task
Correct ↓ · said →truefalse
true80793
false100800
Laya, closed-world task
Correct ↓ · said →truefalse
true223677
false122778

Jev said true on 50% of problems; the correct answer is true on 50%. Laya said false on 81% of problems; the correct answer is false on 50%.

Where Laya errs

Accuracy alone hides which answers a model gets wrong. Rows are the correct answer; columns are what the model said.

right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.

Laya, open-world task

Laya, open-world task
Correct ↓ · said →truefalseunknownRecall
true243354840.2%
false89515185.1%
unknown12745761.0%

Laya answered false 73.7%, true 25.5%, unknown 0.8% of the time; the correct answers are false 33.6%, true 33.6%, unknown 32.8%.

Laya, closed-world task

Laya, closed-world task
Correct ↓ · said →truefalseRecall
true22367724.8%
false12277886.4%

Laya answered false 80.8%, true 19.2% of the time; the correct answers are false 50.0%, true 50.0%.

Laya at depth 0 and depth 5

Laya, open-world task, depth 0
Correct ↓ · said →truefalseunknown
true89101
false3970
unknown18802
Laya, open-world task, depth 5
Correct ↓ · said →truefalseunknown
true17840
false19820
unknown13740

Open-world task. At depth 0, Laya was right on 63% of problems (the green diagonal); at depth 5, on 34%. The answer whose share moved most is false: 62% of its answers at depth 0, 83% at depth 5, against 33% and 35% of the correct answers.

The closed-world task at depth 0 and depth 5
Laya, closed-world task, depth 0
Correct ↓ · said →truefalse
true7773
false11139
Laya, closed-world task, depth 5
Correct ↓ · said →truefalse
true13137
false25125

Closed-world task. At depth 0, Laya was right on 72% of problems (the green diagonal); at depth 5, on 46%. The answer whose share moved most is true: 29% of its answers at depth 0, 13% at depth 5, against 50% and 50% of the correct answers.

Laya across every breakdown

For each property of the problems, the weakest and strongest value on the open-world task (slices of at least 50 items). Only depth was controlled; the rest are descriptive.

BreakdownWeakest · strongest
Proof depthdepth 5 34.3% · depth 0 62.7% (28.4 points)
Gold answerunknown 1.0% · false 85.1% (84.1 points)
Theory kindattribute 41.2% · relation 44.3% (3.1 points)
Negation in the theorywithout negation 38.8% · with negation 46.2% (7.4 points)
Negated statementplain statement 38.7% · negated statement 46.2% (7.5 points)
Question strategyinv-rconc 0.8% · inv-proof 85.1% (84.3 points)
Paraphrased rulestemplated 41.7% · paraphrased (NatLang) 44.4% (2.7 points)
Theory's deepest proof0 39.4% · 1 48.3% (8.8 points)
Theory length80-109 40.3% · 0-49 47.8% (7.5 points)
Number of rules8+ 35.7% · 0-2 53.9% (18.2 points)
Number of facts4-7 41.6% · 13+ 44.0% (2.4 points)
Proof size0-1 24.3% · 2 63.6% (39.2 points)

Exactly what Laya was asked

One open-world problem at depth 0, as each model received it. The correct answer is unknown.

The bear is green. Green things are not young.

Statement: The bear is not big.
Jevright answered unknown (100.0%); correct answer unknown

POST https://api.typesafe.ai/v1/systemone

Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%.

The full request and response, exactly as sent and recorded

Request

{
  "model": "jev-latest",
  "state": {
    "text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
  },
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "jev-1.13.0",
  "answers": {
    "Decision": {
      "choice": "unknown",
      "confidence": 1,
      "probabilities": {
        "false": 0,
        "true": 0,
        "unknown": 1
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 460,
    "output_tokens": 38
  }
}
Layawrong answered false (50.2%); correct answer unknown

Run on our laptop in Python: laya.load().system_one(state, questions)

The laya package on our laptop; it takes the same questions as Jev, with the problem as the state.

The text the model reads

The bear is green. Green things are not young.

Statement: The bear is not big.

The question, sent as its own field

Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.

The answers it can give, each with its meaning

true: The statement follows from the facts and rules.
false: The negation of the statement follows from the facts and rules.
unknown: Neither the statement nor its negation follows from the facts and rules.

Its probability for each answer: false 50.2%, true 6.0%, unknown 43.7%.

The full request and response, exactly as sent and recorded

Request

{
  "call": "laya.load().system_one(state, questions)",
  "state": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.",
  "questions": {
    "Decision": {
      "type": "choice",
      "instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
      "criteria": {
        "true": "The statement follows from the facts and rules.",
        "false": "The negation of the statement follows from the facts and rules.",
        "unknown": "Neither the statement nor its negation follows from the facts and rules."
      }
    }
  }
}

Response (the saved record)

{
  "model": "laya-upstream:0.3.21",
  "answers": {
    "Decision": {
      "action": {
        "act_probability": 1
      },
      "answer_confidence": 0.5021,
      "choice": "false",
      "confidence": 0.2014,
      "probabilities": {
        "false": 0.5021,
        "true": 0.0605,
        "unknown": 0.4374
      },
      "type": "choice"
    }
  },
  "usage": {
    "input_tokens": 162,
    "output_tokens": 0
  }
}

Consistency and calibration

Asked every open world question a second time, Laya gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 42.4% the first time and 42.4% the second.

Asked every closed world question a second time, Laya gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 55.6% the first time and 55.6% the second.

Calibration on the open world task: Laya's probability for its own answer averaged 74.7% against 42.4% accuracy; expected calibration error 32.3 points.

Calibration on the closed world task: Laya's probability for its own answer averaged 87.3% against 55.6% accuracy; expected calibration error 31.7 points.

Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.61 on the open world task against Jev's 0.85; 0.56 on the closed world task against Jev's 0.86.

Size, hardware and speed

ModelKindParametersRan onMedian time per decisionMemory peak
Jevhosted decision modelundisclosedvendor's servers189 msnot applicable (hosted)
Layaopen decision model421Mour laptop (Apple M1 Max, 32 GB)140 ms10.0 GB

Times are open-world, one request at a time. Hosted models were timed over the network against the vendor's servers and open models on one laptop, so speed is not like for like across the two. Speed, size and memory for every model.

Laya

Parameters
421M
Architecture
ModernBERT-large encoder with decision heads, trained with RLCD
Weights
0.84 GB (model.safetensors, convaiinnovations/laya repo root)
Precision
as loaded by the laya package (PyTorch)
Where it ran
this machine, laya 0.3.21 package, PyTorch on MPS
Maker's stated hardware
not stated by the maker
Memory measured
10.0 GB in use, 10.0 GB peak
Time per decision, open world
median 140 ms, 90th percentile 479 ms, on a laptop
Time per decision, closed world
median 121 ms, 90th percentile 398 ms, on a laptop
Source
huggingface.co/convaiinnovations/laya (model card checkpoint table)

The models

Jev Jev is a hosted decision model made by TypeSafe. We call it through its maker's software kit, typesafe-sdk, on paid API access; the version is recorded on every answer (jev-1.13.0). Its size and architecture are not published.

Laya Laya is an open decision model made by Convai Innovations: a 421M-parameter ModernBERT-large encoder with decision heads. We ran it on our own laptop with the laya package, PyTorch on Apple's GPU.

Version v0.1.0, unreleased.

Every number on this site is worked out from the models' saved answers, on 2026-11-02, from the harness's working tree at commit b4e9183 on main; results scored since that commit are included. You can download every number in one data file. About this site.

Hard-Decisions is MIT-licensed and made by Anthus AI Solutions. ProofWriter (Tafjord, Dalvi and Clark, 2021) is published by AI2 under CC BY 4.0. Typefaces: Jersey 25 and Montserrat, SIL Open Font License.