Requests and responses · every model
Exactly what each model was asked, and what it answered
The request each model received for the same problems, field for field, and the answer it sent back, taken from the saved records. Every model got the same problem, the same question and the same meaning for each answer; where a model's interface is shaped differently, this shows exactly where each piece went.
The problems shown are fixed by a rule, not chosen for their answers: for each task, the shortest problem at depth 0 and the shortest at depth 3 written in ProofWriter's templated wording. Each request is reproduced by the same code the benchmark used to send it, with keys and authentication left out; each response is the saved record.
Open world: true, false or unknown, depth 0
The problem, as every model saw it. The correct answer is unknown.
The bear is green. Green things are not young. Statement: The bear is not big.
Jev answered unknown (100.0%); correct answer unknown
POST https://api.typesafe.ai/v1/systemone
Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 0.0%, true 0.0%, unknown 100.0%.
The full request and response, exactly as sent and recorded
Request
{
"model": "jev-latest",
"state": {
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "jev-1.13.0",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 1,
"probabilities": {
"false": 0,
"true": 0,
"unknown": 1
},
"type": "choice"
}
},
"usage": {
"input_tokens": 460,
"output_tokens": 38
}
}GLiDE, Oct 3 rerun answered unknown (98.3%); correct answer unknown
POST https://api.fastino.ai/v1/systemone
The same request as GLiDE's first run, sent again on October 3 at Fastino's request.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 0.7%, true 1.0%, unknown 98.3%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.9724408909000181,
"probabilities": {
"false": 0.007048396451747089,
"true": 0.010255356324117408,
"unknown": 0.9826962472241355
},
"type": "choice"
}
},
"usage": {
"input_tokens": 203,
"output_tokens": 1
}
}GLiDE, Oct 2 answered unknown (98.3%); correct answer unknown
POST https://api.fastino.ai/v1/systemone
Fastino serves GLiDE on the same System One format as Jev, so this is Jev's request with a different address and model name.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 0.7%, true 1.0%, unknown 98.3%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.9724408909000181,
"probabilities": {
"false": 0.007048396451747089,
"true": 0.010255356324117408,
"unknown": 0.9826962472241355
},
"type": "choice"
}
},
"usage": {
"input_tokens": 203,
"output_tokens": 1
}
}GPT-6 Luna answered unknown; correct answer unknown
POST https://api.openai.com/v1/chat/completions
GPT-6 Luna with reasoning off (reasoning_effort none, its lowest setting), as a direct classifier. One chat message carries the problem, the question and every option with its meaning; the strict JSON schema only allows one of the options as the answer.
The text the model reads
The bear is green. Green things are not young.
Statement: The bear is not big.
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
Options:
- true: The statement follows from the facts and rules.
- false: The negation of the statement follows from the facts and rules.
- unknown: Neither the statement nor its negation follows from the facts and rules.
Reply with a JSON object {"answer": <option>} where <option> is exactly one of: true, false, unknown.The answers it can give
true false unknown
The full request and response, exactly as sent and recorded
Request
{
"model": "gpt-6-luna",
"max_completion_tokens": 1024,
"reasoning_effort": "none",
"messages": [
{
"role": "user",
"content": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.\n\nReply with a JSON object {\"answer\": <option>} where <option> is exactly one of: true, false, unknown."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "decision",
"strict": true,
"schema": {
"type": "object",
"properties": {
"answer": {
"type": "string",
"enum": [
"true",
"false",
"unknown"
]
}
},
"required": [
"answer"
],
"additionalProperties": false
}
}
}
}Response (the saved record)
{
"model": "gpt-6-luna",
"answers": {
"Decision": {
"choice": "unknown",
"raw": null,
"type": "choice"
}
},
"usage": {
"input_tokens": 221,
"output_tokens": 11,
"stop_reason": "stop"
}
}Kev-9B answered unknown (98.7%); correct answer unknown
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-9B on our laptop, answering Jev's System One format.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 0.9%, true 0.4%, unknown 98.7%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.9803,
"probabilities": {
"false": 0.0093,
"true": 0.0038,
"unknown": 0.9869
},
"type": "choice"
}
},
"usage": {
"input_tokens": 162,
"output_tokens": 63
}
}Kev-4B answered unknown (99.4%); correct answer unknown
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-4B on our laptop, answering Jev's System One format.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 0.3%, true 0.3%, unknown 99.4%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.9909,
"probabilities": {
"false": 0.0034,
"true": 0.0027,
"unknown": 0.9939
},
"type": "choice"
}
},
"usage": {
"input_tokens": 162,
"output_tokens": 63
}
}Kev-0.8B answered unknown (42.8%); correct answer unknown
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-0.8B on our laptop, answering Jev's System One format.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 40.7%, true 16.5%, unknown 42.8%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.1413,
"probabilities": {
"false": 0.4072,
"true": 0.1653,
"unknown": 0.4275
},
"type": "choice"
}
},
"usage": {
"input_tokens": 162,
"output_tokens": 63
}
}GLiNER2.5-Decide answered unknown (92.1%); correct answer unknown
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
Fastino's hosted API for this model accepts a text and a list of answer labels, with no field for a question or for what each label means (the open-source package also accepts a short question as a prompt and a description per label; passed that way, our instructions made the model give the same answer to every problem). So the question and every option's meaning go into the text, after the problem, word for word as GPT-6 Luna received them, without Luna's line about replying in JSON. The labels are the three answers.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big. Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived. Options: - true: The statement follows from the facts and rules. - false: The negation of the statement follows from the facts and rules. - unknown: Neither the statement nor its negation follows from the facts and rules.
The answers it can give
true false unknown
Its probability for each answer: false 3.7%, true 4.2%, unknown 92.1%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.",
"tasks": {
"Decision": [
"true",
"false",
"unknown"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.9209111332893372,
"probabilities": {
"false": 0.036921169608831406,
"true": 0.04216771200299263,
"unknown": 0.9209111332893372
},
"type": "choice"
}
},
"usage": null
}GLiNER2.5-Decide, without instructions answered false (49.2%); correct answer unknown
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
The same model, asked without the instructions: the text is only the problem (the facts and rules, then the statement), and the labels are the three answers. Nothing tells the model what true, false or unknown mean. This is the bare form Fastino's hosted API documents.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The answers it can give
true false unknown
Its probability for each answer: false 49.2%, true 31.3%, unknown 19.5%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.",
"tasks": {
"Decision": [
"true",
"false",
"unknown"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.4924229383468628,
"probabilities": {
"false": 0.4924229383468628,
"true": 0.3129580020904541,
"unknown": 0.1946191042661667
},
"type": "choice"
}
},
"usage": null
}Laya answered false (50.2%); correct answer unknown
Run on our laptop in Python: laya.load().system_one(state, questions)
The laya package on our laptop; it takes the same questions as Jev, with the problem as the state.
The text the model reads
The bear is green. Green things are not young. Statement: The bear is not big.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 50.2%, true 6.0%, unknown 43.7%.
The full request and response, exactly as sent and recorded
Request
{
"call": "laya.load().system_one(state, questions)",
"state": "The bear is green. Green things are not young.\n\nStatement: The bear is not big.",
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "laya-upstream:0.3.21",
"answers": {
"Decision": {
"action": {
"act_probability": 1
},
"answer_confidence": 0.5021,
"choice": "false",
"confidence": 0.2014,
"probabilities": {
"false": 0.5021,
"true": 0.0605,
"unknown": 0.4374
},
"type": "choice"
}
},
"usage": {
"input_tokens": 162,
"output_tokens": 0
}
}Open world: true, false or unknown, depth 3
The problem, as every model saw it. The correct answer is true.
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
Jev answered true (100.0%); correct answer true
POST https://api.typesafe.ai/v1/systemone
Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 0.0%, true 100.0%, unknown 0.0%.
The full request and response, exactly as sent and recorded
Request
{
"model": "jev-latest",
"state": {
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "jev-1.13.0",
"answers": {
"Decision": {
"choice": "true",
"confidence": 1,
"probabilities": {
"false": 0,
"true": 1,
"unknown": 0
},
"type": "choice"
}
},
"usage": {
"input_tokens": 473,
"output_tokens": 38
}
}GLiDE, Oct 3 rerun answered true (69.9%); correct answer true
POST https://api.fastino.ai/v1/systemone
The same request as GLiDE's first run, sent again on October 3 at Fastino's request.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 10.1%, true 69.9%, unknown 20.0%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.4987494352608489,
"probabilities": {
"false": 0.10070379431627627,
"true": 0.6990228204722864,
"unknown": 0.20027338521143745
},
"type": "choice"
}
},
"usage": {
"input_tokens": 216,
"output_tokens": 1
}
}GLiDE, Oct 2 answered true (85.0%); correct answer true
POST https://api.fastino.ai/v1/systemone
Fastino serves GLiDE on the same System One format as Jev, so this is Jev's request with a different address and model name.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 5.0%, true 85.0%, unknown 10.0%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.7493747169675863,
"probabilities": {
"false": 0.05035189736107074,
"true": 0.8495114098032578,
"unknown": 0.10013669283567149
},
"type": "choice"
}
},
"usage": {
"input_tokens": 3144,
"output_tokens": 1143
}
}GPT-6 Luna answered unknown; correct answer true
POST https://api.openai.com/v1/chat/completions
GPT-6 Luna with reasoning off (reasoning_effort none, its lowest setting), as a direct classifier. One chat message carries the problem, the question and every option with its meaning; the strict JSON schema only allows one of the options as the answer.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.
Statement: The squirrel is red.
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
Options:
- true: The statement follows from the facts and rules.
- false: The negation of the statement follows from the facts and rules.
- unknown: Neither the statement nor its negation follows from the facts and rules.
Reply with a JSON object {"answer": <option>} where <option> is exactly one of: true, false, unknown.The answers it can give
true false unknown
The full request and response, exactly as sent and recorded
Request
{
"model": "gpt-6-luna",
"max_completion_tokens": 1024,
"reasoning_effort": "none",
"messages": [
{
"role": "user",
"content": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.\n\nReply with a JSON object {\"answer\": <option>} where <option> is exactly one of: true, false, unknown."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "decision",
"strict": true,
"schema": {
"type": "object",
"properties": {
"answer": {
"type": "string",
"enum": [
"true",
"false",
"unknown"
]
}
},
"required": [
"answer"
],
"additionalProperties": false
}
}
}
}Response (the saved record)
{
"model": "gpt-6-luna",
"answers": {
"Decision": {
"choice": "unknown",
"raw": null,
"type": "choice"
}
},
"usage": {
"input_tokens": 234,
"output_tokens": 11,
"stop_reason": "stop"
}
}Kev-9B answered true (93.0%); correct answer true
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-9B on our laptop, answering Jev's System One format.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 3.9%, true 93.0%, unknown 3.1%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.8945,
"probabilities": {
"false": 0.039,
"true": 0.9297,
"unknown": 0.0314
},
"type": "choice"
}
},
"usage": {
"input_tokens": 175,
"output_tokens": 62
}
}Kev-4B answered unknown (46.9%); correct answer true
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-4B on our laptop, answering Jev's System One format.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 43.5%, true 9.6%, unknown 46.9%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.204,
"probabilities": {
"false": 0.4349,
"true": 0.0958,
"unknown": 0.4693
},
"type": "choice"
}
},
"usage": {
"input_tokens": 175,
"output_tokens": 62
}
}Kev-0.8B answered false (70.1%); correct answer true
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-0.8B on our laptop, answering Jev's System One format.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 70.1%, true 6.7%, unknown 23.2%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.5516,
"probabilities": {
"false": 0.7011,
"true": 0.0665,
"unknown": 0.2324
},
"type": "choice"
}
},
"usage": {
"input_tokens": 175,
"output_tokens": 63
}
}GLiNER2.5-Decide answered unknown (94.2%); correct answer true
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
Fastino's hosted API for this model accepts a text and a list of answer labels, with no field for a question or for what each label means (the open-source package also accepts a short question as a prompt and a description per label; passed that way, our instructions made the model give the same answer to every problem). So the question and every option's meaning go into the text, after the problem, word for word as GPT-6 Luna received them, without Luna's line about replying in JSON. The labels are the three answers.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red. Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived. Options: - true: The statement follows from the facts and rules. - false: The negation of the statement follows from the facts and rules. - unknown: Neither the statement nor its negation follows from the facts and rules.
The answers it can give
true false unknown
Its probability for each answer: false 2.7%, true 3.2%, unknown 94.2%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red.\n\nUsing only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.\n\nOptions:\n- true: The statement follows from the facts and rules.\n- false: The negation of the statement follows from the facts and rules.\n- unknown: Neither the statement nor its negation follows from the facts and rules.",
"tasks": {
"Decision": [
"true",
"false",
"unknown"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "unknown",
"confidence": 0.9415450692176819,
"probabilities": {
"false": 0.026757700368762016,
"true": 0.03169730305671692,
"unknown": 0.9415450692176819
},
"type": "choice"
}
},
"usage": null
}GLiNER2.5-Decide, without instructions answered false (60.3%); correct answer true
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
The same model, asked without the instructions: the text is only the problem (the facts and rules, then the statement), and the labels are the three answers. Nothing tells the model what true, false or unknown mean. This is the bare form Fastino's hosted API documents.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The answers it can give
true false unknown
Its probability for each answer: false 60.3%, true 23.4%, unknown 16.4%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red.",
"tasks": {
"Decision": [
"true",
"false",
"unknown"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.6029250025749207,
"probabilities": {
"false": 0.6029249429702759,
"true": 0.23353159427642822,
"unknown": 0.1635434329509735
},
"type": "choice"
}
},
"usage": null
}Laya answered false (96.1%); correct answer true
Run on our laptop in Python: laya.load().system_one(state, questions)
The laya package on our laptop; it takes the same questions as Jev, with the problem as the state.
The text the model reads
The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green. Statement: The squirrel is red.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.
The answers it can give, each with its meaning
true: The statement follows from the facts and rules. false: The negation of the statement follows from the facts and rules. unknown: Neither the statement nor its negation follows from the facts and rules.
Its probability for each answer: false 96.1%, true 1.9%, unknown 1.9%.
The full request and response, exactly as sent and recorded
Request
{
"call": "laya.load().system_one(state, questions)",
"state": "The squirrel is kind. All kind people are big. Green, big people are red. Big, kind people are green.\n\nStatement: The squirrel is red.",
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true, false, or unknown? A statement is true if it can be derived from the facts and rules, and false if its negation can be derived. If neither the statement nor its negation can be derived, it is unknown. A rule applies only when all of its conditions are established; 'not' in a condition requires the negation to be stated or derived.",
"criteria": {
"true": "The statement follows from the facts and rules.",
"false": "The negation of the statement follows from the facts and rules.",
"unknown": "Neither the statement nor its negation follows from the facts and rules."
}
}
}
}Response (the saved record)
{
"model": "laya-upstream:0.3.21",
"answers": {
"Decision": {
"action": {
"act_probability": 1
},
"answer_confidence": 0.9614,
"choice": "false",
"confidence": 0.827,
"probabilities": {
"false": 0.9614,
"true": 0.0195,
"unknown": 0.019
},
"type": "choice"
}
},
"usage": {
"input_tokens": 177,
"output_tokens": 0
}
}Closed world: true or false, depth 0
The problem, as every model saw it. The correct answer is false.
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
Jev answered false (100.0%); correct answer false
POST https://api.typesafe.ai/v1/systemone
Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 100.0%, true 0.0%.
The full request and response, exactly as sent and recorded
Request
{
"model": "jev-latest",
"state": {
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "jev-1.13.0",
"answers": {
"Decision": {
"choice": "false",
"confidence": 1,
"probabilities": {
"false": 1,
"true": 0
},
"type": "choice"
}
},
"usage": {
"input_tokens": 436,
"output_tokens": 31
}
}GLiDE, Oct 3 rerun answered false (99.2%); correct answer false
POST https://api.fastino.ai/v1/systemone
The same request as GLiDE's first run, sent again on October 3 at Fastino's request.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 99.2%, true 0.8%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.9838760180308329,
"probabilities": {
"false": 0.9919380090154164,
"true": 0.008061990984583519
},
"type": "choice"
}
},
"usage": {
"input_tokens": 185,
"output_tokens": 1
}
}GLiDE, Oct 2 answered false (99.2%); correct answer false
POST https://api.fastino.ai/v1/systemone
Fastino serves GLiDE on the same System One format as Jev, so this is Jev's request with a different address and model name.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 99.2%, true 0.8%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.9838760180308329,
"probabilities": {
"false": 0.9919380090154164,
"true": 0.008061990984583519
},
"type": "choice"
}
},
"usage": {
"input_tokens": 185,
"output_tokens": 1
}
}GPT-6 Luna answered false; correct answer false
POST https://api.openai.com/v1/chat/completions
GPT-6 Luna with reasoning off (reasoning_effort none, its lowest setting), as a direct classifier. One chat message carries the problem, the question and every option with its meaning; the strict JSON schema only allows one of the options as the answer.
The text the model reads
The mouse is kind. If something is kind then it is rough.
Statement: The mouse is young.
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
Options:
- true: The statement holds under the closed-world assumption.
- false: The statement does not hold under the closed-world assumption.
Reply with a JSON object {"answer": <option>} where <option> is exactly one of: true, false.The answers it can give
true false
The full request and response, exactly as sent and recorded
Request
{
"model": "gpt-6-luna",
"max_completion_tokens": 1024,
"reasoning_effort": "none",
"messages": [
{
"role": "user",
"content": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young.\n\nUsing only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.\n\nOptions:\n- true: The statement holds under the closed-world assumption.\n- false: The statement does not hold under the closed-world assumption.\n\nReply with a JSON object {\"answer\": <option>} where <option> is exactly one of: true, false."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "decision",
"strict": true,
"schema": {
"type": "object",
"properties": {
"answer": {
"type": "string",
"enum": [
"true",
"false"
]
}
},
"required": [
"answer"
],
"additionalProperties": false
}
}
}
}Response (the saved record)
{
"model": "gpt-6-luna",
"answers": {
"Decision": {
"choice": "false",
"raw": null,
"type": "choice"
}
},
"usage": {
"input_tokens": 201,
"output_tokens": 11,
"stop_reason": "stop"
}
}Kev-9B answered false (96.3%); correct answer false
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-9B on our laptop, answering Jev's System One format.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 96.3%, true 3.7%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.9256,
"probabilities": {
"false": 0.9628,
"true": 0.0372
},
"type": "choice"
}
},
"usage": {
"input_tokens": 145,
"output_tokens": 52
}
}Kev-4B answered false (95.1%); correct answer false
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-4B on our laptop, answering Jev's System One format.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 95.1%, true 4.9%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.9014,
"probabilities": {
"false": 0.9507,
"true": 0.0493
},
"type": "choice"
}
},
"usage": {
"input_tokens": 145,
"output_tokens": 52
}
}Kev-0.8B answered false (67.1%); correct answer false
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-0.8B on our laptop, answering Jev's System One format.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 67.1%, true 32.9%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.3424,
"probabilities": {
"false": 0.6712,
"true": 0.3288
},
"type": "choice"
}
},
"usage": {
"input_tokens": 145,
"output_tokens": 52
}
}GLiNER2.5-Decide answered false (54.2%); correct answer false
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
Fastino's hosted API for this model accepts a text and a list of answer labels, with no field for a question or for what each label means (the open-source package also accepts a short question as a prompt and a description per label; passed that way, our instructions made the model give the same answer to every problem). So the question and every option's meaning go into the text, after the problem, word for word as GPT-6 Luna received them, without Luna's line about replying in JSON. The labels are the three answers.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young. Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived. Options: - true: The statement holds under the closed-world assumption. - false: The statement does not hold under the closed-world assumption.
The answers it can give
true false
Its probability for each answer: false 54.2%, true 45.8%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young.\n\nUsing only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.\n\nOptions:\n- true: The statement holds under the closed-world assumption.\n- false: The statement does not hold under the closed-world assumption.",
"tasks": {
"Decision": [
"true",
"false"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.5422529578208923,
"probabilities": {
"false": 0.5422529578208923,
"true": 0.45774707198143005
},
"type": "choice"
}
},
"usage": null
}GLiNER2.5-Decide, without instructions answered true (54.4%); correct answer false
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
The same model, asked without the instructions: the text is only the problem (the facts and rules, then the statement), and the labels are the three answers. Nothing tells the model what true, false or unknown mean. This is the bare form Fastino's hosted API documents.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The answers it can give
true false
Its probability for each answer: false 45.6%, true 54.4%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young.",
"tasks": {
"Decision": [
"true",
"false"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.5443795323371887,
"probabilities": {
"false": 0.4556204080581665,
"true": 0.5443795323371887
},
"type": "choice"
}
},
"usage": null
}Laya answered false (90.5%); correct answer false
Run on our laptop in Python: laya.load().system_one(state, questions)
The laya package on our laptop; it takes the same questions as Jev, with the problem as the state.
The text the model reads
The mouse is kind. If something is kind then it is rough. Statement: The mouse is young.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 90.5%, true 9.4%.
The full request and response, exactly as sent and recorded
Request
{
"call": "laya.load().system_one(state, questions)",
"state": "The mouse is kind. If something is kind then it is rough.\n\nStatement: The mouse is young.",
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "laya-upstream:0.3.21",
"answers": {
"Decision": {
"action": {
"act_probability": 1
},
"answer_confidence": 0.9055,
"choice": "false",
"confidence": 0.5486,
"probabilities": {
"false": 0.9055,
"true": 0.0945
},
"type": "choice"
}
},
"usage": {
"input_tokens": 148,
"output_tokens": 0
}
}Closed world: true or false, depth 3
The problem, as every model saw it. The correct answer is true.
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
Jev answered true (100.0%); correct answer true
POST https://api.typesafe.ai/v1/systemone
Through TypeSafe's typesafe-sdk; jev-latest is the SDK's default model name.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 0.0%, true 100.0%.
The full request and response, exactly as sent and recorded
Request
{
"model": "jev-latest",
"state": {
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "jev-1.13.0",
"answers": {
"Decision": {
"choice": "true",
"confidence": 1,
"probabilities": {
"false": 0,
"true": 1
},
"type": "choice"
}
},
"usage": {
"input_tokens": 447,
"output_tokens": 31
}
}GLiDE, Oct 3 rerun answered true (75.5%); correct answer true
POST https://api.fastino.ai/v1/systemone
The same request as GLiDE's first run, sent again on October 3 at Fastino's request.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 24.5%, true 75.5%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.5098299737352565,
"probabilities": {
"false": 0.24508501313237172,
"true": 0.7549149868676283
},
"type": "choice"
}
},
"usage": {
"input_tokens": 196,
"output_tokens": 1
}
}GLiDE, Oct 2 answered true (75.5%); correct answer true
POST https://api.fastino.ai/v1/systemone
Fastino serves GLiDE on the same System One format as Jev, so this is Jev's request with a different address and model name.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 24.5%, true 75.5%.
The full request and response, exactly as sent and recorded
Request
{
"model": "fastino/GLiDE",
"state": {
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "glide",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.5098299737352565,
"probabilities": {
"false": 0.24508501313237172,
"true": 0.7549149868676283
},
"type": "choice"
}
},
"usage": {
"input_tokens": 196,
"output_tokens": 1
}
}GPT-6 Luna answered false; correct answer true
POST https://api.openai.com/v1/chat/completions
GPT-6 Luna with reasoning off (reasoning_effort none, its lowest setting), as a direct classifier. One chat message carries the problem, the question and every option with its meaning; the strict JSON schema only allows one of the options as the answer.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.
Statement: The bald eagle is green.
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
Options:
- true: The statement holds under the closed-world assumption.
- false: The statement does not hold under the closed-world assumption.
Reply with a JSON object {"answer": <option>} where <option> is exactly one of: true, false.The answers it can give
true false
The full request and response, exactly as sent and recorded
Request
{
"model": "gpt-6-luna",
"max_completion_tokens": 1024,
"reasoning_effort": "none",
"messages": [
{
"role": "user",
"content": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green.\n\nUsing only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.\n\nOptions:\n- true: The statement holds under the closed-world assumption.\n- false: The statement does not hold under the closed-world assumption.\n\nReply with a JSON object {\"answer\": <option>} where <option> is exactly one of: true, false."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "decision",
"strict": true,
"schema": {
"type": "object",
"properties": {
"answer": {
"type": "string",
"enum": [
"true",
"false"
]
}
},
"required": [
"answer"
],
"additionalProperties": false
}
}
}
}Response (the saved record)
{
"model": "gpt-6-luna",
"answers": {
"Decision": {
"choice": "false",
"raw": null,
"type": "choice"
}
},
"usage": {
"input_tokens": 212,
"output_tokens": 11,
"stop_reason": "stop"
}
}Kev-9B answered true (88.5%); correct answer true
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-9B on our laptop, answering Jev's System One format.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 11.5%, true 88.5%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.7697,
"probabilities": {
"false": 0.1152,
"true": 0.8848
},
"type": "choice"
}
},
"usage": {
"input_tokens": 156,
"output_tokens": 52
}
}Kev-4B answered false (83.6%); correct answer true
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-4B on our laptop, answering Jev's System One format.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 83.6%, true 16.4%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.6713,
"probabilities": {
"false": 0.8357,
"true": 0.1643
},
"type": "choice"
}
},
"usage": {
"input_tokens": 156,
"output_tokens": 52
}
}Kev-0.8B answered false (93.4%); correct answer true
POST http://127.0.0.1:8009/v1/systemone
Kev's own server running Kev-0.8B on our laptop, answering Jev's System One format.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 93.4%, true 6.6%.
The full request and response, exactly as sent and recorded
Request
{
"model": "kev-latest",
"state": {
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green."
},
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "kev-latest",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.8682,
"probabilities": {
"false": 0.9341,
"true": 0.0659
},
"type": "choice"
}
},
"usage": {
"input_tokens": 156,
"output_tokens": 52
}
}GLiNER2.5-Decide answered false (56.9%); correct answer true
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
Fastino's hosted API for this model accepts a text and a list of answer labels, with no field for a question or for what each label means (the open-source package also accepts a short question as a prompt and a description per label; passed that way, our instructions made the model give the same answer to every problem). So the question and every option's meaning go into the text, after the problem, word for word as GPT-6 Luna received them, without Luna's line about replying in JSON. The labels are the three answers.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green. Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived. Options: - true: The statement holds under the closed-world assumption. - false: The statement does not hold under the closed-world assumption.
The answers it can give
true false
Its probability for each answer: false 56.9%, true 43.1%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green.\n\nUsing only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.\n\nOptions:\n- true: The statement holds under the closed-world assumption.\n- false: The statement does not hold under the closed-world assumption.",
"tasks": {
"Decision": [
"true",
"false"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "false",
"confidence": 0.569098174571991,
"probabilities": {
"false": 0.569098174571991,
"true": 0.43090182542800903
},
"type": "choice"
}
},
"usage": null
}GLiNER2.5-Decide, without instructions answered true (77.6%); correct answer true
Run on our laptop in Python: AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide").classify_text(text, tasks, include_confidence=True)
The same model, asked without the instructions: the text is only the problem (the facts and rules, then the statement), and the labels are the three answers. Nothing tells the model what true, false or unknown mean. This is the bare form Fastino's hosted API documents.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The answers it can give
true false
Its probability for each answer: false 22.4%, true 77.6%.
The full request and response, exactly as sent and recorded
Request
{
"call": "AutoExtractor.from_pretrained(\"fastino/GLiNER2.5-Decide\").classify_text(text, tasks, include_confidence=True)",
"text": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green.",
"tasks": {
"Decision": [
"true",
"false"
]
}
}Response (the saved record)
{
"model": "fastino/GLiNER2.5-Decide@5a7adf7 gliner2-2.0.0 mps",
"answers": {
"Decision": {
"choice": "true",
"confidence": 0.7761543393135071,
"probabilities": {
"false": 0.22384561598300934,
"true": 0.7761543393135071
},
"type": "choice"
}
},
"usage": null
}Laya answered false (87.8%); correct answer true
Run on our laptop in Python: laya.load().system_one(state, questions)
The laya package on our laptop; it takes the same questions as Jev, with the problem as the state.
The text the model reads
The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big. Statement: The bald eagle is green.
The question, sent as its own field
Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.
The answers it can give, each with its meaning
true: The statement holds under the closed-world assumption. false: The statement does not hold under the closed-world assumption.
Its probability for each answer: false 87.8%, true 12.2%.
The full request and response, exactly as sent and recorded
Request
{
"call": "laya.load().system_one(state, questions)",
"state": "The bald eagle is cold. Big things are green. Cold things are blue. All blue, cold things are big.\n\nStatement: The bald eagle is green.",
"questions": {
"Decision": {
"type": "choice",
"instructions": "Using only the facts and rules in the text, is the statement true or false? Assume that anything which cannot be derived from the facts and rules is false: a positive statement is true only if it can be derived, and a statement with 'not' is true when the unnegated statement cannot be derived. A rule applies when all of its conditions hold; 'not' in a condition holds when the unnegated condition cannot be derived.",
"criteria": {
"true": "The statement holds under the closed-world assumption.",
"false": "The statement does not hold under the closed-world assumption."
}
}
}
}Response (the saved record)
{
"model": "laya-upstream:0.3.21",
"answers": {
"Decision": {
"action": {
"act_probability": 1
},
"answer_confidence": 0.878,
"choice": "false",
"confidence": 0.4648,
"probabilities": {
"false": 0.878,
"true": 0.122
},
"type": "choice"
}
},
"usage": {
"input_tokens": 159,
"output_tokens": 0
}
}