Every model's answers as a confusion matrix
A confusion matrix sorts every problem by its correct answer (the rows) and what the model said (the columns). Right answers sit on the diagonal; everything off it is an error, and where the errors land shows which answer a model reaches for when it is wrong. Fully scored models, most accurate first, on the same 1,800 open-world problems.
right answers (the diagonal)errorsDarker means a larger share of that row. Rows are the correct answer; columns are what the model said; each cell shows its count.
Jev got 74% of unknown items right; it said true on 21% of them. GLiDE, Oct 2 got 76% of false items right; it said unknown on 24% of them. GLiDE, Oct 3 rerun got 65% of false items right; it said unknown on 34% of them. GPT-6 Luna (reasoning off) got 51% of false items right; it said unknown on 48% of them. Kev-9B got 49% of false items right; it said unknown on 39% of them. Kev-4B got 36% of false items right; it said unknown on 57% of them. Kev-0.8B got 12% of unknown items right; it said false on 63% of them. GLiNER2.5-Decide, without instructions got 1% of unknown items right; it said false on 54% of them. Laya got 1% of unknown items right; it said false on 77% of them. GLiNER2.5-Decide got 0% of true items right; it said unknown on 100% of them.
The closed-world task (true or false)
Jev said true on 50% of problems; the correct answer is true on 50%. GLiDE, Oct 2 said true on 54% of problems; the correct answer is true on 50%. GLiDE, Oct 3 rerun said true on 54% of problems; the correct answer is true on 50%. GPT-6 Luna (reasoning off) said true on 51% of problems; the correct answer is true on 50%. Kev-9B said true on 51% of problems; the correct answer is true on 50%. Kev-4B said false on 67% of problems; the correct answer is false on 50%. Kev-0.8B said false on 60% of problems; the correct answer is false on 50%. Laya said false on 81% of problems; the correct answer is false on 50%. GLiNER2.5-Decide, without instructions said false on 57% of problems; the correct answer is false on 50%. GLiNER2.5-Decide said false on 98% of problems; the correct answer is false on 50%.