GLiNER2.5-Decide
Asked every open world question a second time, GLiNER2.5-Decide gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 32.8% the first time and 32.8% the second.
Asked every closed world question a second time, GLiNER2.5-Decide gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 50.3% the first time and 50.3% the second.
Calibration on the open world task: GLiNER2.5-Decide's probability for its own answer averaged 93.0% against 32.8% accuracy; expected calibration error 60.2 points.
Calibration on the closed world task: GLiNER2.5-Decide's probability for its own answer averaged 64.6% against 50.3% accuracy; expected calibration error 14.3 points.
Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.51 on the open world task against Jev's 0.85; 0.53 on the closed world task against Jev's 0.86.
GLiNER2.5-Decide, without instructions
Asked every open world question a second time, GLiNER2.5-Decide, without instructions gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 45.5% the first time and 45.5% the second.
Asked every closed world question a second time, GLiNER2.5-Decide, without instructions gave the same answer on 100.0% of 1,800 items (0 changed; Gwet's AC1 1.000, interval 1.000 to 1.000). Accuracy was 54.3% the first time and 54.3% the second.
Calibration on the open world task: GLiNER2.5-Decide, without instructions's probability for its own answer averaged 68.8% against 45.5% accuracy; expected calibration error 23.3 points.
Calibration on the closed world task: GLiNER2.5-Decide, without instructions's probability for its own answer averaged 87.3% against 54.3% accuracy; expected calibration error 33.1 points.
Whether its probability tells right answers from wrong ones (AUROC, where 0.5 is no better than chance and 1 is perfect): 0.60 on the open world task against Jev's 0.85; 0.55 on the closed world task against Jev's 0.86.
Consistency and calibration for every model, Jev included