deVision v0.3 evaluation
Generated by scripts/release/hf.py from the evaluation of this release.
| Set | Questions | Accuracy | Mismatched | NLL | ECE |
|---|---|---|---|---|---|
| COCO object presence | 1,120 | 0.947 | 0.504 | 0.163 | 0.034 |
| VQAv2 multiple choice | 1,420 | 0.909 | 0.380 | 0.271 | 0.038 |
| COCO size | 1,100 | 0.872 | 0.511 | 0.336 | 0.029 |
| POPE, project filtered set | 8,676 | 0.850 | 0.527 | 0.368 | 0.039 |
| GQA val subset | 992 | 0.790 | 0.550 | 0.410 | 0.040 |
| COCO position | 1,274 | 0.885 | 0.493 | 0.288 | 0.029 |
| VQAv2 yes/no subset | 1,000 | 0.704 | 0.512 | 0.574 | 0.026 |
| COCO relative position | 1,950 | 0.755 | 0.496 | 0.490 | 0.038 |
| VSR, project held-out split | 904 | 0.661 | 0.487 | 0.620 | 0.046 |
| Visual7W, project held-out split | 1,000 | 0.721 | 0.419 | 0.673 | 0.041 |
| Fresh counting test (unseen pictures) | 600 | 0.763 | 0.405 | 0.574 | 0.054 |
| CountBench, 0–10 choices | 491 | 0.356 | 0.104 | 1.602 | 0.052 |
| COCO mirrored position | 640 | 0.847 | 0.477 | 0.359 | 0.044 |
| COCO mirrored relative position | 1,492 | 0.698 | 0.507 | 0.545 | 0.030 |
| GQA binding test | 1,484 | 0.676 | 0.498 | 0.559 | 0.028 |
| What'sUp | 820 | 0.287 | 0.245 | 1.491 | 0.245 |
Laya Vision 201M, same questions
| Set | Questions | Laya Vision | deVision | Difference |
|---|---|---|---|---|
| VQAv2 yes/no | 4,887 | 0.717 | 0.725 | +0.8 [-0.8, +2.4] |
| A-OKVQA | 1,138 | 0.598 | 0.613 | +1.4 [-1.8, +4.7] |
| ScienceQA with images | 2,097 | 0.824 | 0.789 | -3.5 [-5.6, -1.4] |
| ScienceQA natural science, needs the picture | 323 | 0.700 | 0.495 | -20.4 [-27.2, -13.3] |
On the full POPE random / popular / adversarial sets (3,000 questions each) deVision scores 0.880 / 0.859 / 0.805, against Laya Vision's published 0.836 / 0.819 / 0.777 (aggregate scores only, not paired).
Temperatures
| Bucket | Temperature |
|---|---|
| Choice, 11+ options | 1.2012 |
| Choice, 2 options | 1.9488 |
| Choice, 3–5 options | 1.7575 |
| Noul (yes / no) | 1.8258 |
| Choice, any other count | 1.5922 |