Keywords
Benchmark evaluation
Answer distribution bias
Position bias
OmniBench
Submitted : 2026-05-30
Accepted : 2026-06-14
Published : 2026-06-29
Abstract
OmniBench is a widely used tri-modal (image–audio–text) benchmark containing 1,142 four-choice multiple-choice questions. We discover a severe answer-position skew in OmniBench: option D is correct 48.6% of the time (χ² = 384.34, p = 5.46×10⁻⁸³), nearly twice the expected 25%. To test whether this skew distorts evaluation outcomes, we design an option-shuffling experiment: keeping all question content unchanged, we randomly reassign letter labels so that the correct answer is uniformly distributed (D ≈ 25%), then re-evaluate the same models. Results show that accuracy changes significantly in two of three tested models after shuffling (up to 4.20%, p < 0.01), demonstrating that unequal answer distribution can significantly bias model evaluation outcomes. Furthermore, we propose a label-free content-scoring evaluation method based on conditional log-probability, which achieves distribution-invariant evaluation (accuracy difference ≤ 0.18%, p > 0.4).
References
Li Y, Wu B, Zhao F, et al., 2024, OmniBench: Towards the Future of Universal Omni-Language Models, arXiv preprint arXiv:2409.15272.
Yue X, Ni Y, Zhang K, et al., 2023, MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, Proceedings of CVPR 2024. arXiv:2311.16502.
Liu Y, Duan H, Zhang Y, et al., 2023, MMBench: Is Your Multi-modal Model an All-around Player? Proceedings of ECCV 2024. arXiv:2307.06281.
Fu C, Chen P, Shen Y, et al., 2023, MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, arXiv preprint arXiv:2306.13394.
Wang B, Zou X, Lin G, et al., 2024, AudioBench: A Universal Benchmark for Audio Large Language Models, arXiv preprint arXiv:2406.16020.
Pezeshkpour P, Hruschka E, 2023, Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions, Findings of NAACL-HLT 2024. arXiv:2308.11483.
Zheng C, Zhou H, Meng F, et al., 2023, Large Language Models Are Not Robust Multiple Choice Selectors, Proceedings of ICLR 2024 (Spotlight). arXiv:2309.03882
Zhao Z, Wallace E, Feng S, et al., 2021, Calibrate Before Use: Improving Few-Shot Performance of Language Models, Proceedings of ICML 2021. arXiv:2102.09690.
Zhou H, Wan X, Proleev L, et al., 2023, Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering, Proceedings of ICLR 2024. arXiv:2309.17249.
Yue X, Zheng T, Zhang K, et al., 2024, MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark, arXiv preprint arXiv:2409.02813.