OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning

Hao Liu (Bytedance) · Hao Feng (University of Science and Technology of China) · Xiang Bai (Huazhong University of Science and Technology) · Xinyu Wang (The University of Adelaide) · Wei Chen (Huazhong University of Science and Technology) · Qi Liu (Bytedance Inc.) · Ling Fu (Huazhong University of Science and Technology) · Zhebin Kuang (Huazhong University of Science and Technology) · Jiajun Song (Huazhong University of Science and Technology) · Mingxin Huang (South China University of Technology) · Biao Yang (Huazhong University of Science and Technology) · Yuzhe Li (Huazhong University of Science and Technology) · Linghao Zhu (Huazhong University of Science and Technology) · Qidi Luo (Huazhong University of Science and Technology) · Hao Lu (Huazhong University of Science and Technology) · Zhang Li (Huazhong University of Science and Technology) · Guozhi Tang (ByteDance Inc.) · Bin Shan (bytedance) · Chunhui Lin (Bytedance) · Binghong Wu (Tencent) · Can Huang (Bytedance) · Jingqun Tang (Bytedance China) · Lianwen Jin (South China University of Technology) · Yuliang Liu (Huazhong University of Science and Technology)
bilingual text-centric benchmarkcomplex element parsingevaluation metricsfine-grained perceptionhandwritten content extractionhuman-verified question-answering pairslarge multimodal modelslayout perceptionlogical reasoningmanually annotated imagesocrbench v2optical character recognitionprivate test setstate-of-the-art lmmstext localization

Scoring the Optical Character Recognition (OCR) capabilities of Large Multimodal Models (LMMs) has witnessed growing interest. Existing benchmarks have highlighted the impressive performance of LMMs in text recognition; however, their abilities in certain challenging tasks, such as text localization, handwritten content extraction, and logical reasoning, remain underexplored. To bridge this gap, we introduce OCRBench v2, a large-scale bilingual text-centric benchmark with currently the most comprehensive set of tasks ($4\times$ more tasks than the previous multi-scene benchmark OCRBench), the widest coverage of scenarios ($31$ diverse scenarios), and thorough evaluation metrics, with $10,000$ human-verified question-answering pairs and a high proportion of difficult samples. Moreover, we construct a private test set with $1,500$ manually annotated images. The consistent evaluation trends observed across both public and private test sets validate the OCRBench v2's reliability. After carefully benchmarking state-of-the-art LMMs, we find that most LMMs score below $50$ ($100$ in total) and suffer from five-type limitations, including less frequently encountered text recognition, fine-grained perception, layout perception, complex element parsing, and logical reasoning. The benchmark and evaluation scripts are available at https://github.com/Yuliang-Liu/MultimodalOCR.