MS-Bench: Evaluating LMMs in Ancient Manuscript Study through a Dunhuang Case Study

Junchi Yan (Shanghai Jiao Tong University) · Fei Wu · Yuqing Zhang (University of Queensland) · Zemin Liu (Singapore Management University) · Kun Kuang (Zhejiang University) · Yue Han (Zhejiang University) · Shuanghe Zhu (Zhejiang University) · Haoxiang Wu (Zhejiang University) · Hangqi Li (University of California, Berkeley) · Shengyu Zhang (Zhejiang University) · Huaiyong Dou (Zhejiang University) · Yongquan Zhang (Zhejiang University)

archaeological workflowscultural contextualizationdomain expertiseexpert-curated questionshigh-resolution imageshuman–ai collaborationlmmsmanuscript studyms-benchperformance evaluation prompting strategies robustnessscale-driven performancetask-specific preferencesvisual retrieval-augmented prompts

Analyzing ancient manuscripts has traditionally been a labor-intensive and time-consuming task for philologists. While recent advancements in LMMs have demonstrated their potential across diverse domains, their effectiveness in manuscript study remains underexplored. In this paper, we introduce MS-Bench, the first comprehensive benchmark co-developed with archaeologists, comprising 5,076 high-resolution images from 4th to 14th century and 9,982 expert-curated questions across nine sub-tasks aligned with archaeological workflows. Through four prompting strategies, we systematically evaluate 32 LMMs on their effectiveness, robustness, and cultural contextualization. Our analysis reveals scale-driven performance and reliability improvements, prompting strategies' impact on performance (CoT has two-sides effect, while visual retrieval-augmented prompts provide consistent boost), and task-specific preferences depending on LMM’s visual capabilities. Although current LMMs are not yet capable of replacing domain expertise, they demonstrate promising potential to accelerate manuscript research through future human–AI collaboration.