AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation

Shang Gao (Deakin University) · Junjun He (Shanghai AI Laboratory) · Jilan Xu (Fudan University) · Seongsu Bae (KAIST) · Qingqiu Li (Fudan University) · Zihang Cui (Xidian University) · Runtian Yuan (Fudan University) · Yuejie Zhang (Fudan University) · Rui Feng (Fudan University) · Quanli Shen (Fudan University) · Xiaobo Zhang (Fudan University) · Shujun Wang (Hong Kong Polytechnic University)
anatomical ontology-guided reasoninganatomy-centric reasoningaor-instructionautomated cxr interpretationdiagnostic accuracyexpert physiciansexplainabilityinstruction datasetinteractivitymedical large multimodal modelsmultimodal multi-step reasoningperformance evaluationregion-level understandingreport generationvisual question answering

Chest X-rays (CXRs) are the most frequently performed imaging examinations in clinical settings. Recent advancements in Medical Large Multimodal Models (MLMMs) have enabled automated CXR interpretation, improving diagnostic accuracy and efficiency. However, despite their strong visual understanding, current MLMMs still face two major challenges: (1) insufficient region-level understanding and interaction, and (2) limited accuracy and interpretability due to single-step prediction. In this paper, we address these challenges by empowering MLMMs with anatomy-centric reasoning capabilities to enhance their interactivity and explainability. Specifically, we propose an Anatomical Ontology-Guided Reasoning (AOR) framework that accommodates both textual and optional visual prompts, centered on region-level information to enable multimodal multi-step reasoning. We also develop AOR-Instruction, a large instruction dataset for MLMMs training, under the guidance of expert physicians. Our experiments demonstrate AOR's superior performance in both Visual Question Answering (VQA) and report generation tasks. Code and data are available at: https://github.com/Liqq1/AOR.