DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response

Zhihao Liu (The University of Tokyo) · Weihao Xuan (The University of Tokyo & RIKEN AIP) · Junjue Wang (The University of Tokyo) · Heli Qi (Nara Institute of Science and Technology, Japan) · Zhuo Zheng (Stanford University) · Junshi Xia (RIKEN) · Naoto YOKOYA (University of Tokyo & RIKEN AIP) · Kunyi Liu (Waseda University) · Yuhan Wu (State University of New York at Stony Brook) · Hongruixuan Chen (The University of Tokyo) · JIAN SONG (The University of Tokyo, RIKEN AIP)

bi-temporal satellite imagescross-sensor gapdamage object countingdisaster assessmentdisaster-related tasksdisaster-specific corpusfine-tuning generalization capabilitieslong-form disaster reportsmulti-hazardmulti-sensorreasoning tasks remote sensingsynthetic aperture radarvision-language models visual perception tasks

Large vision-language models (VLMs) have made great achievements in Earth vision. However, complex disaster scenes with diverse disaster types, geographic regions, and satellite sensors have posed new challenges for VLM applications. To fill this gap, we curate the first remote sensing vision-language dataset (DisasterM3) for global-scale disaster assessment and response. DisasterM3 includes 26,988 bi-temporal satellite images and 123k instruction pairs across 5 continents, with three characteristics: **1) Multi-hazard**: DisasterM3 involves 36 historical disaster events with significant impacts, which are categorized into 10 common natural and man-made disasters. **2) Multi-sensor**: Extreme weather during disasters often hinders optical sensor imaging, making it necessary to combine Synthetic Aperture Radar (SAR) imagery for post-disaster scenes. **3) Multi-task**: Based on real-world scenarios, DisasterM3 includes 9 disaster-related visual perception and reasoning tasks, harnessing the full potential of VLM's reasoning ability with progressing from disaster-bearing body recognition to structural damage assessment and object relational reasoning, culminating in the generation of long-form disaster reports. We extensively evaluated 14 generic and remote sensing VLMs on our benchmark, revealing that state-of-the-art models struggle with the disaster tasks, largely due to the lack of a disaster-specific corpus, cross-sensor gap, and damage object counting insensitivity. Focusing on these issues, we fine-tune four VLMs using our dataset and achieve stable improvements (up to 10.4\%$\uparrow$QA, 2.1$\uparrow$Report, 40.8\%$\uparrow$Referring Seg.) with robust cross-sensor and cross-disaster generalization capabilities. Project: https://github.com/Junjue-Wang/DisasterM3.