Multimodal Causal Reasoning for UAV Object Detection

Ce Zhu (University of Electronic Science and Technology of China) · Nianxin Li (University of Electronic Science and Technology of China) · Mao Ye (School of Computer Science and Engineering, University of Electronic Science and Technology of China) · Lihua Zhou (University of Electronic Science and Technology of China) · Shuaifeng Li (University of Electronic Science and Technology of China) · Song Tang (University of Shanghai for Science and Technology) · Luping Ji (University of Electronic Science and Technology of China)

backdoor adjustmentcausal inference clipcondition variationscondition-invariant representationconfounder dictionarylanguage guided region explorationmultimodel causal reasoningobject detectionprior probability distributionsemantic text promptstext embeddingstext-image contrastive learningunmanned aerial vehiclevision-language modelyolo backbone

Unmanned Aerial Vehicle (UAV) object detection faces significant challenges due to complex environmental conditions and different imaging conditions. These factors introduce significant changes in scale and appearance, particularly for small objects that occupy limited pixels and exhibit limited information, complicating detection tasks. To address these challenges, we propose a Multimodel Causal Reasoning framework based on YOLO backbone for UAV Object Detection (MCR-UOD). The key idea is to use the backdoor adjustment to discover the condition-invariant object representation for easy detection. Specifically, the YOLO backbone is first adjusted to incorporate the pre-trained vision-language model. The original category labels are replaced with semantic text prompts, and the detection head is replaced with text-image contrastive learning. Based on this backbone, our method consists of two parts. The first part, named language guided region exploration, discovers the regions with high probability of object existence using text embeddings based on vision-language model such as CLIP. Another part is the backdoor adjustment casual reasoning module, which constructs a confounder dictionary tailored to different imaging conditions to capture global image semantics and derives a prior probability distribution of shooting conditions. During causal inference, we use the confounder dictionary and the prior to intervene on local instance features, disentangling condition variations, and obtaining condition-invariant representations. Experimental results on several public datasets confirm the state-of-the-art performance of our approach. The code, data and models will be released upon publication of this paper.