SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving

Zhen Li (Shenzhen Future Network of Intelligence Institute and Guangdong Provincial Key Laboratory of Future Networks of Intelligence, The Chinese University of Hong Kong (Shenzhen)) · Haiming Zhang (The Chinese University of Hong Kong, Shenzhen) · Yiyao Zhu (Hong Kong University of Science and Technology) · Wending Zhou ( The Chinese University of Hongkong, Shenzhen) · Xu Yan (The Chinese University of Hongkong, Shenzhen) · Yingjie CAI (Huawei Technologies Ltd.) · Bingbing Liu (Huawei) · Shuguang Cui (The Chinese University of Hong Kong, Shenzhen)

3d gaussian representations3d object detectionautonomous driving benchmarksdepth mapsdownstream networksfine-grained contextual featuresmulti-view imagesoccupancy prediction performance gains pre-trainingquery interaction mechanismsquery-driven paradigmself-supervised splattingsparse perception modelsstate-of-the-art pre-training approaches

Sparse Perception Models (SPMs) adopt a query-driven paradigm that forgoes explicit dense BEV or volumetric construction, enabling highly efficient computation and accelerated inference. In this paper, we introduce SQS, a novel query-based splatting pre-training specifically designed to advance SPMs in autonomous driving. SQS introduces a plug-in module that predicts 3D Gaussian representations from sparse queries during pre-training, leveraging self-supervised splatting to learn fine-grained contextual features through the reconstruction of multi-view images and depth maps. During fine-tuning, the pre-trained Gaussian queries are seamlessly integrated into downstream networks via query interaction mechanisms that explicitly connect pre-trained queries with task-specific queries, effectively accommodating the diverse requirements of occupancy prediction and 3D object detection. Extensive experiments on autonomous driving benchmarks demonstrate that SQS delivers considerable performance gains across multiple query-based 3D perception tasks, notably in occupancy prediction and 3D object detection, outperforming prior state-of-the-art pre-training approaches by a significant margin (i.e., +1.3 mIoU on occupancy prediction and +1.0 NDS on 3D detection).