Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

Hao Zhao (Tsinghua University, Tsinghua University) · Hang Zhao (Harbin Institute of Technology) · Chenyu Liu (Nanyang Technological University) · Ziming Liu (MIT, Stanford) · Haohan Chi (Tsinghua University) · Huan-ang Gao (Tsinghua University) · HAO SUN (Bosch) · Jianing Liu (中央民族大学) · Jinwei Li (Tsinghua University) · Kaisen Yang (Tsinghua University) · Yangcheng Yu (Tsinghua University) · Zeda Wang (Tsinghua University, Tsinghua University) · Wenyi Li (Institute of Software, Chinese Academy of Sciences) · Leichen Wang (Bosch) · Xingtao HU (Bosch)
action trajectoriesautonomous drivingcollision ratesimpromptu vla datasetl2 accuracyneuroncap scoresnuscenes trajectory predictionperceptionplanningplanning-oriented question-answeringpredictionq&a suiteunstructured corner casesvideo clipsvision-language-actionvlm improvements

Vision-Language-Action (VLA) models for autonomous driving show promise but falter in unstructured corner case scenarios, largely due to a scarcity of targeted benchmarks. To address this, we introduce Impromptu VLA. Our core contribution is the Impromptu VLA Dataset: over 80,000 meticulously curated video clips, distilled from over 2M source clips sourced from 8 open-source large-scale datasets. This dataset is built upon our novel taxonomy of four challenging unstructured categories and features rich, planning-oriented question-answering annotations and action trajectories. Crucially, experiments demonstrate that VLAs trained with our dataset achieve substantial performance gains on established benchmarks—improving closed-loop NeuroNCAP scores and collision rates, and reaching near state-of-the-art L2 accuracy in open-loop nuScenes trajectory prediction. Furthermore, our Q&A suite serves as an effective diagnostic, revealing clear VLM improvements in perception, prediction, and planning. Our code, data and models are available at https://github.com/ahydchh/Impromptu-VLA