BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models

Tao Kong (Bytedance) · Xiao Ma (National University of Singapore) · Liang Wang (NLPR, China) · Hongtao Wu (Bytedance Research) · Tieniu Tan (Chinese Academy of Sciences) · Peiyan Li (Institute of Automation, Chinese Academy of Sciences) · Yixiang Chen (Institute of automation, Chinese academy of science, Chinese Academy of Sciences) · Xiangnan Wu (University of Science and Technology Beijing) · Yan Huang (CRIPAC, CASIA)

3d signalsaction prediction data efficiency fine-tuninggeneralization settingsheatmapsmulti-view imagesout-of-distributionpoint cloud inputspre-trainingrobot manipulation learningspatial structure success rate trajectories vision-language models vision-language-action

Recently, leveraging pre-trained vision-language models (VLMs) for building vision-language-action (VLA) models has emerged as a promising approach to effective robot manipulation learning. However, only few methods incorporate 3D signals into VLMs for action prediction, and they do not fully leverage the spatial structure inherent in 3D data, leading to low data efficiency. In this paper, we introduce a new paradigm for constructing 3D VLAs. Specifically, we first pre-train the VLM backbone to take 2D images as input and produce 2D heatmaps as output. Using this pre-trained VLM as the backbone, we then fine-tune the entire VLA model while maintaining alignment between inputs and outputs by: (1) projecting raw point cloud inputs into multi-view images, and (2) predicting heatmaps before generating the final action. Extensive experiments show that the resulting model, BridgeVLA, can learn 3D manipulation both efficiently and effectively. BridgeVLA outperforms state-of-the-art baselines across three simulation benchmarks. In RLBench, it improves the average success rate from 81.4\% to 88.2\%. In COLOSSEUM, it demonstrates significantly better performance in challenging generalization settings, boosting the average success rate from 56.7\% to 64.0\%. In GemBench, it surpasses all the comparing baseline methods in terms of average success rate. In real-robot experiments, BridgeVLA outperforms a state-of-the-art baseline method by 32\% on average. It generalizes robustly in multiple out-of-distribution settings, including visual disturbances and unseen instructions. Remarkably, it is able to achieve a success rate of 95.4\% on 10+ tasks with only 3 trajectories per task, while other VLA methods such as $\pi_{0}$ fail completely. Project Website: https://bridgevla.github.io/.