SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications

Jinyang Li (The University of Hong Kong) · Reynold Cheng (University of Hong Kong) · Xiaolong Li (Tencent America LLC) · Yue Zhang (Suzhou University) · Yongfeng Huang (Tsinghua University, Tsinghua University) · Ge Qu (The University of Hong Kong) · Per Jacobsson (Google) · Bowen Qin (Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Chinese Academy of Sciences) · Binyuan Hui (Alibaba Group) · Shuzheng Si (Tsinghua University) · Nan Huo (the University of Hong Kong, University of Hong Kong) · Xiaohan Xu (The University of Hong Kong) · Ziwei Tang (University of Hong Kong) · Yuanshuai Li (The Chinese University of Hong Kong,Shenhen) · Florensia Widjaja (The Chinese University of Hong Kong, Shenzhen) · Xintong Zhu (The Chinese University of Hong Kong, Shenzhen) · Feige Zhou (The Chinese University of Hong Kong) · Yannis Papakonstantinou (Google) · Fatma Ozcan (Google) · Ma Chenhao (The Chinese University of Hong Kong, Shenzhen)

bird-criticbird-fixerdebugging plansexecutable issue-solution datasetsf-plan boostingmulti-dialect tasksopen-source modelspostgresql tasksqwen-2.5-coder-14breasoning modelsql debuggingsql-rewind strategysupervisory signalstrajectory-based fine-tuning

Resolution of complex SQL issues persists as a significant bottleneck in real-world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL translation, have not been rigorously evaluated on the more challenging task of debugging on SQL issues. In order to address this gap, we introduce **BIRD-CRITIC**, a new SQL issue debugging benchmark comprising 530 carefully curated PostgreSQL tasks (**BIRD-CRITIC-PG**) and 570 multi-dialect tasks (**BIRD-CRITIC-Multi**), which are distilled from authentic user issues and replayed within new environments to facilitate rigorous and contamination-free evaluation. Baseline evaluations on BIRD-CRITIC underscore the task's complexity, with the leading reasoning model **O3-Mini** achieving only 38.87% success rate on **BIRD-CRITIC-PG** and 33.33% on **BIRD-CRITIC-Multi**. Meanwhile, realizing open-source models for database tasks is crucial which can empower local development while safeguarding data privacy. Therefore, we present **Six-Gym** (**S**ql-f**IX**-Gym), a training environment for elevating the capabilities of open-source models specifically for SQL issue debugging. This environment leverages **SQL-Rewind** strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory-based fine-tuning methods do not explore substantial supervisory signals. We further propose *f*-Plan Boosting, which extracts high-level debugging plans automatically from SQL solutions, enabling the teacher LLMs to harvest and produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, **BIRD-Fixer**. Based on Qwen-2.5-Coder-14B, **BIRD-Fixer** raises its success rate to 38.11% on **BIRD-CRITIC-PG** and 29.65% on **BIRD-CRITIC-Multi**, surpassing many leading proprietary models such as Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing sophisticated SQL-debugging capabilities for both research and industry.