CReFT-CAD: Boosting Orthographic Projection Reasoning for CAD via Reinforcement Fine-Tuning

Haiyang Yu (Texas A&M University - College Station) · Bin Li (Microsoft) · Ke Niu (Fudan University) · Zhuofan Chen (Fudan University) · Yuwen Chen (Fudan University) · Teng Fu (Fudan University) · Mengyang Zhao (Fudan University) · Xiangyang Xue (Fudan University)
3d reconstruction pipelinescomplex reasoning taskscomputer-aided designcurriculum-driven reinforcement learningdeep-learning approachesdifficulty-aware rewardsdimensional annotationsinstruction followinglarge-scale benchmarkorthographic projection reasoningout-of-distribution performanceparametric editabilitysemantic extractionsupervised fine-tuningvision-language models

Computer-Aided Design (CAD) is pivotal in industrial manufacturing, with orthographic projection reasoning foundational to its entire workflow—encompassing design, manufacturing, and simulation. However, prevailing deep-learning approaches employ standard 3D reconstruction pipelines as an alternative, which often introduce imprecise dimensions and limit the parametric editability required for CAD workflows. Recently, some researchers adopt vision–language models (VLMs), particularly supervised fine-tuning (SFT), to tackle CAD-related challenges. SFT shows promise but often devolves into pattern memorization, resulting in poor out-of-distribution (OOD) performance on complex reasoning tasks. To tackle these limitations, we introduce CReFT-CAD, a two-stage fine-tuning paradigm: first, a curriculum-driven reinforcement learning stage with difficulty-aware rewards to steadily build reasoning abilities; second, supervised post-tuning to refine instruction following and semantic extraction. Complementing this, we release TriView2CAD, the first large-scale, open-source benchmark for orthographic projection reasoning, comprising 200,000 synthetic and 3,000 real-world orthographic projections with precise dimensional annotations and six interoperable data modalities. Benchmarking leading VLMs on orthographic projection reasoning, we show that CReFT-CAD significantly improves reasoning accuracy and OOD generalizability in real-world scenarios, providing valuable insights to advance CAD reasoning research. The code and adopted datasets are available at \url{https://github.com/KeNiu042/CReFT-CAD}.