FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation

Xin Li (Huazhong University of Science and Technology) · Ming Tang (Southern University of Science and Technology) · Fan Yang (Kuaishou- 快手科技) · Yousong Zhu (China University of Mining Technology - Beijing) · Yufei Zhan (Institute of automation, Chinese academy of science, Chinese Academy of Sciences) · Hongyin Zhao (Institute of automation, Chinese academy of science, Chinese Academy of Sciences) · Shurong Zheng (Institute of automation, Chinese academy of science, Chinese Academy of Sciences) · Yaowei Wang (Pengcheng Laboratory) · Jinqiao Wang (Institute of Automation, Chinese Academy of Sciences)
conditional promptsdiffusion decoderdiscrete visual tokensdual-branch visual encodergenerative modelinglarge vision language modelsmovqganmultimodal understandingobject segmentationprogressive multi-stage trainingreferring segmentation accuracysegmentation maskssemantic contextspatial detailsvisual tokenizervisual understanding

Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling, enabling both accurate content understanding and flexible editing. However, current approaches treat \textbf{\textit{"what to see"}} and \textbf{\textit{"how to edit"}} separately: they either perform isolated object segmentation or utilize segmentation masks merely as conditional prompts for local edit generation tasks, often relying on multiple disjointed models. To bridge these gaps, we introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework. FOCUS employs a dual-branch visual encoder to simultaneously capture global semantic context and fine-grained spatial details. In addition, we leverage a MoVQGAN-based visual tokenizer to produce discrete visual tokens that enhance generation quality. To enable accurate and controllable image editing, we propose a progressive multi-stage training pipeline, where segmentation masks are jointly optimized and used as spatial condition prompts to guide the diffusion decoder. This strategy aligns visual encoding, segmentation, and generation modules, effectively bridging segmentation-aware perception with fine-grained visual synthesis. Extensive experiments across three core tasks, including multimodal understanding, referring segmentation accuracy, and controllable image generation, demonstrate that FOCUS achieves strong performance by jointly optimizing visual perception and generative capabilities.