DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks

Hao Chen (Harbin Institute of Technology) · Tong He (Shanghai AI lab) · Canyu Zhao (Zhejiang University) · Yanlong Sun (Tsinghua University, Tsinghua University) · Mingyu Liu (Zhejiang University) · Huanyi Zheng (Zhejiang University) · Muzhi Zhu (Zhejiang University) · Zhiyue Zhao (Zhejiang University) · Chunhua Shen (University of Adelaide)

architecture experimentsclassifier-free guidancecomputational cost reductioncomputational resource constraintsdepth estimationdiceptionfine-tuning efficiencygeneralist perception modelinput paradigmslimited training datapixel-aligned trainingpixel-level annotated imagesprior knowledge preservationsingle-task specialist modelstext-to-image diffusion models

This paper's primary objective is to develop a robust generalist perception model capable of addressing multiple tasks under constraints of computational resources and limited training data. We leverage text-to-image diffusion models pre-trained on billions of images and successfully introduce our DICEPTION, a visual generalist model. Exhaustive evaluations demonstrate that DICEPTION effectively tackles diverse perception tasks, even achieving performance comparable to SOTA single-task specialist models. Specifically, we achieve results on par with SAM-vit-h using only 0.06% of their data (e.g., 600K vs.\ 1B pixel-level annotated images). We designed comprehensive experiments on architectures and input paradigms, demonstrating that the key to successfully re-purposing a single diffusion model for multiple perception tasks lies in maximizing the preservation of the pre-trained model's prior knowledge. Consequently, DICEPTION can be trained with substantially lower computational costs than conventional models requiring training from scratch. Furthermore, adapting DICEPTION to novel tasks is highly efficient, necessitating fine-tuning on as few as 50 images and approximately 1% of its parameters. Finally, we demonstrate that a subtle application of classifier-free guidance can improve the model's performance on depth and normal estimation. We also show that pixel-aligned training, as is characteristic of perception tasks, significantly enhances the model's ability to preserve fine details. DICEPTION offers valuable insights and presents a promising direction for the development of advanced diffusion-based visual generalist models.