3D Gaussian Flats: Hybrid 2D/3D Photometric Scene Reconstruction

Zhen Li (Shenzhen Future Network of Intelligence Institute and Guangdong Provincial Key Laboratory of Future Networks of Intelligence, The Chinese University of Hong Kong (Shenzhen)) · Lily Goli (University of Toronto) · Maria Taktasheva (Simon Fraser University) · Alessandro Fiorini (University of Bologna) · Daniel Rebain (University of British Columbia) · Andrea Tagliasacchi (Simon Fraser University)
constrained planar gaussiansdepth estimationdigital twinsfreeform gaussiansgeometric accuracyhigh-quality reconstructionhybrid representationindoor scenesmesh extractionnovel view synthesisoverfittingphotometric reconstructionradiance fieldsscannet++surface reconstructionvisual fidelity

Recent advances in radiance fields and novel view synthesis enable creation of realistic digital twins from photographs. However, current methods struggle with flat, texture-less surfaces, creating uneven and semi-transparent reconstructions, due to an ill-conditioned photometric reconstruction objective. Surface reconstruction methods solve this issue but sacrifice visual quality. We propose a novel hybrid 2D/3D representation that jointly optimizes constrained planar (2D) Gaussians for modeling flat surfaces and freeform (3D) Gaussians for the rest of the scene. Our end-to-end approach dynamically detects and refines planar regions, improving both visual fidelity and geometric accuracy. It achieves state-of-the-art depth estimation on ScanNet++ and ScanNetv2, and excels at mesh extraction without overfitting to a specific camera model, showing its effectiveness in producing high-quality reconstruction of indoor scenes.