Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues

Chinmay Talegaonkar (University of California, San Diego) · Nikhil Gandudi Suresh (University of California, San Diego) · Zachary Novack (University of California, San Diego) · Yash Belhe (University of California, San Diego) · Priyanka Nagasamudra (University of California, San Diego) · Nicholas Antipa (University of California, San Diego)

defocus blur cuesdefocus-blur image formation modelgradients from loss functioninference timemetric depth predictormetric depth scaling parametersmonocular metric depth estimationnoise latentsout-of-distribution datasetspre-trained diffusion modelqualitative improvementsquantitative improvementsscale-invariant monocular depth estimationstate-of-the-art methods training-free manner zero-shot generalization

Recent monocular metric depth estimation (MMDE) methods have made notable progress towards zero-shot generalization. However, they still exhibit a significant performance drop on out-of-distribution datasets. We address this limitation by injecting defocus blur cues at inference time into Marigold, a \textit{pre-trained} diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE). Our method effectively turns Marigold into a metric depth predictor in a training-free manner. To incorporate defocus cues, we capture two images with a small and a large aperture from the same viewpoint. To recover metric depth, we then optimize the metric depth scaling parameters and the noise latents of Marigold at inference time using gradients from a loss function based on the defocus-blur image formation model. We compare our method against existing state-of-the-art zero-shot MMDE methods on a self-collected real dataset, showing quantitative and qualitative improvements.