DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation

Xiaoliang Ju1 *, Zhaoyang Huang1 *, Yijin Li2, Guofeng Zhang2, Yu Qiao3, Hongsheng Li1
1MMLab, The Chinese University of Hong Kong
, 2Zhejiang University
, 3Shanghai AI Laboratory
*Indicates Equal Contribution
Paper [CVPR2024] Supplementary arXiv

Demonstration of our whole pipeline and the generation results.

MY ALT TEXT

Coarse-to-fine indoor scene geometry generation using a sparse diffusion framework. For better visualization, the texture is produced by DreamSpace after the scene geometry is generated by our DiffInDScene.

Abstract

We present DiffInDScene, a novel framework for tackling the problem of high-quality 3D indoor scene generation, which is challenging due to the complexity and diversity of the indoor scene geometry. Although diffusion-based generative models have previously demonstrated impressive performance in image generation and object-level 3D generation, they have not yet been applied to room-level 3D generation due to their computationally intensive costs. In DiffInDScene, we propose a cascaded 3D diffusion pipeline that is efficient and possesses strong generative performance for Truncated Signed Distance Function (TSDF). The whole pipeline is designed to run on a sparse occupancy space in a coarse-to-fine fashion. Inspired by KinectFusion's incremental alignment and fusion of local TSDF volumes, we propose a diffusion-based SDF fusion approach that iteratively diffuses and fuses local TSDF volumes, facilitating the generation of an entire room environment. The generated results demonstrate that our work is capable to achieve high-quality room generation directly in three-dimensional space, starting from scratch. In addition to the scene generation, the final part of DiffInDScene can be used as a post-processing module to refine the 3D reconstruction results from multi-view stereo. According to the user study, the mesh quality generated by our DiffInDScene can even outperform the ground truth mesh provided by ScanNet.

BibTeX

@unpublished{ju2023diffroom,
        title={DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation}, 
        author={Xiaoliang Ju and Zhaoyang Huang and Yijin Li and Guofeng Zhang and Yu Qiao and Hongsheng Li},
        year={2023},
        eprint={2306.00519},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
  }