Item

Abstract

Applying data-driven approaches to non-rigid 3D reconstruction has been
difficult, which we believe can be attributed to the lack of a large-scale
training corpus. One recent approach proposes self-supervision based on
non-rigid reconstruction. Unfortunately, this method fails for important cases
such as highly non-rigid deformations. We first address this problem of lack of
data by introducing a novel semi-supervised strategy to obtain dense
inter-frame correspondences from a sparse set of annotations. This way, we
obtain a large dataset of 400 scenes, over 390,000 RGB-D frames, and 2,537
densely aligned frame pairs; in addition, we provide a test set along with
several metrics for evaluation. Based on this corpus, we introduce a
data-driven non-rigid feature matching approach, which we integrate into an
optimization-based reconstruction pipeline. Here, we propose a new neural
network that operates on RGB-D frames, while maintaining robustness under large
non-rigid deformations and producing accurate predictions. Our approach
significantly outperforms both existing non-rigid reconstruction methods that
do not use learned data terms, as well as learning-based approaches that only
use self-supervision.