アイテム詳細

要旨

Robust face reconstruction from monocular image in general lighting
conditions is challenging. Methods combining deep neural network encoders with
differentiable rendering have opened up the path for very fast monocular
reconstruction of geometry, lighting and reflectance. They can also be trained
in self-supervised manner for increased robustness and better generalization.
However, their differentiable rasterization based image formation models, as
well as underlying scene parameterization, limit them to Lambertian face
reflectance and to poor shape details. More recently, ray tracing was
introduced for monocular face reconstruction within a classic
optimization-based framework and enables state-of-the art results. However
optimization-based approaches are inherently slow and lack robustness. In this
paper, we build our work on the aforementioned approaches and propose a new
method that greatly improves reconstruction quality and robustness in general
scenes. We achieve this by combining a CNN encoder with a differentiable ray
tracer, which enables us to base the reconstruction on much more advanced
personalized diffuse and specular albedos, a more sophisticated illumination
model and a plausible representation of self-shadows. This enables to take a
big leap forward in reconstruction quality of shape, appearance and lighting
even in scenes with difficult illumination. With consistent face attributes
reconstruction, our method leads to practical applications such as relighting
and self-shadows removal. Compared to state-of-the-art methods, our results
show improved accuracy and validity of the approach.