hide
Free keywords:
Computer Science, Computer Vision and Pattern Recognition, cs.CV
Abstract:
We present the first end to end approach for real time material estimation
for general object shapes with uniform material that only requires a single
color image as input. In addition to Lambertian surface properties, our
approach fully automatically computes the specular albedo, material shininess,
and a foreground segmentation. We tackle this challenging and ill posed inverse
rendering problem using recent advances in image to image translation
techniques based on deep convolutional encoder decoder architectures. The
underlying core representations of our approach are specular shading, diffuse
shading and mirror images, which allow to learn the effective and accurate
separation of diffuse and specular albedo. In addition, we propose a novel
highly efficient perceptual rendering loss that mimics real world image
formation and obtains intermediate results even during run time. The estimation
of material parameters at real time frame rates enables exciting mixed reality
applications, such as seamless illumination consistent integration of virtual
objects into real world scenes, and virtual material cloning. We demonstrate
our approach in a live setup, compare it to the state of the art, and
demonstrate its effectiveness through quantitative and qualitative evaluation.
