hide
Free keywords:
Computer Science, Computer Vision and Pattern Recognition, cs.CV,Computer Science, Graphics, cs.GR,Computer Science, Learning, cs.LG
Abstract:
We propose Neural Actor (NA), a new method for high-quality synthesis of
humans from arbitrary viewpoints and under arbitrary controllable poses. Our
method is built upon recent neural scene representation and rendering works
which learn representations of geometry and appearance from only 2D images.
While existing works demonstrated compelling rendering of static scenes and
playback of dynamic scenes, photo-realistic reconstruction and rendering of
humans with neural implicit methods, in particular under user-controlled novel
poses, is still difficult. To address this problem, we utilize a coarse body
model as the proxy to unwarp the surrounding 3D space into a canonical pose. A
neural radiance field learns pose-dependent geometric deformations and pose-
and view-dependent appearance effects in the canonical space from multi-view
video input. To synthesize novel views of high fidelity dynamic geometry and
appearance, we leverage 2D texture maps defined on the body model as latent
variables for predicting residual deformations and the dynamic appearance.
Experiments demonstrate that our method achieves better quality than the
state-of-the-arts on playback as well as novel pose synthesis, and can even
generalize well to new poses that starkly differ from the training poses.
Furthermore, our method also supports body shape control of the synthesized
results.
