Item

Abstract

Creating a controllable and relightable digital avatar from multi-view video
with fixed illumination is a very challenging problem since humans are highly
articulated, creating pose-dependent appearance effects, and skin as well as
clothing require space-varying BRDF modeling. Existing works on creating
animatible avatars either do not focus on relighting at all, require controlled
illumination setups, or try to recover a relightable avatar from very low cost
setups, i.e. a single RGB video, at the cost of severely limited result
quality, e.g. shadows not even being modeled. To address this, we propose
Relightable Neural Actor, a new video-based method for learning a pose-driven
neural human model that can be relighted, allows appearance editing, and models
pose-dependent effects such as wrinkles and self-shadows. Importantly, for
training, our method solely requires a multi-view recording of the human under
a known, but static lighting condition. To tackle this challenging problem, we
leverage an implicit geometry representation of the actor with a drivable
density field that models pose-dependent deformations and derive a dynamic
mapping between 3D and UV spaces, where normal, visibility, and materials are
effectively encoded. To evaluate our approach in real-world scenarios, we
collect a new dataset with four identities recorded under different light
conditions, indoors and outdoors, providing the first benchmark of its kind for
human relighting, and demonstrating state-of-the-art relighting results for
novel human poses.