hide
Free keywords:
Computer Science, Computer Vision and Pattern Recognition, cs.CV,Computer Science, Graphics, cs.GR
Abstract:
We propose an image-based, facial reenactment system that replaces the face
of an actor in an existing target video with the face of a user from a source
video, while preserving the original target performance. Our system is fully
automatic and does not require a database of source expressions. Instead, it is
able to produce convincing reenactment results from a short source video
captured with an off-the-shelf camera, such as a webcam, where the user
performs arbitrary facial gestures. Our reenactment pipeline is conceived as
part image retrieval and part face transfer: The image retrieval is based on
temporal clustering of target frames and a novel image matching metric that
combines appearance and motion to select candidate frames from the source
video, while the face transfer uses a 2D warping strategy that preserves the
user's identity. Our system excels in simplicity as it does not rely on a 3D
face model, it is robust under head motion and does not require the source and
target performance to be similar. We show convincing reenactment results for
videos that we recorded ourselves and for low-quality footage taken from the
Internet.
