Fully automatic photorealistic facial expression and eye gaze transfer with a single image

Facial expression transfer has many applications in animation, special effects, and social media. Due to the variability in pose and environmental condition, generating a photo-realistic facial expression with a single image under uncon-trolled environment is a challenging problem. In this paper, we present a novel approach to simultaneously transfer both the facial expression and the eye gaze of an actor from one image to another. Based on the estimated facial geometry, the key idea of our work is to use mesh deformation to manipulate the expression of the actor in the target image. We start the process by reconstructing 3D facial geometry from un-constrained 2D images using a two-stage optimization strategy. A non-rigid registration is then used to find correspon-dences and alignment between reconstructed models, while eye gaze transfer is accomplished by piecewise affine warping. Finally we re-render the manipulated image with the use of Poisson image blending. We demonstrate the effectiveness of our method on images with various lighting conditions, head poses, and facial expressions.