We present a novel deep learning approach to synthesize complete face images in the presence of large ocular region occlusions. This is motivated by recent surge of VR/AR displays that hinder face-to-face communications. Different from the state-of-the-art face inpainting methods that have no control over the synthesized content and can only handle frontal face pose, our approach can faithfully recover the missing content under various head poses while preserving the identity. At the core of our method is a novel generative network with dedicated constraints to regularize the synthesis process. To preserve the identity, our network takes an arbitrary occlusion-free image of the target identity to infer the missing content, and its high-level CNN features as an identity prior to regularize the searching space of generator. Since the input reference image may have a different pose, a pose map and a novel pose discriminator are further adopted to supervise the learning of implicit pose transformations. Our method is capable of generating coherent facial inpainting with consistent identity over videos with large variations of head motions. Experiments on both synthesized and real data demonstrate that our method greatly outperforms the state-of-the-art methods in terms of both synthesis quality and robustness.
|State||Published - 2019|
|Event||29th British Machine Vision Conference, BMVC 2018 - Newcastle, United Kingdom|
Duration: Sep 3 2018 → Sep 6 2018
|Conference||29th British Machine Vision Conference, BMVC 2018|
|Period||9/3/18 → 9/6/18|
Bibliographical noteFunding Information:
We gratefully acknowledge the support of NSFC grant(61502133). We would like to show our gratitude to Nathan Jacobs, Xinyu Huang for sharing their pearls of wisdom with us during the course of this research, and we thank Qingguo Xu for assistance with experiments and comments that greatly improved the manuscript.
© 2018. The copyright of this document resides with its authors.
ASJC Scopus subject areas
- Computer Vision and Pattern Recognition