Wearable VR/AR devices provide users with fully immersive experience in a virtual environment, enabling possibilities to reshape the forms of entertainment and telepresence. While the body language is a crucial element in effective communication, wearing a head-mounted display (HMD) could severely hinder the eye contact and block facial expressions. We present a novel headset removal technique that enables high-quality occlusion-free communication in virtual environment. In particular, our solution synthesizes photoreal faces in the occluded region with faithful reconstruction of facial expressions and eye movements. Towards this goal, we develop a novel capture setup that consists of two near-infrared (NIR) cameras inside the HMD for eye capturing and one external RGB camera for recording visible face regions. To enable realistic face synthesis with consistent illuminations, we propose a data-driven approach to fuse the narrow-field-of-view NIR images with the RGB image captured from the external camera. In addition, to generate pho-torealistic eyes, a dedicated algorithm is proposed to colorize the NIR eye images and further rectify the color distortion caused by the non-linear mapping of IR light sensitivity. Experimental results demonstrate that our framework is capable to synthesize high-fidelity unoccluded facial images with accurate tracking of head motion, facial expression and eye movement.
|Title of host publication||26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Proceedings|
|Number of pages||10|
|State||Published - Mar 2019|
|Event||26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Osaka, Japan|
Duration: Mar 23 2019 → Mar 27 2019
|Name||26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Proceedings|
|Conference||26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019|
|Period||3/23/19 → 3/27/19|
Bibliographical noteFunding Information:
This work is partially supported by the US NFS (IIS-1231545, IIP-1543172), NSFC (No. 61332017, 61305011), National High Technology Research and Development Program of China (No. 2015AA015905).
© 2019 IEEE.
Copyright 2019 Elsevier B.V., All rights reserved.
- Eye synthesis
- Face inpainting
- Headset removal
ASJC Scopus subject areas
- Human-Computer Interaction
- Media Technology