Estimating pose and illumination direction for frontal face synthesis

Face pose and illumination estimation is an important pre-processing step in many face analysis problems. In this paper, we present a new method to estimate the face pose and illumination direction from one single image. The basic idea is to compare the reconstruction residuals between the input image and a small set of reference images under different poses and illumination directions. Based on the estimated pose and illumination direction, we develop a face synthesis framework to rectify the input image to the frontal view under standard illumination. Experiments show that our estimation method is both fast (less than one second per frame) and accurate (even less than three degrees) and our face synthesis method can generate visually plausible results, in particular for challenging inputs with with large pose changes and poor lighting conditions. The synthesized frontal face views increase the face recognition rate significantly from 1.5% to 62.1%.