In this paper, we propose a novel approach to convert given speech audio to a photo-realistic speaking video of a specific person, where the output video has synchronized, realistic, and expressive rich body dynamics. We achieve this by first generating 3D skeleton movements from the audio sequence using a recurrent neural network (RNN), and then synthesizing the output video via a conditional generative adversarial network (GAN). To make the skeleton movement realistic and expressive, we embed the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic gestures into the generation process in both learning and testing pipelines. The former prevents the generation of unreasonable body distortion, while the later helps our model quickly learn meaningful body movement through a few recorded videos. To produce photo-realistic and high-resolution video with motion details, we propose to insert part attention mechanisms in the conditional GAN, where each detailed part, e.g. head and hand, is automatically zoomed in to have their own discriminators. To validate our approach, we collect a dataset with 20 high-quality videos from 1 male and 1 female model reading various documents under different topics. Compared with previous SoTA pipelines handling similar tasks, our approach achieves better results by a user study.
|Title of host publication||Computer Vision – ACCV 2020 - 15th Asian Conference on Computer Vision, 2020, Revised Selected Papers|
|Editors||Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, Jianbo Shi|
|Number of pages||16|
|State||Published - 2021|
|Event||15th Asian Conference on Computer Vision, ACCV 2020 - Virtual, Online|
Duration: Nov 30 2020 → Dec 4 2020
|Name||Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|
|Conference||15th Asian Conference on Computer Vision, ACCV 2020|
|Period||11/30/20 → 12/4/20|
Bibliographical noteFunding Information:
This work was supported by the NSFC grant No. 62001213.
This work was supported by the NFC grant No. 62001213.
© 2021, Springer Nature Switzerland AG.
- Body pose
- Image and video synthesis
- Vision and language
ASJC Scopus subject areas
- Theoretical Computer Science
- Computer Science (all)