High-speed video has been commonly adopted in consumergrade cameras, augmenting these videos with a corresponding depth stream will enable new multimedia applications, such as 3D slow-motion video. In this paper, we present a hybrid camera system that combines a high-speed color camera with a depth sensor, e.g. Kinect depth sensor, to generate a depth stream that can produce both high-speed and high-resolution RGB+depth stream. Simply interpolating the low-speed depth frames is not satisfactory, where interpolation artifacts and lose in surface details are often visible. We have developed a novel framework that utilizes both shading constraints within each frame and optical ow constraints between neighboring frames. More specifically we present (a) an effective method to find the intrinsics images to allow more accurate normal estimation; and (b) an optimization-based framework to estimate the high-resolution/high-speed depth stream, taking into consideration temporal smoothness and shading/depth consistency. We evaluated our holistic framework with both synthetic and real sequences, it showed superior performance than previous state-of-the-art.
|Title of host publication||MM 2016 - Proceedings of the 2016 ACM Multimedia Conference|
|Number of pages||10|
|State||Published - Oct 1 2016|
|Event||24th ACM Multimedia Conference, MM 2016 - Amsterdam, United Kingdom|
Duration: Oct 15 2016 → Oct 19 2016
|Name||MM 2016 - Proceedings of the 2016 ACM Multimedia Conference|
|Conference||24th ACM Multimedia Conference, MM 2016|
|Period||10/15/16 → 10/19/16|
Bibliographical noteFunding Information:
The work is supported in part by US National Science Foundation grants IIS-1231545 and IIP-1543172, Natural Science Foundation of China (No.61332017, 61572243), Key Industrial Innovation Chain of Shaanxi Province Industrial Area (2015KTZDGY04-01), Chinese Scholarship Council, and Zhejiang Provincial Natural Science Foundation grant LY13F020019. Jiangbin Zheng and Ruigang Yang are the co-corresponding authors for this paper.
© 2016 ACM.
- Depth stream
- High-speed imaging
- Intrinisic decomposition
- Shape from shading
ASJC Scopus subject areas
- Computer Graphics and Computer-Aided Design
- Human-Computer Interaction
- Computer Vision and Pattern Recognition