High-speed depth stream generation from a hybrid camera

Xinxin Zuo, Sen Wang, Jiangbin Zheng, Ruigang Yang

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

3 Scopus citations

Abstract

High-speed video has been commonly adopted in consumergrade cameras, augmenting these videos with a corresponding depth stream will enable new multimedia applications, such as 3D slow-motion video. In this paper, we present a hybrid camera system that combines a high-speed color camera with a depth sensor, e.g. Kinect depth sensor, to generate a depth stream that can produce both high-speed and high-resolution RGB+depth stream. Simply interpolating the low-speed depth frames is not satisfactory, where interpolation artifacts and lose in surface details are often visible. We have developed a novel framework that utilizes both shading constraints within each frame and optical ow constraints between neighboring frames. More specifically we present (a) an effective method to find the intrinsics images to allow more accurate normal estimation; and (b) an optimization-based framework to estimate the high-resolution/high-speed depth stream, taking into consideration temporal smoothness and shading/depth consistency. We evaluated our holistic framework with both synthetic and real sequences, it showed superior performance than previous state-of-the-art.

Original languageEnglish
Title of host publicationMM 2016 - Proceedings of the 2016 ACM Multimedia Conference
Pages878-887
Number of pages10
ISBN (Electronic)9781450336031
DOIs
StatePublished - Oct 1 2016
Event24th ACM Multimedia Conference, MM 2016 - Amsterdam, United Kingdom
Duration: Oct 15 2016Oct 19 2016

Publication series

NameMM 2016 - Proceedings of the 2016 ACM Multimedia Conference

Conference

Conference24th ACM Multimedia Conference, MM 2016
Country/TerritoryUnited Kingdom
CityAmsterdam
Period10/15/1610/19/16

Bibliographical note

Funding Information:
The work is supported in part by US National Science Foundation grants IIS-1231545 and IIP-1543172, Natural Science Foundation of China (No.61332017, 61572243), Key Industrial Innovation Chain of Shaanxi Province Industrial Area (2015KTZDGY04-01), Chinese Scholarship Council, and Zhejiang Provincial Natural Science Foundation grant LY13F020019. Jiangbin Zheng and Ruigang Yang are the co-corresponding authors for this paper.

Publisher Copyright:
© 2016 ACM.

Keywords

  • Depth stream
  • High-speed imaging
  • Intrinisic decomposition
  • Shape from shading

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Software

Fingerprint

Dive into the research topics of 'High-speed depth stream generation from a hybrid camera'. Together they form a unique fingerprint.

Cite this