3D Reconstruction in the Presence of Glass and Mirrors by Acoustic and Visual Fusion

Yu Zhang, Mao Ye, Dinesh Manocha, Ruigang Yang

Research output: Contribution to journalArticlepeer-review

8 Scopus citations

Abstract

We present a practical and inexpensive method to reconstruct 3D scenes that include transparent and mirror objects. Our work is motivated by the need for automatically generating 3D models of interior scenes, which commonly include glass. These large structures are often invisible to cameras or even to our human visual system. Existing 3D reconstruction methods for transparent objects are usually not applicable in such a room-sized reconstruction setting. Our simple hardware setup augments a regular depth camera (e.g., the Microsoft Kinect camera) with a single ultrasonic sensor, which is able to measure the distance to any object, including transparent surfaces. The key technical challenge is the sparse sampling rate from the acoustic sensor, which only takes one point measurement per frame. To address this challenge, we take advantage of the fact that the large scale glass structures in indoor environments are usually either piece-wise planar or a simple parametric surface. Based on these assumptions, we have developed a novel sensor fusion algorithm that first segments the (hybrid) depth map into different categories such as opaque/transparent/infinity (e.g., too far to measure) and then updates the depth map based on the segmentation outcome. We validated our algorithms with a number of challenging cases, including multiple panes of glass, mirrors, and even a curved glass cabinet.

Original languageEnglish
Pages (from-to)1785-1798
Number of pages14
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume40
Issue number8
DOIs
StatePublished - Aug 1 2018

Bibliographical note

Funding Information:
3 10.1109/TPAMI.2017.2723883 28692965 0b00006485de81e0 Active orig-research F T F F F F F Publish 8 IEEE 0162-8828 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information. We present a practical and inexpensive method to reconstruct 3D scenes that include transparent and mirror objects. Our work is motivated by the need for automatically generating 3D models of interior scenes, which commonly include glass. These large structures are often invisible to cameras or even to our human visual system. Existing 3D reconstruction methods for transparent objects are usually not applicable in such a room-sized reconstruction setting. Our simple hardware setup augments a regular depth camera (e.g., the Microsoft Kinect camera) with a single ultrasonic sensor, which is able to measure the distance to any object, including transparent surfaces. The key technical challenge is the sparse sampling rate from the acoustic sensor, which only takes one point measurement per frame. To address this challenge, we take advantage of the fact that the large scale glass structures in indoor environments are usually either piece-wise planar or a simple parametric surface. Based on these assumptions, we have developed a novel sensor fusion algorithm that first segments the (hybrid) depth map into different categories such as opaque/transparent/infinity (e.g., too far to measure) and then updates the depth map based on the segmentation outcome. We validated our algorithms with a number of challenging cases, including multiple panes of glass, mirrors, and even a curved glass cabinet. 0 Zhang, Y. Yu Zhang Yu Yu Zhang Zhang School of Electronic Science and Engineering, Nanjing University, Nanjing, China Author zhangyu606@gmail.com 0 Ye, M. Mao Ye Mao Mao Ye Ye University of Kentucky, Lexington, KY Author mao.ye@uky.edu 0 0000-0001-7047-9801 Manocha, D. Dinesh Manocha Dinesh Dinesh Manocha Manocha University of Maryland, College Park, MD Author dm@cs.umd.edu 0 0000-0001-5296-6307 Yang, R. Ruigang Yang Ruigang Ruigang Yang Yang University of Kentucky, Lexington, KY Author ryang@cs.uky.edu 2018 Aug. 1 2017 7 6 2018 6 29 1523241 07970123.pdf 1-1 7970123 Three-dimensional displays Acoustics Glass Robot sensing systems Cameras Surface reconstruction Ultrasonic imaging 3D reconstruction sensor fusion ultrasonic range finding transparent/mirrored surface modeling US National Science Foundation IIS-1231545 IIP-1543172 Army Research W911NF-14-1-0437 Natural Science Foundation of China 61332017

Funding Information:
This work is supported in part by US National Science Foundation grants IIS-1231545, IIP-1543172, Army Research grant W911NF-14-1-0437 and Natural Science Foundation of China grant 61332017. Yu Zhang and Mao Ye contributed equally to this work.

Publisher Copyright:
© 1979-2012 IEEE.

Keywords

  • 3D reconstruction
  • sensor fusion
  • transparent/mirrored surface modeling
  • ultrasonic range finding

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint

Dive into the research topics of '3D Reconstruction in the Presence of Glass and Mirrors by Acoustic and Visual Fusion'. Together they form a unique fingerprint.

Cite this