Omnidirectional Depth Extension Networks

Xinjing Cheng, Peng Wang, Yanqi Zhou, Chenye Guan, Ruigang Yang

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

15 Scopus citations


Omnidirectional 360° camera proliferates rapidly for autonomous robots since it significantly enhances the perception ability by widening the field of view (FoV). However, corresponding 360° depth sensors, which are also critical for the perception system, are still difficult or expensive to have. In this paper, we propose a low-cost 3D sensing system that combines an omnidirectional camera with a calibrated projective depth camera, where the depth from the limited FoV can be automatically extended to the rest of recorded omnidirectional image. To accurately recover the missing depths, we design an omnidirectional depth extension convolutional neural network (ODE-CNN), in which a spherical feature transform layer (SFTL) is embedded at the end of feature encoding layers, and a deformable convolutional spatial propagation network (D-CSPN) is appended at the end of feature decoding layers. The former re-samples the neighborhood of each pixel in the omnidirectional coordination to the projective coordination, which reduce the difficulty of feature learning, and the later automatically finds a proper context to well align the structures in the estimated depths via CNN w.r.t. the reference image, which significantly improves the visual quality. Finally, we demonstrate the effectiveness of proposed ODE-CNN over the popular 360D dataset, and show that ODE-CNN significantly outperforms (relatively 33% reduction in depth error) other state-of-the-art (SoTA) methods.

Original languageEnglish
Title of host publication2020 IEEE International Conference on Robotics and Automation, ICRA 2020
Number of pages7
ISBN (Electronic)9781728173955
StatePublished - May 2020
Event2020 IEEE International Conference on Robotics and Automation, ICRA 2020 - Paris, France
Duration: May 31 2020Aug 31 2020

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
ISSN (Print)1050-4729


Conference2020 IEEE International Conference on Robotics and Automation, ICRA 2020

Bibliographical note

Publisher Copyright:
© 2020 IEEE.

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Electrical and Electronic Engineering
  • Control and Systems Engineering


Dive into the research topics of 'Omnidirectional Depth Extension Networks'. Together they form a unique fingerprint.

Cite this