Channel Attention Based Iterative Residual Learning for Depth Map Super-Resolution

Xibin Song; Yuchao Dai; Dingfu Zhou; Liu Liu; Wei Li; Hongdong Li; Ruigang Yang

doi:10.1109/CVPR42600.2020.00567

Channel Attention Based Iterative Residual Learning for Depth Map Super-Resolution

Xibin Song, Yuchao Dai, Dingfu Zhou, Liu Liu, Wei Li, Hongdong Li, Ruigang Yang

Computer Science

Research output: Contribution to journal › Conference article › peer-review

38 Scopus citations

Abstract

Despite the remarkable progresses made in deep learning based depth map super-resolution (DSR), how to tackle real-world degradation in low-resolution (LR) depth maps remains a major challenge. Existing DSR model is generally trained and tested on synthetic dataset, which is very different from what would get from a real depth sensor. In this paper, we argue that DSR models trained under this setting are restrictive and not effective in dealing with realworld DSR tasks. We make two contributions in tackling real-world degradation of different depth sensors. First, we propose to classify the generation of LR depth maps into two types: non-linear downsampling with noise and interval downsampling, for which DSR models are learned correspondingly. Second, we propose a new framework for real-world DSR, which consists of four modules : 1) An iterative residual learning module with deep supervision to learn effective high-frequency components of depth maps in a coarse-to-fine manner; 2) A channel attention strategy to enhance channels with abundant high-frequency components; 3) A multi-stage fusion module to effectively reexploit the results in the coarse-to-fine process; and 4) A depth refinement module to improve the depth map by TGV regularization and input loss. Extensive experiments on benchmarking datasets demonstrate the superiority of our method over current state-of-the-art DSR methods.

Original language	English
Article number	9156284
Pages (from-to)	5630-5639
Number of pages	10
Journal	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
DOIs	https://doi.org/10.1109/CVPR42600.2020.00567
State	Published - 2020
Event	2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020 - Virtual, Online, United States Duration: Jun 14 2020 → Jun 19 2020

Bibliographical note

Funding Information:
In this paper, we have proposed an effective depth map super-resolution method that accounts for real-world degradation processes of different types of physical depth sensors. We have envisaged the employment of our new method to super-resolve depth maps captured by commodity depth sensors such as Microsoft Kinect and Lidar. We analyze two different LR depth map simulation schemes: non-linear downsampling and interval downsampling. Furthermore, we have devised a channel attention based iterative residual learning framework to address real world depth map super-resolution. Extensive experiments across different benchmarks have demonstrated the superiority of our proposed approach over the state-of-the-art. Acknowledgment. The work is supported by Baidu Research. Yuchao Dai’s research is supported in part by the National Key Research and Development Program of China under Grant 2018AAA0102803 and Natural Science Foundation of China grants (61871325, 61420106007, 61671387), and Hongdong Li’s research is supported in part by the ARC Centre of Excellence for Robotics Vision (CE140100016) AND ARC-Discovery (DP 190102261), ARC-LIEF (190100080) grants. The authors of ANU gratefully acknowledge the GPUs donated by NVIDIA Corporation. We thank all anonymous reviewers and ACs for their constructive comments.

Publisher Copyright:
© 2020 IEEE.

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/CVPR42600.2020.00567

Cite this

@article{8530a952189b41c59c91e8a8aac9150a,

title = "Channel Attention Based Iterative Residual Learning for Depth Map Super-Resolution",

abstract = "Despite the remarkable progresses made in deep learning based depth map super-resolution (DSR), how to tackle real-world degradation in low-resolution (LR) depth maps remains a major challenge. Existing DSR model is generally trained and tested on synthetic dataset, which is very different from what would get from a real depth sensor. In this paper, we argue that DSR models trained under this setting are restrictive and not effective in dealing with realworld DSR tasks. We make two contributions in tackling real-world degradation of different depth sensors. First, we propose to classify the generation of LR depth maps into two types: non-linear downsampling with noise and interval downsampling, for which DSR models are learned correspondingly. Second, we propose a new framework for real-world DSR, which consists of four modules : 1) An iterative residual learning module with deep supervision to learn effective high-frequency components of depth maps in a coarse-to-fine manner; 2) A channel attention strategy to enhance channels with abundant high-frequency components; 3) A multi-stage fusion module to effectively reexploit the results in the coarse-to-fine process; and 4) A depth refinement module to improve the depth map by TGV regularization and input loss. Extensive experiments on benchmarking datasets demonstrate the superiority of our method over current state-of-the-art DSR methods.",

author = "Xibin Song and Yuchao Dai and Dingfu Zhou and Liu Liu and Wei Li and Hongdong Li and Ruigang Yang",

note = "Funding Information: In this paper, we have proposed an effective depth map super-resolution method that accounts for real-world degradation processes of different types of physical depth sensors. We have envisaged the employment of our new method to super-resolve depth maps captured by commodity depth sensors such as Microsoft Kinect and Lidar. We analyze two different LR depth map simulation schemes: non-linear downsampling and interval downsampling. Furthermore, we have devised a channel attention based iterative residual learning framework to address real world depth map super-resolution. Extensive experiments across different benchmarks have demonstrated the superiority of our proposed approach over the state-of-the-art. Acknowledgment. The work is supported by Baidu Research. Yuchao Dai{\textquoteright}s research is supported in part by the National Key Research and Development Program of China under Grant 2018AAA0102803 and Natural Science Foundation of China grants (61871325, 61420106007, 61671387), and Hongdong Li{\textquoteright}s research is supported in part by the ARC Centre of Excellence for Robotics Vision (CE140100016) AND ARC-Discovery (DP 190102261), ARC-LIEF (190100080) grants. The authors of ANU gratefully acknowledge the GPUs donated by NVIDIA Corporation. We thank all anonymous reviewers and ACs for their constructive comments. Publisher Copyright: {\textcopyright} 2020 IEEE.; null ; Conference date: 14-06-2020 Through 19-06-2020",

year = "2020",

doi = "10.1109/CVPR42600.2020.00567",

language = "English",

pages = "5630--5639",

}

TY - JOUR

T1 - Channel Attention Based Iterative Residual Learning for Depth Map Super-Resolution

AU - Song, Xibin

AU - Dai, Yuchao

AU - Zhou, Dingfu

AU - Liu, Liu

AU - Li, Wei

AU - Li, Hongdong

AU - Yang, Ruigang

N1 - Funding Information: In this paper, we have proposed an effective depth map super-resolution method that accounts for real-world degradation processes of different types of physical depth sensors. We have envisaged the employment of our new method to super-resolve depth maps captured by commodity depth sensors such as Microsoft Kinect and Lidar. We analyze two different LR depth map simulation schemes: non-linear downsampling and interval downsampling. Furthermore, we have devised a channel attention based iterative residual learning framework to address real world depth map super-resolution. Extensive experiments across different benchmarks have demonstrated the superiority of our proposed approach over the state-of-the-art. Acknowledgment. The work is supported by Baidu Research. Yuchao Dai’s research is supported in part by the National Key Research and Development Program of China under Grant 2018AAA0102803 and Natural Science Foundation of China grants (61871325, 61420106007, 61671387), and Hongdong Li’s research is supported in part by the ARC Centre of Excellence for Robotics Vision (CE140100016) AND ARC-Discovery (DP 190102261), ARC-LIEF (190100080) grants. The authors of ANU gratefully acknowledge the GPUs donated by NVIDIA Corporation. We thank all anonymous reviewers and ACs for their constructive comments. Publisher Copyright: © 2020 IEEE.

PY - 2020

Y1 - 2020

N2 - Despite the remarkable progresses made in deep learning based depth map super-resolution (DSR), how to tackle real-world degradation in low-resolution (LR) depth maps remains a major challenge. Existing DSR model is generally trained and tested on synthetic dataset, which is very different from what would get from a real depth sensor. In this paper, we argue that DSR models trained under this setting are restrictive and not effective in dealing with realworld DSR tasks. We make two contributions in tackling real-world degradation of different depth sensors. First, we propose to classify the generation of LR depth maps into two types: non-linear downsampling with noise and interval downsampling, for which DSR models are learned correspondingly. Second, we propose a new framework for real-world DSR, which consists of four modules : 1) An iterative residual learning module with deep supervision to learn effective high-frequency components of depth maps in a coarse-to-fine manner; 2) A channel attention strategy to enhance channels with abundant high-frequency components; 3) A multi-stage fusion module to effectively reexploit the results in the coarse-to-fine process; and 4) A depth refinement module to improve the depth map by TGV regularization and input loss. Extensive experiments on benchmarking datasets demonstrate the superiority of our method over current state-of-the-art DSR methods.

AB - Despite the remarkable progresses made in deep learning based depth map super-resolution (DSR), how to tackle real-world degradation in low-resolution (LR) depth maps remains a major challenge. Existing DSR model is generally trained and tested on synthetic dataset, which is very different from what would get from a real depth sensor. In this paper, we argue that DSR models trained under this setting are restrictive and not effective in dealing with realworld DSR tasks. We make two contributions in tackling real-world degradation of different depth sensors. First, we propose to classify the generation of LR depth maps into two types: non-linear downsampling with noise and interval downsampling, for which DSR models are learned correspondingly. Second, we propose a new framework for real-world DSR, which consists of four modules : 1) An iterative residual learning module with deep supervision to learn effective high-frequency components of depth maps in a coarse-to-fine manner; 2) A channel attention strategy to enhance channels with abundant high-frequency components; 3) A multi-stage fusion module to effectively reexploit the results in the coarse-to-fine process; and 4) A depth refinement module to improve the depth map by TGV regularization and input loss. Extensive experiments on benchmarking datasets demonstrate the superiority of our method over current state-of-the-art DSR methods.

UR - http://www.scopus.com/inward/record.url?scp=85094862015&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85094862015&partnerID=8YFLogxK

U2 - 10.1109/CVPR42600.2020.00567

DO - 10.1109/CVPR42600.2020.00567

M3 - Conference article

AN - SCOPUS:85094862015

SP - 5630

EP - 5639

M1 - 9156284

Y2 - 14 June 2020 through 19 June 2020

ER -

Channel Attention Based Iterative Residual Learning for Depth Map Super-Resolution

Abstract

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this