Improved techniques for training adaptive deep networks

Hao Li; Hong Zhang; Xiaojuan Qi; Yang Ruigang; Gao Huang

doi:10.1109/ICCV.2019.00198

Improved techniques for training adaptive deep networks

Hao Li, Hong Zhang, Xiaojuan Qi, Yang Ruigang, Gao Huang

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

60 Scopus citations

Abstract

Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.

Original language	English
Title of host publication	Proceedings - 2019 International Conference on Computer Vision, ICCV 2019
Pages	1891-1900
Number of pages	10
ISBN (Electronic)	9781728148038
DOIs	https://doi.org/10.1109/ICCV.2019.00198
State	Published - Oct 2019
Event	17th IEEE/CVF International Conference on Computer Vision, ICCV 2019 - Seoul, Korea, Republic of Duration: Oct 27 2019 → Nov 2 2019

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
Volume	2019-October
ISSN (Print)	1550-5499

Conference

Conference	17th IEEE/CVF International Conference on Computer Vision, ICCV 2019
Country/Territory	Korea, Republic of
City	Seoul
Period	10/27/19 → 11/2/19

Bibliographical note

Funding Information:
Acknowledgements. Gao Huang is supported in part by Beijing Academy of Artificial Intelligence under grant BAAI2019QN0106. Hao Li is supported in part by Ts-inghua University Initiative Scientific Research Program and Tsinghua Academic Fund for Undergraduate Overseas Studies. We thank Danlu Chen for helpful discussions.

Publisher Copyright:
© 2019 IEEE.

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/ICCV.2019.00198

Cite this

@inproceedings{48d3d40690624d78b826162b17a90553,

title = "Improved techniques for training adaptive deep networks",

abstract = "Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.",

author = "Hao Li and Hong Zhang and Xiaojuan Qi and Yang Ruigang and Gao Huang",

note = "Funding Information: Acknowledgements. Gao Huang is supported in part by Beijing Academy of Artificial Intelligence under grant BAAI2019QN0106. Hao Li is supported in part by Ts-inghua University Initiative Scientific Research Program and Tsinghua Academic Fund for Undergraduate Overseas Studies. We thank Danlu Chen for helpful discussions. Publisher Copyright: {\textcopyright} 2019 IEEE.; 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019 ; Conference date: 27-10-2019 Through 02-11-2019",

year = "2019",

month = oct,

doi = "10.1109/ICCV.2019.00198",

language = "English",

series = "Proceedings of the IEEE International Conference on Computer Vision",

pages = "1891--1900",

booktitle = "Proceedings - 2019 International Conference on Computer Vision, ICCV 2019",

}

Li, H, Zhang, H, Qi, X, Ruigang, Y & Huang, G 2019, Improved techniques for training adaptive deep networks. in Proceedings - 2019 International Conference on Computer Vision, ICCV 2019., 9010043, Proceedings of the IEEE International Conference on Computer Vision, vol. 2019-October, pp. 1891-1900, 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea, Republic of, 10/27/19. https://doi.org/10.1109/ICCV.2019.00198

Improved techniques for training adaptive deep networks. / Li, Hao; Zhang, Hong; Qi, Xiaojuan et al.
Proceedings - 2019 International Conference on Computer Vision, ICCV 2019. 2019. p. 1891-1900 9010043 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2019-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Improved techniques for training adaptive deep networks

AU - Li, Hao

AU - Zhang, Hong

AU - Qi, Xiaojuan

AU - Ruigang, Yang

AU - Huang, Gao

N1 - Funding Information: Acknowledgements. Gao Huang is supported in part by Beijing Academy of Artificial Intelligence under grant BAAI2019QN0106. Hao Li is supported in part by Ts-inghua University Initiative Scientific Research Program and Tsinghua Academic Fund for Undergraduate Overseas Studies. We thank Danlu Chen for helpful discussions. Publisher Copyright: © 2019 IEEE.

PY - 2019/10

Y1 - 2019/10

N2 - Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.

AB - Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.

UR - http://www.scopus.com/inward/record.url?scp=85081925431&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85081925431&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2019.00198

DO - 10.1109/ICCV.2019.00198

M3 - Conference contribution

AN - SCOPUS:85081925431

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 1891

EP - 1900

BT - Proceedings - 2019 International Conference on Computer Vision, ICCV 2019

T2 - 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019

Y2 - 27 October 2019 through 2 November 2019

ER -

Improved techniques for training adaptive deep networks

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this