Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum

Bao Wang, Qiang Ye

Research output: Contribution to journalArticlepeer-review


Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from <inline-formula> <tex-math notation="LaTeX">$5.25\%$</tex-math> </inline-formula> to <inline-formula> <tex-math notation="LaTeX">$4.64\%$</tex-math> </inline-formula> and <inline-formula> <tex-math notation="LaTeX">$23.75\%$</tex-math> </inline-formula> to <inline-formula> <tex-math notation="LaTeX">$20.03\%$</tex-math> </inline-formula>, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.

Original languageEnglish
Pages (from-to)1-13
Number of pages13
JournalIEEE Transactions on Neural Networks and Learning Systems
StateAccepted/In press - 2023

Bibliographical note

Publisher Copyright:


  • Adaptive momentum
  • Classification algorithms
  • Convergence
  • Deep learning
  • Image classification
  • Robustness
  • Stochastic processes
  • Training
  • deep learning
  • image classification
  • nonlinear conjugate gradient (NCG)

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence


Dive into the research topics of 'Improving Deep Neural Networks&#x2019; Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum'. Together they form a unique fingerprint.

Cite this