Understanding the Generalization Power of Overfitted NTK Models: 3-layer vs. 2-layer (Extended Abstract)

Peizhong Ju; Xiaojun Lin; Ness B. Shroff

doi:10.1109/Allerton49937.2022.9929374

Understanding the Generalization Power of Overfitted NTK Models: 3-layer vs. 2-layer (Extended Abstract)

Peizhong Ju, Xiaojun Lin, Ness B. Shroff

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Neural tangent kernel (NTK) models [1] have been recently used as an important intermediate step to understand the exceptional generalization power of overparameterized deep neural networks (DNNs). Compared to linear models with simple Gaussian or Fourier features, NTK models can capture the nonlinear features inherent in neural networks. Indeed, the work in [2] has shown that, for a 2-layer NTK model, the generalization error of an overfitted solution decreases as the number of neurons increases. Further, this descent behavior is qualitatively different from that of linear models with simple Gaussian and Fourier features, and closer to that of an actual neural network.

Original language	English
Title of host publication	2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022
ISBN (Electronic)	9798350399981
DOIs	https://doi.org/10.1109/Allerton49937.2022.9929374
State	Published - 2022
Event	58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022 - Monticello, United States Duration: Sep 27 2022 → Sep 30 2022

Publication series

Name	2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022

Conference

Conference	58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022
Country/Territory	United States
City	Monticello
Period	9/27/22 → 9/30/22

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Computer Science Applications
Computer Vision and Pattern Recognition
Signal Processing
Control and Optimization

Access to Document

10.1109/Allerton49937.2022.9929374

Cite this

Ju, P., Lin, X., & Shroff, N. B. (2022). Understanding the Generalization Power of Overfitted NTK Models: 3-layer vs. 2-layer (Extended Abstract). In 2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022 (2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022). https://doi.org/10.1109/Allerton49937.2022.9929374

@inproceedings{ec18ef5563a440769e6f74c65d3ca255,

title = "Understanding the Generalization Power of Overfitted NTK Models: 3-layer vs. 2-layer (Extended Abstract)",

abstract = "Neural tangent kernel (NTK) models [1] have been recently used as an important intermediate step to understand the exceptional generalization power of overparameterized deep neural networks (DNNs). Compared to linear models with simple Gaussian or Fourier features, NTK models can capture the nonlinear features inherent in neural networks. Indeed, the work in [2] has shown that, for a 2-layer NTK model, the generalization error of an overfitted solution decreases as the number of neurons increases. Further, this descent behavior is qualitatively different from that of linear models with simple Gaussian and Fourier features, and closer to that of an actual neural network.",

author = "Peizhong Ju and Xiaojun Lin and Shroff, {Ness B.}",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022 ; Conference date: 27-09-2022 Through 30-09-2022",

year = "2022",

doi = "10.1109/Allerton49937.2022.9929374",

language = "English",

series = "2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022",

booktitle = "2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022",

}

Ju, P, Lin, X & Shroff, NB 2022, Understanding the Generalization Power of Overfitted NTK Models: 3-layer vs. 2-layer (Extended Abstract). in 2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022. 2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022, 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022, Monticello, United States, 9/27/22. https://doi.org/10.1109/Allerton49937.2022.9929374

Understanding the Generalization Power of Overfitted NTK Models: 3-layer vs. 2-layer (Extended Abstract). / Ju, Peizhong; Lin, Xiaojun; Shroff, Ness B.
2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022. 2022. (2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Understanding the Generalization Power of Overfitted NTK Models

T2 - 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022

AU - Ju, Peizhong

AU - Lin, Xiaojun

AU - Shroff, Ness B.

PY - 2022

Y1 - 2022

N2 - Neural tangent kernel (NTK) models [1] have been recently used as an important intermediate step to understand the exceptional generalization power of overparameterized deep neural networks (DNNs). Compared to linear models with simple Gaussian or Fourier features, NTK models can capture the nonlinear features inherent in neural networks. Indeed, the work in [2] has shown that, for a 2-layer NTK model, the generalization error of an overfitted solution decreases as the number of neurons increases. Further, this descent behavior is qualitatively different from that of linear models with simple Gaussian and Fourier features, and closer to that of an actual neural network.

AB - Neural tangent kernel (NTK) models [1] have been recently used as an important intermediate step to understand the exceptional generalization power of overparameterized deep neural networks (DNNs). Compared to linear models with simple Gaussian or Fourier features, NTK models can capture the nonlinear features inherent in neural networks. Indeed, the work in [2] has shown that, for a 2-layer NTK model, the generalization error of an overfitted solution decreases as the number of neurons increases. Further, this descent behavior is qualitatively different from that of linear models with simple Gaussian and Fourier features, and closer to that of an actual neural network.

UR - http://www.scopus.com/inward/record.url?scp=85142657681&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85142657681&partnerID=8YFLogxK

U2 - 10.1109/Allerton49937.2022.9929374

DO - 10.1109/Allerton49937.2022.9929374

M3 - Conference contribution

AN - SCOPUS:85142657681

T3 - 2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022

BT - 2022 58th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2022

Y2 - 27 September 2022 through 30 September 2022

ER -

Understanding the Generalization Power of Overfitted NTK Models: 3-layer vs. 2-layer (Extended Abstract)

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this