Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis

Xin Xing; Gongbo Liang; Yu Zhang; Subash Khanal; Ai Ling Lin; Nathan Jacobs

doi:10.1109/ISBI52829.2022.9761584

Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis

Xin Xing, Gongbo Liang, Yu Zhang, Subash Khanal, Ai Ling Lin, Nathan Jacobs

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

26 Scopus citations

Abstract

We present a new model trained on multi-modalities of Positron Emission Tomography images (PET-AV45 and PET-FDG) for Alzheimer's Disease (AD) diagnosis. Unlike the conventional methods using multi-modal 3D/2D CNN architecture, our design replaces the Convolutional Neural Net-work (CNN) by Vision Transformer (ViT). Considering the high computation cost of 3D images, we firstly employ a 3D-to-2D operation to project the 3D PET images into 2D fusion images. Then, we forward the fused multi-modal 2D images to a parallel ViT model for feature extraction, followed by classification for AD diagnosis. For evaluation, we use PET images from ADNI. The proposed model outperforms several strong baseline models in our experiments and achieves 0.91 accuracy and 0.95 AUC.

Original language	English
Title of host publication	ISBI 2022 - Proceedings
Subtitle of host publication	2022 IEEE International Symposium on Biomedical Imaging
ISBN (Electronic)	9781665429238
DOIs	https://doi.org/10.1109/ISBI52829.2022.9761584
State	Published - 2022
Event	19th IEEE International Symposium on Biomedical Imaging, ISBI 2022 - Kolkata, India Duration: Mar 28 2022 → Mar 31 2022

Publication series

Name	Proceedings - International Symposium on Biomedical Imaging
Volume	2022-March
ISSN (Print)	1945-7928
ISSN (Electronic)	1945-8452

Conference

Conference	19th IEEE International Symposium on Biomedical Imaging, ISBI 2022
Country/Territory	India
City	Kolkata
Period	3/28/22 → 3/31/22

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

Keywords

Alzheimer's Disease
Multi-modalities
PET image
Vision Transformer (ViT)

ASJC Scopus subject areas

Biomedical Engineering
Radiology Nuclear Medicine and imaging

Access to Document

10.1109/ISBI52829.2022.9761584

Cite this

Xing, X., Liang, G., Zhang, Y., Khanal, S., Lin, A. L., & Jacobs, N. (2022). Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis. In ISBI 2022 - Proceedings: 2022 IEEE International Symposium on Biomedical Imaging (Proceedings - International Symposium on Biomedical Imaging; Vol. 2022-March). https://doi.org/10.1109/ISBI52829.2022.9761584

@inproceedings{69227dfbf1a845f9a6f0a7efa18491a8,

title = "Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis",

abstract = "We present a new model trained on multi-modalities of Positron Emission Tomography images (PET-AV45 and PET-FDG) for Alzheimer's Disease (AD) diagnosis. Unlike the conventional methods using multi-modal 3D/2D CNN architecture, our design replaces the Convolutional Neural Net-work (CNN) by Vision Transformer (ViT). Considering the high computation cost of 3D images, we firstly employ a 3D-to-2D operation to project the 3D PET images into 2D fusion images. Then, we forward the fused multi-modal 2D images to a parallel ViT model for feature extraction, followed by classification for AD diagnosis. For evaluation, we use PET images from ADNI. The proposed model outperforms several strong baseline models in our experiments and achieves 0.91 accuracy and 0.95 AUC.",

keywords = "Alzheimer's Disease, Multi-modalities, PET image, Vision Transformer (ViT)",

author = "Xin Xing and Gongbo Liang and Yu Zhang and Subash Khanal and Lin, {Ai Ling} and Nathan Jacobs",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 19th IEEE International Symposium on Biomedical Imaging, ISBI 2022 ; Conference date: 28-03-2022 Through 31-03-2022",

year = "2022",

doi = "10.1109/ISBI52829.2022.9761584",

language = "English",

series = "Proceedings - International Symposium on Biomedical Imaging",

booktitle = "ISBI 2022 - Proceedings",

}

Xing, X, Liang, G, Zhang, Y, Khanal, S, Lin, AL & Jacobs, N 2022, Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis. in ISBI 2022 - Proceedings: 2022 IEEE International Symposium on Biomedical Imaging. Proceedings - International Symposium on Biomedical Imaging, vol. 2022-March, 19th IEEE International Symposium on Biomedical Imaging, ISBI 2022, Kolkata, India, 3/28/22. https://doi.org/10.1109/ISBI52829.2022.9761584

Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis. / Xing, Xin; Liang, Gongbo; Zhang, Yu et al.
ISBI 2022 - Proceedings: 2022 IEEE International Symposium on Biomedical Imaging. 2022. (Proceedings - International Symposium on Biomedical Imaging; Vol. 2022-March).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Advit

T2 - 19th IEEE International Symposium on Biomedical Imaging, ISBI 2022

AU - Xing, Xin

AU - Liang, Gongbo

AU - Zhang, Yu

AU - Khanal, Subash

AU - Lin, Ai Ling

AU - Jacobs, Nathan

PY - 2022

Y1 - 2022

N2 - We present a new model trained on multi-modalities of Positron Emission Tomography images (PET-AV45 and PET-FDG) for Alzheimer's Disease (AD) diagnosis. Unlike the conventional methods using multi-modal 3D/2D CNN architecture, our design replaces the Convolutional Neural Net-work (CNN) by Vision Transformer (ViT). Considering the high computation cost of 3D images, we firstly employ a 3D-to-2D operation to project the 3D PET images into 2D fusion images. Then, we forward the fused multi-modal 2D images to a parallel ViT model for feature extraction, followed by classification for AD diagnosis. For evaluation, we use PET images from ADNI. The proposed model outperforms several strong baseline models in our experiments and achieves 0.91 accuracy and 0.95 AUC.

AB - We present a new model trained on multi-modalities of Positron Emission Tomography images (PET-AV45 and PET-FDG) for Alzheimer's Disease (AD) diagnosis. Unlike the conventional methods using multi-modal 3D/2D CNN architecture, our design replaces the Convolutional Neural Net-work (CNN) by Vision Transformer (ViT). Considering the high computation cost of 3D images, we firstly employ a 3D-to-2D operation to project the 3D PET images into 2D fusion images. Then, we forward the fused multi-modal 2D images to a parallel ViT model for feature extraction, followed by classification for AD diagnosis. For evaluation, we use PET images from ADNI. The proposed model outperforms several strong baseline models in our experiments and achieves 0.91 accuracy and 0.95 AUC.

KW - Alzheimer's Disease

KW - Multi-modalities

KW - PET image

KW - Vision Transformer (ViT)

UR - http://www.scopus.com/inward/record.url?scp=85129689131&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85129689131&partnerID=8YFLogxK

U2 - 10.1109/ISBI52829.2022.9761584

DO - 10.1109/ISBI52829.2022.9761584

M3 - Conference contribution

AN - SCOPUS:85129689131

T3 - Proceedings - International Symposium on Biomedical Imaging

BT - ISBI 2022 - Proceedings

Y2 - 28 March 2022 through 31 March 2022

ER -

Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this