Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging

Gongbo Liang, Connor Greenwell, Yu Zhang, Xin Xing, Xiaoqin Wang, Ramakanth Kavuluru, Nathan Jacobs

Research output: Contribution to journalArticlepeer-review

6 Scopus citations


A key challenge in training neural networks for a given medical imaging task is the difficulty of obtaining a sufficient number of manually labeled examples. In contrast, textual imaging reports are often readily available in medical records and contain rich but unstructured interpretations written by experts as part of standard clinical practice. We propose using these textual reports as a form of weak supervision to improve the image interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune it in a transfer learning setting for a supervised task using a small labeled dataset. The end result is a neural network that automatically interprets imagery without requiring textual reports during inference. We evaluate our method on three classification tasks and find consistent performance improvements, reducing the need for labeled data by 67%-98%.

Original languageEnglish
Pages (from-to)1640-1649
Number of pages10
JournalIEEE Journal of Biomedical and Health Informatics
Issue number4
StatePublished - Apr 1 2022

Bibliographical note

Publisher Copyright:
© 2013 IEEE.


  • Annotation-efficient modeling
  • convolutional neural network
  • pre-training
  • text-image matching

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Informatics
  • Electrical and Electronic Engineering
  • Health Information Management


Dive into the research topics of 'Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging'. Together they form a unique fingerprint.

Cite this