Using ensembles and distillation to optimize the deployment of deep learning models for the classification of electronic cancer pathology reports

Kevin De Angeli, Shang Gao, Andrew Blanchard, Eric B. Durbin, Xiao Cheng Wu, Antoinette Stroup, Jennifer Doherty, Stephen M. Schwartz, Charles Wiggins, Linda Coyle, Lynne Penberthy, Georgia Tourassi, Hong Jun Yoon

Research output: Contribution to journalArticlepeer-review


Objective: We aim to reduce overfitting and model overconfidence by distilling the knowledge of an ensemble of deep learning models into a single model for the classification of cancer pathology reports. Materials and Methods: We consider the text classification problem that involves 5 individual tasks. The baseline model consists of a multitask convolutional neural network (MtCNN), and the implemented ensemble (teacher) consists of 1000 MtCNNs. We performed knowledge transfer by training a single model (student) with soft labels derived through the aggregation of ensemble predictions. We evaluate performance based on accuracy and abstention rates by using softmax thresholding. Results: The student model outperforms the baseline MtCNN in terms of abstention rates and accuracy, thereby allowing the model to be used with a larger volume of documents when deployed. The highest boost was observed for subsite and histology, for which the student model classified an additional 1.81% reports for subsite and 3.33% reports for histology. Discussion: Ensemble predictions provide a useful strategy for quantifying the uncertainty inherent in labeled data and thereby enable the construction of soft labels with estimated probabilities for multiple classes for a given document. Training models with the derived soft labels reduce model confidence in difficult-to-classify documents, thereby leading to a reduction in the number of highly confident wrong predictions. Conclusions: Ensemble model distillation is a simple tool to reduce model overconfidence in problems with extreme class imbalance and noisy datasets. These methods can facilitate the deployment of deep learning models in high-risk domains with low computational resources where minimizing inference time is required.

Original languageEnglish
Article numberooac075
JournalJAMIA Open
Issue number3
StatePublished - Oct 1 2022

Bibliographical note

Publisher Copyright:
© 2022 The Author(s). Published by Oxford University Press on behalf of the American Medical Informatics Association.


  • CNN
  • NLP
  • deep learning
  • ensemble distillation
  • selective classification

ASJC Scopus subject areas

  • Health Informatics


Dive into the research topics of 'Using ensembles and distillation to optimize the deployment of deep learning models for the classification of electronic cancer pathology reports'. Together they form a unique fingerprint.

Cite this