An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts

Danielle S. Bitterman; Eli Goldner; Sean Finan; David Harris; Eric B. Durbin; Harry Hochheiser; Jeremy L. Warner; Raymond H. Mak; Timothy Miller; Guergana K. Savova

doi:10.1016/j.ijrobp.2023.03.055

An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts

Danielle S. Bitterman
, Eli Goldner
, Sean Finan
, David Harris
, Eric B. Durbin
, Harry Hochheiser
, Jeremy L. Warner
, Raymond H. Mak
, Timothy Miller
, Guergana K. Savova

Research output: Contribution to journal › Article › peer-review

17 Scopus citations

Abstract

Purpose: Real-world evidence for radiation therapy (RT) is limited because it is often documented only in the clinical narrative. We developed a natural language processing system for automated extraction of detailed RT events from text to support clinical phenotyping. Methods and Materials: A multi-institutional data set of 96 clinician notes, 129 North American Association of Central Cancer Registries cancer abstracts, and 270 RT prescriptions from HemOnc.org was used and divided into train, development, and test sets. Documents were annotated for RT events and associated properties: dose, fraction frequency, fraction number, date, treatment site, and boost. Named entity recognition models for properties were developed by fine-tuning BioClinicalBERT and RoBERTa transformer models. A multiclass RoBERTa-based relation extraction model was developed to link each dose mention with each property in the same event. Models were combined with symbolic rules to create a hybrid end-to-end pipeline for comprehensive RT event extraction. Results: Named entity recognition models were evaluated on the held-out test set with F1 results of 0.96, 0.88, 0.94, 0.88, 0.67, and 0.94 for dose, fraction frequency, fraction number, date, treatment site, and boost, respectively. The relation model achieved an average F1 of 0.86 when the input was gold-labeled entities. The end-to-end system F1 result was 0.81. The end-to-end system performed best on North American Association of Central Cancer Registries abstracts (average F1 0.90), which are mostly copy–paste content from clinician notes. Conclusions: We developed methods and a hybrid end-to-end system for RT event extraction, which is the first natural language processing system for this task. This system provides proof-of-concept for real-world RT data collection for research and is promising for the potential of natural language processing methods to support clinical care.

Original language	English
Pages (from-to)	262-273
Number of pages	12
Journal	International Journal of Radiation Oncology Biology Physics
Volume	117
Issue number	1
DOIs	https://doi.org/10.1016/j.ijrobp.2023.03.055
State	Published - Sep 1 2023

Bibliographical note

Publisher Copyright:
© 2023 Elsevier Inc.

Funding

The work was supported by National Institutes of Health grants UH3CA243120 (E.G., S.F., D.H., E.B.D., H.H., J.L.W., G.K.S.), U24CA248010 (E.G., S.F., H.H., J.L.W., G.K.S.), R01LM010090 (T.M., G.K.S.), R01LM013486 (T.M., G.K.S.), 5R01GM11435 (T.M., G.K.S.), U24CA265879 (J.L.W.), HHSN261201800013I / HHSN26100001 (E.B.D.), and P30 CA177558 (E.B.D.). Disclosures: J.L.W. received funding from Brown Physicians Inc outside the submitted work; is a consultant for Westat, Melax Tech, Roche, and Flatiron all outside the submitted work; and is the deputy editor of HemOnc.org and cofounder of HemOnc.org LLC. D.S.B. is an associate editor of the Radiation Oncology section of HemOnc.org.

Funders	Funder number
National Institutes of Health (NIH)	P30 CA177558, 5R01GM11435, U24CA248010, R01LM010090, UH3CA243120, U24CA265879, HHSN261201800013I / HHSN26100001, R01LM013486
Roche Canada

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

ASJC Scopus subject areas

Radiation
Oncology
Radiology Nuclear Medicine and imaging
Cancer Research

Access to Document

10.1016/j.ijrobp.2023.03.055

Cite this

Bitterman, D. S., Goldner, E., Finan, S., Harris, D., Durbin, E. B., Hochheiser, H., Warner, J. L., Mak, R. H., Miller, T., & Savova, G. K. (2023). An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts. International Journal of Radiation Oncology Biology Physics, 117(1), 262-273. https://doi.org/10.1016/j.ijrobp.2023.03.055

@article{4a7fe61ded3643b3a3c43284507e012f,

title = "An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts",

abstract = "Purpose: Real-world evidence for radiation therapy (RT) is limited because it is often documented only in the clinical narrative. We developed a natural language processing system for automated extraction of detailed RT events from text to support clinical phenotyping. Methods and Materials: A multi-institutional data set of 96 clinician notes, 129 North American Association of Central Cancer Registries cancer abstracts, and 270 RT prescriptions from HemOnc.org was used and divided into train, development, and test sets. Documents were annotated for RT events and associated properties: dose, fraction frequency, fraction number, date, treatment site, and boost. Named entity recognition models for properties were developed by fine-tuning BioClinicalBERT and RoBERTa transformer models. A multiclass RoBERTa-based relation extraction model was developed to link each dose mention with each property in the same event. Models were combined with symbolic rules to create a hybrid end-to-end pipeline for comprehensive RT event extraction. Results: Named entity recognition models were evaluated on the held-out test set with F1 results of 0.96, 0.88, 0.94, 0.88, 0.67, and 0.94 for dose, fraction frequency, fraction number, date, treatment site, and boost, respectively. The relation model achieved an average F1 of 0.86 when the input was gold-labeled entities. The end-to-end system F1 result was 0.81. The end-to-end system performed best on North American Association of Central Cancer Registries abstracts (average F1 0.90), which are mostly copy–paste content from clinician notes. Conclusions: We developed methods and a hybrid end-to-end system for RT event extraction, which is the first natural language processing system for this task. This system provides proof-of-concept for real-world RT data collection for research and is promising for the potential of natural language processing methods to support clinical care.",

author = "Bitterman, \{Danielle S.\} and Eli Goldner and Sean Finan and David Harris and Durbin, \{Eric B.\} and Harry Hochheiser and Warner, \{Jeremy L.\} and Mak, \{Raymond H.\} and Timothy Miller and Savova, \{Guergana K.\}",

note = "Publisher Copyright: {\textcopyright} 2023 Elsevier Inc.",

year = "2023",

month = sep,

day = "1",

doi = "10.1016/j.ijrobp.2023.03.055",

language = "English",

volume = "117",

pages = "262--273",

number = "1",

}

Bitterman, DS, Goldner, E, Finan, S, Harris, D, Durbin, EB, Hochheiser, H, Warner, JL, Mak, RH, Miller, T & Savova, GK 2023, 'An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts', International Journal of Radiation Oncology Biology Physics, vol. 117, no. 1, pp. 262-273. https://doi.org/10.1016/j.ijrobp.2023.03.055

TY - JOUR

T1 - An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts

AU - Bitterman, Danielle S.

AU - Goldner, Eli

AU - Finan, Sean

AU - Harris, David

AU - Durbin, Eric B.

AU - Hochheiser, Harry

AU - Warner, Jeremy L.

AU - Mak, Raymond H.

AU - Miller, Timothy

AU - Savova, Guergana K.

PY - 2023/9/1

Y1 - 2023/9/1

N2 - Purpose: Real-world evidence for radiation therapy (RT) is limited because it is often documented only in the clinical narrative. We developed a natural language processing system for automated extraction of detailed RT events from text to support clinical phenotyping. Methods and Materials: A multi-institutional data set of 96 clinician notes, 129 North American Association of Central Cancer Registries cancer abstracts, and 270 RT prescriptions from HemOnc.org was used and divided into train, development, and test sets. Documents were annotated for RT events and associated properties: dose, fraction frequency, fraction number, date, treatment site, and boost. Named entity recognition models for properties were developed by fine-tuning BioClinicalBERT and RoBERTa transformer models. A multiclass RoBERTa-based relation extraction model was developed to link each dose mention with each property in the same event. Models were combined with symbolic rules to create a hybrid end-to-end pipeline for comprehensive RT event extraction. Results: Named entity recognition models were evaluated on the held-out test set with F1 results of 0.96, 0.88, 0.94, 0.88, 0.67, and 0.94 for dose, fraction frequency, fraction number, date, treatment site, and boost, respectively. The relation model achieved an average F1 of 0.86 when the input was gold-labeled entities. The end-to-end system F1 result was 0.81. The end-to-end system performed best on North American Association of Central Cancer Registries abstracts (average F1 0.90), which are mostly copy–paste content from clinician notes. Conclusions: We developed methods and a hybrid end-to-end system for RT event extraction, which is the first natural language processing system for this task. This system provides proof-of-concept for real-world RT data collection for research and is promising for the potential of natural language processing methods to support clinical care.

AB - Purpose: Real-world evidence for radiation therapy (RT) is limited because it is often documented only in the clinical narrative. We developed a natural language processing system for automated extraction of detailed RT events from text to support clinical phenotyping. Methods and Materials: A multi-institutional data set of 96 clinician notes, 129 North American Association of Central Cancer Registries cancer abstracts, and 270 RT prescriptions from HemOnc.org was used and divided into train, development, and test sets. Documents were annotated for RT events and associated properties: dose, fraction frequency, fraction number, date, treatment site, and boost. Named entity recognition models for properties were developed by fine-tuning BioClinicalBERT and RoBERTa transformer models. A multiclass RoBERTa-based relation extraction model was developed to link each dose mention with each property in the same event. Models were combined with symbolic rules to create a hybrid end-to-end pipeline for comprehensive RT event extraction. Results: Named entity recognition models were evaluated on the held-out test set with F1 results of 0.96, 0.88, 0.94, 0.88, 0.67, and 0.94 for dose, fraction frequency, fraction number, date, treatment site, and boost, respectively. The relation model achieved an average F1 of 0.86 when the input was gold-labeled entities. The end-to-end system F1 result was 0.81. The end-to-end system performed best on North American Association of Central Cancer Registries abstracts (average F1 0.90), which are mostly copy–paste content from clinician notes. Conclusions: We developed methods and a hybrid end-to-end system for RT event extraction, which is the first natural language processing system for this task. This system provides proof-of-concept for real-world RT data collection for research and is promising for the potential of natural language processing methods to support clinical care.

UR - https://www.scopus.com/pages/publications/85153183126

UR - https://www.scopus.com/pages/publications/85153183126#tab=citedBy

U2 - 10.1016/j.ijrobp.2023.03.055

DO - 10.1016/j.ijrobp.2023.03.055

M3 - Article

C2 - 36990288

AN - SCOPUS:85153183126

SN - 0360-3016

VL - 117

SP - 262

EP - 273

JO - International Journal of Radiation Oncology Biology Physics

JF - International Journal of Radiation Oncology Biology Physics

IS - 1

ER -

An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts

Abstract

Bibliographical note

Funding

UN SDGs

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this