Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task

Abeed Sarker; Maksim Belousov; Jasper Friedrichs; Kai Hakala; Svetlana Kiritchenko; Farrokh Mehryary; Sifei Han; Tung Tran; Anthony Rios; Ramakanth Kavuluru; Berry De Bruijn; Filip Ginter; Debanjan Mahata; Saif M. Mohammad; Goran Nenadic; Graciela Gonzalez-Hernandez

doi:10.1093/jamia/ocy114

Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task

Abeed Sarker, Maksim Belousov, Jasper Friedrichs, Kai Hakala, Svetlana Kiritchenko, Farrokh Mehryary, Sifei Han, Tung Tran, Anthony Rios, Ramakanth Kavuluru, Berry De Bruijn, Filip Ginter, Debanjan Mahata, Saif M. Mohammad, Goran Nenadic, Graciela Gonzalez-Hernandez

Internal Medicine

Research output: Contribution to journal › Article › peer-review

58 Scopus citations

Abstract

Objective We executed the Social Media Mining for Health (SMM4H) 2017 shared tasks to enable the community-driven development and large-scale evaluation of automatic text processing methods for the classification and normalization of health-related text from social media. An additional objective was to publicly release manually annotated data. Materials and Methods We organized 3 independent subtasks: Automatic classification of self-reports of 1) adverse drug reactions (ADRs) and 2) medication consumption, from medication-mentioning tweets, and 3) normalization of ADR expressions. Training data consisted of 15 717 annotated tweets for (1), 10 260 for (2), and 6650 ADR phrases and identifiers for (3); and exhibited typical properties of social-media-based health-related texts. Systems were evaluated using 9961, 7513, and 2500 instances for the 3 subtasks, respectively. We evaluated performances of classes of methods and ensembles of system combinations following the shared tasks. Results Among 55 system runs, the best system scores for the 3 subtasks were 0.435 (ADR class F 1-score) for subtask-1, 0.693 (micro-Averaged F 1-score over two classes) for subtask-2, and 88.5% (accuracy) for subtask-3. Ensembles of system combinations obtained best scores of 0.476, 0.702, and 88.7%, outperforming individual systems. Discussion Among individual systems, support vector machines and convolutional neural networks showed high performance. Performance gains achieved by ensembles of system combinations suggest that such strategies may be suitable for operational systems relying on difficult text classification tasks (eg, subtask-1). Conclusions Data imbalance and lack of context remain challenges for natural language processing of social media text. Annotated data from the shared task have been made available as reference standards for future studies (http://dx.doi.org/10.17632/rxwfb3tysd.1).

Original language	English
Pages (from-to)	1274-1283
Number of pages	10
Journal	Journal of the American Medical Informatics Association
Volume	25
Issue number	10
DOIs	https://doi.org/10.1093/jamia/ocy114
State	Published - Oct 1 2018

Bibliographical note

Publisher Copyright:
© © The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association.

Funding

AS and GG were partially supported by the National Institutes of Health (NIH) National Library of Medicine (NLM) grant number NIH NLM R01LM011176. KH, FM, and FG (TurkuNLP) are supported by ATT Tieto kaytt\u00F6\u00F6n grant. SH, TT, AR, and RK (UKNLP) are supported by the NIH National Cancer Institute through grant R21CA218231 and NVIDIA Corporation through the Titan X Pascal GPU donation. MB and GN are supported by the UK EPSRC (grants EP/I028099/1 and EP/N027280/1). The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

Funders	Funder number
ATT
National Institutes of Health (NIH)
Nvidia
Medical Research Council	MR/K006665/1, MC_PC_13042
Engineering and Physical Sciences Research Council	EP/N027280/1, EP/I028099/1
National Childhood Cancer Registry – National Cancer Institute	R21CA218231
U.S. National Library of Medicine	R01LM011176

Keywords

machine learning
natural language processing
pharmacovigilance
social media
text mining

ASJC Scopus subject areas

Health Informatics

Access to Document

10.1093/jamia/ocy114

Cite this

Sarker, A., Belousov, M., Friedrichs, J., Hakala, K., Kiritchenko, S., Mehryary, F., Han, S., Tran, T., Rios, A., Kavuluru, R., De Bruijn, B., Ginter, F., Mahata, D., Mohammad, S. M., Nenadic, G., & Gonzalez-Hernandez, G. (2018). Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task. Journal of the American Medical Informatics Association, 25(10), 1274-1283. https://doi.org/10.1093/jamia/ocy114

@article{a1b9e1233db84be3962d3d56ed2a362b,

title = "Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task",

abstract = "Objective We executed the Social Media Mining for Health (SMM4H) 2017 shared tasks to enable the community-driven development and large-scale evaluation of automatic text processing methods for the classification and normalization of health-related text from social media. An additional objective was to publicly release manually annotated data. Materials and Methods We organized 3 independent subtasks: Automatic classification of self-reports of 1) adverse drug reactions (ADRs) and 2) medication consumption, from medication-mentioning tweets, and 3) normalization of ADR expressions. Training data consisted of 15 717 annotated tweets for (1), 10 260 for (2), and 6650 ADR phrases and identifiers for (3); and exhibited typical properties of social-media-based health-related texts. Systems were evaluated using 9961, 7513, and 2500 instances for the 3 subtasks, respectively. We evaluated performances of classes of methods and ensembles of system combinations following the shared tasks. Results Among 55 system runs, the best system scores for the 3 subtasks were 0.435 (ADR class F 1-score) for subtask-1, 0.693 (micro-Averaged F 1-score over two classes) for subtask-2, and 88.5% (accuracy) for subtask-3. Ensembles of system combinations obtained best scores of 0.476, 0.702, and 88.7%, outperforming individual systems. Discussion Among individual systems, support vector machines and convolutional neural networks showed high performance. Performance gains achieved by ensembles of system combinations suggest that such strategies may be suitable for operational systems relying on difficult text classification tasks (eg, subtask-1). Conclusions Data imbalance and lack of context remain challenges for natural language processing of social media text. Annotated data from the shared task have been made available as reference standards for future studies (http://dx.doi.org/10.17632/rxwfb3tysd.1).",

keywords = "machine learning, natural language processing, pharmacovigilance, social media, text mining",

author = "Abeed Sarker and Maksim Belousov and Jasper Friedrichs and Kai Hakala and Svetlana Kiritchenko and Farrokh Mehryary and Sifei Han and Tung Tran and Anthony Rios and Ramakanth Kavuluru and {De Bruijn}, Berry and Filip Ginter and Debanjan Mahata and Mohammad, {Saif M.} and Goran Nenadic and Graciela Gonzalez-Hernandez",

note = "Publisher Copyright: {\textcopyright} {\textcopyright} The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association.",

year = "2018",

month = oct,

day = "1",

doi = "10.1093/jamia/ocy114",

language = "English",

volume = "25",

pages = "1274--1283",

number = "10",

}

Sarker, A, Belousov, M, Friedrichs, J, Hakala, K, Kiritchenko, S, Mehryary, F, Han, S, Tran, T, Rios, A, Kavuluru, R, De Bruijn, B, Ginter, F, Mahata, D, Mohammad, SM, Nenadic, G & Gonzalez-Hernandez, G 2018, 'Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task', Journal of the American Medical Informatics Association, vol. 25, no. 10, pp. 1274-1283. https://doi.org/10.1093/jamia/ocy114

Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task. / Sarker, Abeed; Belousov, Maksim; Friedrichs, Jasper et al.
In: Journal of the American Medical Informatics Association, Vol. 25, No. 10, 01.10.2018, p. 1274-1283.

Research output: Contribution to journal › Article › peer-review

TY - JOUR

T1 - Data and systems for medication-related text classification and concept normalization from Twitter

T2 - Insights from the Social Media Mining for Health (SMM4H)-2017 shared task

AU - Sarker, Abeed

AU - Belousov, Maksim

AU - Friedrichs, Jasper

AU - Hakala, Kai

AU - Kiritchenko, Svetlana

AU - Mehryary, Farrokh

AU - Han, Sifei

AU - Tran, Tung

AU - Rios, Anthony

AU - Kavuluru, Ramakanth

AU - De Bruijn, Berry

AU - Ginter, Filip

AU - Mahata, Debanjan

AU - Mohammad, Saif M.

AU - Nenadic, Goran

AU - Gonzalez-Hernandez, Graciela

PY - 2018/10/1

Y1 - 2018/10/1

N2 - Objective We executed the Social Media Mining for Health (SMM4H) 2017 shared tasks to enable the community-driven development and large-scale evaluation of automatic text processing methods for the classification and normalization of health-related text from social media. An additional objective was to publicly release manually annotated data. Materials and Methods We organized 3 independent subtasks: Automatic classification of self-reports of 1) adverse drug reactions (ADRs) and 2) medication consumption, from medication-mentioning tweets, and 3) normalization of ADR expressions. Training data consisted of 15 717 annotated tweets for (1), 10 260 for (2), and 6650 ADR phrases and identifiers for (3); and exhibited typical properties of social-media-based health-related texts. Systems were evaluated using 9961, 7513, and 2500 instances for the 3 subtasks, respectively. We evaluated performances of classes of methods and ensembles of system combinations following the shared tasks. Results Among 55 system runs, the best system scores for the 3 subtasks were 0.435 (ADR class F 1-score) for subtask-1, 0.693 (micro-Averaged F 1-score over two classes) for subtask-2, and 88.5% (accuracy) for subtask-3. Ensembles of system combinations obtained best scores of 0.476, 0.702, and 88.7%, outperforming individual systems. Discussion Among individual systems, support vector machines and convolutional neural networks showed high performance. Performance gains achieved by ensembles of system combinations suggest that such strategies may be suitable for operational systems relying on difficult text classification tasks (eg, subtask-1). Conclusions Data imbalance and lack of context remain challenges for natural language processing of social media text. Annotated data from the shared task have been made available as reference standards for future studies (http://dx.doi.org/10.17632/rxwfb3tysd.1).

AB - Objective We executed the Social Media Mining for Health (SMM4H) 2017 shared tasks to enable the community-driven development and large-scale evaluation of automatic text processing methods for the classification and normalization of health-related text from social media. An additional objective was to publicly release manually annotated data. Materials and Methods We organized 3 independent subtasks: Automatic classification of self-reports of 1) adverse drug reactions (ADRs) and 2) medication consumption, from medication-mentioning tweets, and 3) normalization of ADR expressions. Training data consisted of 15 717 annotated tweets for (1), 10 260 for (2), and 6650 ADR phrases and identifiers for (3); and exhibited typical properties of social-media-based health-related texts. Systems were evaluated using 9961, 7513, and 2500 instances for the 3 subtasks, respectively. We evaluated performances of classes of methods and ensembles of system combinations following the shared tasks. Results Among 55 system runs, the best system scores for the 3 subtasks were 0.435 (ADR class F 1-score) for subtask-1, 0.693 (micro-Averaged F 1-score over two classes) for subtask-2, and 88.5% (accuracy) for subtask-3. Ensembles of system combinations obtained best scores of 0.476, 0.702, and 88.7%, outperforming individual systems. Discussion Among individual systems, support vector machines and convolutional neural networks showed high performance. Performance gains achieved by ensembles of system combinations suggest that such strategies may be suitable for operational systems relying on difficult text classification tasks (eg, subtask-1). Conclusions Data imbalance and lack of context remain challenges for natural language processing of social media text. Annotated data from the shared task have been made available as reference standards for future studies (http://dx.doi.org/10.17632/rxwfb3tysd.1).

KW - machine learning

KW - natural language processing

KW - pharmacovigilance

KW - social media

KW - text mining

UR - http://www.scopus.com/inward/record.url?scp=85054889806&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054889806&partnerID=8YFLogxK

U2 - 10.1093/jamia/ocy114

DO - 10.1093/jamia/ocy114

M3 - Article

C2 - 30272184

AN - SCOPUS:85054889806

SN - 1067-5027

VL - 25

SP - 1274

EP - 1283

JO - Journal of the American Medical Informatics Association

JF - Journal of the American Medical Informatics Association

IS - 10

ER -

Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task

Abstract

Bibliographical note

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this