TY - JOUR
T1 - Using Supervised Machine Learning in Automated Content Analysis
T2 - An Example Using Relational Uncertainty
AU - Pilny, Andrew
AU - McAninch, Kelly
AU - Slone, Amanda
AU - Moore, Kelsey
N1 - Publisher Copyright:
© 2019, © 2019 Taylor & Francis Group, LLC.
PY - 2019/10/2
Y1 - 2019/10/2
N2 - The goal of this research is to make progress towards using supervised machine learning for automated content analysis dealing with complex interpretations of text. For Step 1, two humans coded a sub-sample of online forum posts for relational uncertainty. For Step 2, we evaluated reliability, in which we trained three different classifiers to learn from those subjective human interpretations. Reliability was established when two different metrics of inter-coder reliability could not distinguish whether a human or a machine coded the text on a separate hold-out set. Finally, in Step 3 we assessed validity. To accomplish this, we administered a survey in which participants described their own relational uncertainty/certainty via text and completed a questionnaire. After classifying the text, the machine’s classifications of the participants’ text positively correlated with the subjects’ own self-reported relational uncertainty and relational satisfaction. We discuss our results in line with areas of computational communication science, content analysis, and interpersonal communication.
AB - The goal of this research is to make progress towards using supervised machine learning for automated content analysis dealing with complex interpretations of text. For Step 1, two humans coded a sub-sample of online forum posts for relational uncertainty. For Step 2, we evaluated reliability, in which we trained three different classifiers to learn from those subjective human interpretations. Reliability was established when two different metrics of inter-coder reliability could not distinguish whether a human or a machine coded the text on a separate hold-out set. Finally, in Step 3 we assessed validity. To accomplish this, we administered a survey in which participants described their own relational uncertainty/certainty via text and completed a questionnaire. After classifying the text, the machine’s classifications of the participants’ text positively correlated with the subjects’ own self-reported relational uncertainty and relational satisfaction. We discuss our results in line with areas of computational communication science, content analysis, and interpersonal communication.
UR - http://www.scopus.com/inward/record.url?scp=85070830684&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85070830684&partnerID=8YFLogxK
U2 - 10.1080/19312458.2019.1650166
DO - 10.1080/19312458.2019.1650166
M3 - Article
AN - SCOPUS:85070830684
SN - 1931-2458
VL - 13
SP - 287
EP - 304
JO - Communication Methods and Measures
JF - Communication Methods and Measures
IS - 4
ER -