Authentic assessments: a method to detect anomalies in assessment response patterns via neural network

Kate D. Cordell, Himanshu Rao, John Lyons

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

Surveys, questionnaires, and assessments are the only approved means for measuring outcomes, yet there is a paucity of evidence that these tools can perform well as outcomes measures. A major barrier to utilizing these tools as measures of improvement in care is an inability to recognize when the responses are incomplete, inaccurate, or insincere. In this study, we propose a hybrid unsupervised-supervised approach to identify inauthentic assessments via anomalous response patterns. The study utilized 60-question behavioral assessment from a clinical population served by a county-operated public children’s behavioral health services (n = 42,945). A novel hybrid unsupervised-supervised approach was developed to identify inauthentic assessment records from highly dimensional assessment data without the need for a priori record labels, which would otherwise require countless hours of record review by highly trained clinical staff. A neural network trained with 75% of the labeled data recognized anomalous response patterns in the test data with 84.5% sensitivity and 97.3% specificity. The model identified 26% of records as potentially inauthentic based on anomalous response patterns. For mental health and behavioral health, this novel method could flag a relatively small proportion of the records for clinical review while allowing records with probable authenticity to bypass review processes, saving time and creating efficiency in care. This method is also a potential tool for improving service quality through the identification of circumstances that have been overlooked but could otherwise be addressed during care. For decades, assessment data has been collected by policy and ignored by practice due to limitations of utility. With the right approaches, we could uncover the patterns of what works for whom that are currently hidden in our existing assessments.

Original languageEnglish
Pages (from-to)439-458
Number of pages20
JournalHealth Services and Outcomes Research Methodology
Volume21
Issue number4
DOIs
StatePublished - Dec 2021

Bibliographical note

Publisher Copyright:
© 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature.

Keywords

  • Anomaly detection
  • Assessments
  • Behavioral health
  • Neural network
  • Outliers
  • Response pattern

ASJC Scopus subject areas

  • Health Policy
  • Public Health, Environmental and Occupational Health

Fingerprint

Dive into the research topics of 'Authentic assessments: a method to detect anomalies in assessment response patterns via neural network'. Together they form a unique fingerprint.

Cite this