TY - JOUR
T1 - Feasibility of an automated interview grounded in multiple mini interview (MMI) methodology for selection into the health professions
T2 - an international multimethod evaluation
AU - Callwood, Alison
AU - Gillam, Lee
AU - Christidis, Angelos
AU - Doulton, Jia
AU - Harris, Jenny
AU - Piano, Marianne
AU - Kubacki, Angela
AU - Tiffin, Paul A.
AU - Roberts, Karen
AU - Tarmey, Drew
AU - Dalton, Doris
AU - Valentin, Virginia L.
N1 - Publisher Copyright:
© Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.
PY - 2022/2/9
Y1 - 2022/2/9
N2 - ObjectivesGlobal, COVID-driven restrictions around face-to-face interviews for healthcare student selection have forced admission staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first automated interview grounded in multiple mini-interview (MMI) methodology. This study aimed to explore test-retest reliability, acceptability and usability of the system.Design, setting and participantsMultimethod feasibility study in Physician Associate programmes from two UK and one US university during 2019-2020.Primary, secondary outcomesFeasibility measures (test-retest reliability, acceptability and usability) were assessed using intraclass correlation (ICC), descriptive statistics, thematic and content analysis.MethodsVolunteers took (T1), then repeated (T2), the automated MMI, with a 7-day interval (±2) then completed an evaluation questionnaire. Admission staff participated in focus group discussions.ResultsSixty-two students and seven admission staff participated; 34 students and 4 staff from UK and 28 students and 3 staff from US universities. Good-excellent test-retest reliability was observed at two sites (US and UK2) with T1 and T2 ICC between 0.65 and 0.81 (p<0.001) when assessed by individual total scores (range 80.6-119), station total scores 0.6-0.91, p<0.005 and individual site (≥0.79 p<0.001). Mean test re-test ICC across all three sites was 0.82 p<0.001 (95% CI 0.7 to 0.9). Admission staff reported potential to reduce resource costs and bias through a more objective screening tool for preselection or to replace some MMI stations in a ‘hybrid model’. Maintaining human interaction through ‘touch points’ was considered essential. Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.ConclusionThese preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection.
AB - ObjectivesGlobal, COVID-driven restrictions around face-to-face interviews for healthcare student selection have forced admission staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first automated interview grounded in multiple mini-interview (MMI) methodology. This study aimed to explore test-retest reliability, acceptability and usability of the system.Design, setting and participantsMultimethod feasibility study in Physician Associate programmes from two UK and one US university during 2019-2020.Primary, secondary outcomesFeasibility measures (test-retest reliability, acceptability and usability) were assessed using intraclass correlation (ICC), descriptive statistics, thematic and content analysis.MethodsVolunteers took (T1), then repeated (T2), the automated MMI, with a 7-day interval (±2) then completed an evaluation questionnaire. Admission staff participated in focus group discussions.ResultsSixty-two students and seven admission staff participated; 34 students and 4 staff from UK and 28 students and 3 staff from US universities. Good-excellent test-retest reliability was observed at two sites (US and UK2) with T1 and T2 ICC between 0.65 and 0.81 (p<0.001) when assessed by individual total scores (range 80.6-119), station total scores 0.6-0.91, p<0.005 and individual site (≥0.79 p<0.001). Mean test re-test ICC across all three sites was 0.82 p<0.001 (95% CI 0.7 to 0.9). Admission staff reported potential to reduce resource costs and bias through a more objective screening tool for preselection or to replace some MMI stations in a ‘hybrid model’. Maintaining human interaction through ‘touch points’ was considered essential. Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.ConclusionThese preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection.
KW - education & training (see medical education & training)
KW - medical education & training
KW - quality in health care
UR - http://www.scopus.com/inward/record.url?scp=85124302421&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85124302421&partnerID=8YFLogxK
U2 - 10.1136/bmjopen-2021-050394
DO - 10.1136/bmjopen-2021-050394
M3 - Article
C2 - 35140144
AN - SCOPUS:85124302421
SN - 2044-6055
VL - 12
JO - BMJ Open
JF - BMJ Open
IS - 2
M1 - e050394
ER -