A Rater Training Protocol to Assess Team Performance

Walter Eppich, Anna P. Nannicelli, Nicholas P. Seivert, Min Woong Sohn, Ranna Rozenfeld, Donna M. Woods, Jane L. Holl

Research output: Contribution to journalArticlepeer-review

23 Scopus citations


Introduction: Simulation-based methodologies are increasingly used to assess teamwork and communication skills and provide team training. Formative feedback regarding team performance is an essential component. While effective use of simulation for assessment or training requires accurate rating of team performance, examples of rater-training programs in health care are scarce. We describe our rater training program and report interrater reliability during phases of training and independent rating. Methods: We selected an assessment tool shown to yield valid and reliable results and developed a rater training protocol with an accompanying rater training handbook. The rater training program was modeled after previously described high-stakes assessments in the setting of 3 facilitated training sessions. Adjacent agreement was used to measure interrater reliability between raters. Results: Nine raters with a background in health care and/or patient safety evaluated team performance of 42 in-situ simulations using post-hoc video review. Adjacent agreement increased from the second training session (83.6%) to the third training session (85.6%) when evaluating the same video segments. Adjacent agreement for the rating of overall team performance was 78.3%, which was added for the third training session. Adjacent agreement was 97% 4 weeks posttraining and 90.6% at the end of independent rating of all simulation videos. Discussion: Rater training is an important element in team performance assessment, and providing examples of rater training programs is essential. Articulating key rating anchors promotes adequate interrater reliability. In addition, using adjacent agreement as a measure allows differentiation between high- and low-performing teams on video review.

Original languageEnglish
Pages (from-to)83-90
Number of pages8
JournalJournal of Continuing Education in the Health Professions
Issue number2
StatePublished - Mar 1 2015

Bibliographical note

Publisher Copyright:
© 2015 The Alliance for Continuing Education in the Health Professions, the Society for Academic Continuing Medical Education, and the Council on Continuing Medical Education, Association for Hospital Medical Education.


  • Assessment
  • Health services research
  • Multisource feedback/peer assessment
  • Program planning/curriculum development
  • Rater training
  • Simulation
  • Team performance

ASJC Scopus subject areas

  • Medicine (all)


Dive into the research topics of 'A Rater Training Protocol to Assess Team Performance'. Together they form a unique fingerprint.

Cite this