The reliability of six faculty members in identifying important OSCE items

Purpose. To measure the agreement among faculty members about the importance of items on a checklist used to grade an objective structured clinical examination (OSCE) station. Methods. Six faculty members rated the importance of 47 items for an OSCE station in which students took the history of a patient with sore throat and hoarseness. Results. Of the 47 items, the rates independently identified 15 items as very important. The reliability of each individual rater was fair (averaged value of α = .63.) The reliability of the mean rating of the six raters was high (α = .91). Conclusions. The results strongly suggest that when a group of faculty members cooperatively identifies the important items to be included in an OSCE checklist, the reliability of the checklist is superior to one created by a single author.