TY - JOUR
T1 - Medical students' ratings of faculty teaching in a multi-instructor setting
T2 - An examination of monotonic response patterns
AU - Stratton, Terry D.
AU - Witzke, Donald B.
AU - Jacob, Robert J.
AU - Sauer, Marlene J.
AU - Murphy-Spencer, Amy
PY - 2002
Y1 - 2002
N2 - Realizing that the psychometric properties of a measure may be highly variable is especially relevant in a multi-instructor context, since an implicit assumption is that student ratings are equally reliable and valid for all faculty ratees. As a possible indicator of nonattending (i.e. invalid) responses, the authors examined the effects of monotonic response patterns on the reliabilities of students' ratings of faculty teaching - including how an alternative presentation format may reduce the prevalence of this behavior. Second-year medical and dental students (n = 130) enrolled in a required basic science course during the 1998-99 academic year were randomly assigned to one of two groups - each of which evaluated the teaching of 6 different faculty across 6 distinct dimensions (i.e. overall quality, organization, preparation, stimulation, respectfulness, and helpfulness). Using a 'split ballot' design, two versions of the conceptually equivalent faculty evaluation form were distributed at random to students in each group. Form A contained the 'traditional' items-within-faculty format, while Form B listed faculty-within-item. The number of monotonic forms (i.e. the identical rating of all 6 items) varied measurably across faculty ratees, as did the respective effects on scale reliabilities. Alpha was especially inflated where a sizeable proportion of monotonic patterns were located on response categories that were either very high (> +1.28 z m deviations) or very low (< -1.28 z m deviations) compared to the group mean. Lastly, the prevalence of monotonic response patterns was significantly (p = ≤ 0.01) less when a faculty-within-item format is used (Form B). These findings suggest that monotonic response patterns differentially impact the reliabilities and, hence, the validity of students' ratings of individual faculty in a multi-instructor context.
AB - Realizing that the psychometric properties of a measure may be highly variable is especially relevant in a multi-instructor context, since an implicit assumption is that student ratings are equally reliable and valid for all faculty ratees. As a possible indicator of nonattending (i.e. invalid) responses, the authors examined the effects of monotonic response patterns on the reliabilities of students' ratings of faculty teaching - including how an alternative presentation format may reduce the prevalence of this behavior. Second-year medical and dental students (n = 130) enrolled in a required basic science course during the 1998-99 academic year were randomly assigned to one of two groups - each of which evaluated the teaching of 6 different faculty across 6 distinct dimensions (i.e. overall quality, organization, preparation, stimulation, respectfulness, and helpfulness). Using a 'split ballot' design, two versions of the conceptually equivalent faculty evaluation form were distributed at random to students in each group. Form A contained the 'traditional' items-within-faculty format, while Form B listed faculty-within-item. The number of monotonic forms (i.e. the identical rating of all 6 items) varied measurably across faculty ratees, as did the respective effects on scale reliabilities. Alpha was especially inflated where a sizeable proportion of monotonic patterns were located on response categories that were either very high (> +1.28 z m deviations) or very low (< -1.28 z m deviations) compared to the group mean. Lastly, the prevalence of monotonic response patterns was significantly (p = ≤ 0.01) less when a faculty-within-item format is used (Form B). These findings suggest that monotonic response patterns differentially impact the reliabilities and, hence, the validity of students' ratings of individual faculty in a multi-instructor context.
KW - effects
KW - faculty evaluation
KW - monotonic response patterns
KW - multi-instructor course
KW - nonattending behaviors
KW - response
KW - validity
UR - http://www.scopus.com/inward/record.url?scp=9244261111&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=9244261111&partnerID=8YFLogxK
U2 - 10.1023/A:1015735813922
DO - 10.1023/A:1015735813922
M3 - Article
C2 - 12075143
AN - SCOPUS:9244261111
SN - 1382-4996
VL - 7
SP - 99
EP - 116
JO - Advances in Health Sciences Education
JF - Advances in Health Sciences Education
IS - 2
ER -