INTERRATER RELIABILITY RECONSIDERED: PERFORMANCE ASSESSMENT USING ONE EXAMINER PER CANDIDATE

Title INTERRATER RELIABILITY RECONSIDERED: PERFORMANCE ASSESSMENT USING ONE EXAMINER PER CANDIDATE
Author, Co-Author Leon Gross
Topic
Year
1992
Day
Sunday
Program Number
2:00 pm
Room
Scotland A
Affiliation
Abstract The purpose of this study was to determine if adequate levels of interrater reliability could be attained on a national, standardized, clinical skills examination, using one examiner per observation. Traditionally, clinical skills assessments have relied on two examiners because of the subjectivity involved in grading. Cost and logistical issues, however, challenge the feasibility of using two examiners. The data were obtained during the July 1991 National Board Clinical Skills examination. During two sessions of the examination administered in Philadelphia, surplus examiners were available. They were given an honorarium to participate as secondary examiners in the study. Candidates were informed that there would be a second examiner, but that this individual was participating for statistical purposes only, not for evaluation or scoring purposes. Data were obtained for 101 paired candidate observations across four examination stations. The results indicated high levels of interrater reliability. At each station, the agreement between examiners in their dichotomous yes/no observations was 90% or greater. The agreement among examiner-determined scores based on the differentially weighted items was greater. These results indicate that adequate interrater reliability is being attained on the profession's clinical skills examination with only one examiner per observation. This high reliability allows for significantly fewer examiners to be present, thus reducing test center congestion, and examination costs. The ultimate result is that candidates are being examined in a consistent manner, regardless of the examiner with whom they are paired.
Affiliation of Co-Authors
Outline