DSpace DSpace

University of Minnesota Digital Conservancy >
University of Minnesota - Twin Cities >
Applied Psychological Measurement >
Volume 15, 1991 >

Please use this identifier to cite or link to this item: http://hdl.handle.net/11299/114470

Title: Adjustments for rater effects in performance assessment
Authors: Houston, Walter M.
Raymond, Mark R.
Svec, Joseph C.
Issue Date: 1991
Citation: Houston, Walter M, Raymond, Mark R & Svec, Joseph C. (1991). Adjustments for rater effects in performance assessment. Applied Psychological Measurement, 15, 409-421. doi:10.1177/014662169101500411
Abstract: Alternative methods to correct for rater leniency/stringency effects (i.e., rater bias) in performance ratings were investigated. Rater bias effects are of concern when candidates are evaluated by different raters. The three correction methods evaluated were ordinary least squares (OLS), weighted least squares (WLS), and imputation of the missing data (IMPUTE). In addition, the usual procedure of averaging the observed ratings was investigated. Data were simulated from an essentially τ-equivalent measurement model, with true scores and error scores normally distributed. The variables manipulated in the simulations were method of correction (OLS, WLS, IMPUTE, averaging the observed ratings), amount of missing data (50% missing, 75% missing), rater bias (low, high), and number of examinees or candidates (N = 50, N = 100). The accuracy of the methods in estimating true scores was assessed based on the square root of the average squared difference between the estimated and known true scores. The three correction methods consistently outperformed the procedure of averaging the observed ratings. IMPUTE was superior to the least squares methods. Index terms: EM algorithm, incomplete data, incomplete rating designs, least squares adjustments, performance assessment, rater calibration.
URI: http://purl.umn.edu/114470
Appears in Collections:Volume 15, 1991

Files in This Item:

File Description SizeFormat
v15n4p409.pdf868.36 kBPDFView/Open

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.