Estimating individual rater reliabilities

Abstract

Rating scales have no inherent reliability that is independent of the observers who use them. The often reported interrater reliability is an average of perhaps quite different individual rater reliabilities. It is possible to separate out the individual rater reliabilities given a number of independent raters who observe the same sample of ratees. Under certain assumptions, an external measure can replace one of the raters, and individual reliabilities of two independent raters can be estimated. In a somewhat similar fashion, estimates of treatment effects present in ratings by two independent raters can provide the external frame of reference against which differences in their individual reliabilities can be evaluated. Models for estimating individual rater reliabilities are provided for use in selecting, evaluating, and training participants in clinical research. Index terms: attenuation, correlation, individual raters, interrater reliability, multiple raters, rater reliability, rating scales, reliability of ratings, significance.