Rating scales have no inherent reliability that is
independent of the observers who use them. The
often reported interrater reliability is an average of
perhaps quite different individual rater reliabilities.
It is possible to separate out the individual rater
reliabilities given a number of independent raters
who observe the same sample of ratees. Under certain
assumptions, an external measure can replace
one of the raters, and individual reliabilities of two
independent raters can be estimated. In a somewhat
similar fashion, estimates of treatment effects present
in ratings by two independent raters can provide
the external frame of reference against which differences
in their individual reliabilities can be
evaluated. Models for estimating individual rater
reliabilities are provided for use in selecting, evaluating,
and training participants in clinical research.
Index terms: attenuation, correlation, individual
raters, interrater reliability, multiple raters, rater reliability,
rating scales, reliability of ratings, significance.