Occasionally, an inter-rater reliability study must
be designed so that each subject is rated by fewer
than all the participating raters. If there is interest
in comparing the raters’ mean levels of rating, and
if it is desired that each mean be estimated with the
same precision, then a balanced incomplete block
design for the reliability study is indicated.
Methods for executing the design and for analyzing
the resulting data are presented, using data from
an actual study for illustration.