The effect of number of rating scale categories on levels of interrater reliability: A monte carlo investigation

Abstract

A computer simulation study was designed to investigate the extent to which the interrater reliability of a clinical scale is affected by the number of categories or scale points (2, 3, 4, ... ,100). Results indicate that reliability increases steadily up to 7 scale points, beyond which no substantial increases occur, even when the number of scale points is increased to as many as 100. These findings hold under the following conditions: (1) The research investigator has insufficient a priori knowledge to use as a reliable guideline for deciding on an appropriate number of scale points to employ, and (2) the dichotomous and ordinal categories being considered all have an underlying metric or continuous scale format.