Linked raters' judgments: Combating problems of statistical conclusion validity

Abstract

The traditional procedure for obtaining judged ratings, to ascertain if treatment-related change has occurred, involves the randomization of the materials to be rated. An alternative approach (linked judgments) is investigated as a potential solution to certain instrumentation- related threats to statistical conclusion validity of the incumbent rating procedure. Data from a weight reduction study are presented which suggest that linked raters’ judgments provide both a more powerful and a more valid index of treatment effectiveness than the traditional procedure.