Analyzing test content using cluster analysis and multidimensional scaling

Abstract

A new method for evaluating the content representation of a test is illustrated. Item similarity ratings were obtained from content domain experts in order to assess whether their ratings corresponded to item groupings specified in the test blueprint. Three expert judges rated the similarity of items on a 30-item multiple-choice test of study skills. The similarity data were analyzed using a multidimensional scaling (MDS) procedure followed by a hierarchical cluster analysis of the MDS stimulus coordinates. The results indicated a strong correspondence between the similarity data and the arrangement of items as prescribed in the test blueprint. The findings suggest that analyzing item similarity data with MDS and cluster analysis can provide substantive information pertaining to the content representation of a test. The advantages and disadvantages of using MDS and cluster analysis with item similarity data are discussed. Index terms: cluster analysis, content validity, multidimensional scaling, similarity data, test construction.