Cluster analyzing profile data confounded with interrater differences: A comparison of profile association measures

Abstract

Seven association measures were compared for their effectiveness in relating and clustering data profiles confounded with interrater differences. Among these association indices were three distance measures, two measures of angular separation, and two measures of profile overlap. The objects of analysis were 50 jobs that had been rated by different analysts on the items comprising the Occupation Analysis Inventory (OAI). Factor scores based on the OAI job ratings provided the profile data. Each of the seven association measures was applied to all pairwise combinations of factor-score profiles in the job sample, and the seven resultant 50 x 50 job proximity matrices were each subjected to hierarchical cluster analysis. The job proximity matrices and cluster structures based on the different association measures were then compared, respectively, with a criterion proximity matrix and a criterion cluster structure. In relation to these two criteria, the angular measures (product-moment correlation and cosine) performed better than the distance and overlap measures. The results demonstrate the importance of the choice of a profile association measure in cluster analysis. The researcher should be especially cautious when clustering entities that have been rated by different judges. Under such circumstances, it might be advisable to cluster analyze a data set using more than one association measure and then to compare the alternative solutions for clarity and stability.