Measuring Reliability in profile analysis.

Abstract

Profile analysis has been used practically to study assessments with subtests or strands. The variation in profile analysis can be divided into two kinds: variation due to profile level and variation due to profile pattern. The variation in the profile level or the level reliability is the proportion of total profile variation due to the true score variation in the level whereas the variation in the profile pattern or the pattern reliability is the proportion of total profile variation due to the true score variation in the pattern. Methods to compute the level reliability and the pattern reliability are described. The methods are demonstrated using two datasets: a short personality inventory and the Woodcock- Johnson Psychoeducational Battery II. The results showed that pattern reliabilities were higher than the level reliabilities in both the rating scale (r= 0.66) and the forced choice versions (r= 0.71) of the personality inventory while the level reliability was higher in the Woodcock- Johnson Psychoeducational Battery II (r= 0.93). Results demonstrated that when the variation from the pattern and level are unequal, it is critical for researchers to examine whether the level or the pattern has higher reliability and adequately explain the results.