Adaptive profile difference analysis with applications to personality asessment

Abstract

The adaptive measurement of change (AMC) framework uses item response theory (IRT) and computerized adaptive testing (CAT) to detect psychometrically significant change between two or more occasions for a single individual. In recent years, AMC has been extended to include novel omnibus hypothesis tests for detecting change; multiple measurement occasions; polytomous IRT models; and multidimensional IRT models. In addition, numerous AMC studies support AMC’s ability to detect change with high accuracy. One unexplored application of AMC is to the detection of intra-individual, psychometrically significant differences among multiple traits for measurements obtained at a single occasion. Rather than administering one CAT on each occasion for a single trait, a CAT would instead be administered for each trait on one occasion with AMC’s hypothesis tests applied to detect significant differences among traits using IRT-based trait estimates. For example, if one individual’s score on a measure of Extraversion differs significantly from the same individual’s score on a measure of Agreeableness, knowing whether these two personality traits differ significantly could provide useful information about an individual’s personality tendencies. Extending the concept to all Big Five personality traits, understanding how such traits differ within a single person could be used to tailor job training or educational interventions. More generally, this procedure, denoted adaptive profile difference analysis (APDA), could improve the objective interpretation of multiscale assessments. In this study, AMC omnibus hypothesis tests were applied to detect intra-individual differences across multiple traits. A Monte Carlo simulation study was conducted using synthetic data based on three real personality datasets. Nine design factors were varied to examine APDA under various realistic conditions. Two primary outcome measures included the true positive rate (i.e., the proportion of true differences over the total number of detected differences) and the false positive rate (i.e., the proportion of detected differences that are significant under conditions where there is no true difference). Findings indicate that APDA is viable under certain conditions, particularly for personality multiscale assessments. Based on these results, recommendations for assessment design and future research are provided.