Comparability of multiple rank order and paired comparison methods

Abstract

Two studies were conducted to compare multiple rank order and paired comparison methods in terms of psychometric characteristics and user reactions. For both studies, stimuli from the Minnesota Importance Questionnaire (MIQ) were cast in multiple rank order and paired comparison forms and were administered to subjects on two occasions (test-retest) in a counterbalanced design. For the multiple rank order form, item blocks of three stimuli were used in the first study (N = 158, retest after one week), and item blocks of five stimuli in the second study (N = 280, retest after two days). Individual and group item responses, preference counts, and Thurstone normal transform scale values obtained by the multiple rank order method were found to be very similar to those obtained by paired comparisons. Administration time decreased as number of stimuli in the item block increased. Two-thirds of the subjects preferred the multiple rank order method. The equivalence of the two methods is discussed, along with suggestions for further research.