Item bias in a test of reading comprehension

Abstract

The possibility that certain features of items on a reading comprehension test may lead to biased estimates of the reading achievement of particular subgroups of students was investigated. Eight nonoverlapping subgroups of students were defined by the combinations of three factors: student grade level (fifth or sixth), income level of the neighborhood in which the school was located (low and middle or above), and race of the student (black or white). Estimates of student ability and item parameters were obtained separately for each of the eight subgroups using the three-parameter logistic model. Bias indices were computed based on differences in item characteristic curves for pairs of subgroups. A criterion for labeling an item as biased was developed using the distribution of bias indices for subgroups of the same race that differed only in income level or grade level. Using this criterion, three items were consistently identified as biased in four independent comparisons of subgroups of black and white students. Comparisons of content and format characteristics of items that were identified as biased with those that were not, or between items biased in different directions, did not lead to the identification of any systematic content differences. The study did provide strong support for the viability of the estimation procedure; item characteristics, estimated with samples from different populations were very similar. Some suggestions for improvements in methodology are offered.