Data from the California Learning Assessment System
are used to examine certain characteristics of tests
designed as the composites of items of different modes.
The characteristics include rater severity, test information,
and definition of the latent variable. Three different
assessment modes-multiple-choice, open-ended, and
investigation items (the latter two are referred to as performance-
based modes)-were combined in a test across
three different test forms. Rater severity was investigated
by incorporating a rater parameter for each rater in
an item response model that then was used to analyze
the data. Some rater severities were found to be quite
extreme, and the impact of this variation in rater severities
on both total scores and trait level estimates was examined.
Within-rater variation in rater severity also was
examined and was found to have significant variation.
The information contribution of the three modes was
compared. Performance-based items provided more information
than multiple-choice items and also provided
greatest precision for higher levels of the latent variable.
A projection-like method was applied to investigate the
effects of assessment mode on the definition of the latent
variable. The multiple-choice items added information
to the performance-based variable. The results of
the analysis also showed that the projection-like method
did not practically differ from the results when the latent
trait was defined jointly by both the multiple-choice and
the performance-based items. Index terms: equating,
linking, multiple assessment modes, polytomous item response
models, rater effects.
Wilson, Mark & Wang, Wen-chung. (1995). Complex composites: Issues that arise in combining different modes of assessment. Applied Psychological Measurement, 19, 51-71. doi:10.1177/014662169501900107
Wilson, Mark; Wang, Wen-chung.
Complex composites: Issues that arise in combining different modes of assessment.
Retrieved from the University of Minnesota Digital Conservancy,
Content distributed via the University of Minnesota's Digital Conservancy may be subject to additional license and use restrictions applied by the depositor.