Tolerance intervals: Alternatives to credibility intervals in validity generalization research

Abstract

In validity generalization research, the estimated mean and variance of the true validity distribution are often used to construct a credibility interval, an interval containing a specified proportion of the true validity distribution. The statistical interpretation of this interval in the literature has varied between Bayesian and classical (frequentist) viewpoints. Credibility intervals are here discussed from the frequentist perspective. These are known as "tolerance intervals" in the statistical literature. Two new methods for constructing a credibility interval are presented. Unlike the current method of constructing the credibility interval, tolerance intervals have known performance characteristics across repeated applications, justifying confidence statements. The new methods may be useful in validity generalization research involving a small or moderate number of validation studies. Index terms: Bayesian statistics, Credibility intervals, Metaanalysis, Tolerance intervals, True validity distribution, Validity generalization.