A Monte Carlo Study of the Effects of Number of Clusters and Level-2 Residual Distributions on Multilevel Models

Abstract

Hierarchical Linear Modeling (HLM) has become an important approach to analyzing hierarchically structured data, which is common in educational research. But the accuracy of estimators and precision of statistical inference of HLM rely heavily on sufficiently large numbers of clusters, as well as the normality assumption of the residual distributions. The current study had two purposes. First, to synthesize the existing Monte Carlo research literature and identify gaps in the recommended number of clusters. This synthesis prompted two research questions with important implications for educational data analyses involving HLM: 1) What is the minimum required number of clusters for accurate estimation of level-2 parameters when assumptions are satisfied? 2) What is the minimum required number of clusters for accurate estimation of level-2 parameters when assumptions are violated? Much of the rationale for identifying minimum values of J for realistic data conditions is because clusters often require significant resources, leading to an interest in identifying a minimum J. To answer the research questions a Monte Carlo study was used to provide comprehensive recommendations for the minimum required sample size at level-2 of a two-level model for cross-sectional data. In order to fill the gaps of previous literature, the study adopted Latin Hypercube Sampling in the design of the simulation so that the sample sizes of both levels were randomly sampled from a wide range to mirror environments commonly found in educational research. A total of 40 combinations of J and n_j × 3 levels of ICC × 4 level-2 residual distributions × 4 covariate correlates = 1,920 combinations of conditions were studied. Bias in estimating fixed effects and variance components via ABs, ARBs, ln(RMSE)s, as well as Type I error rate and statistical power for corresponding statistical tests of those parameters, were investigated. The results showed that the fixed effects estimates were unbiased and were more accurately estimated when the number of clusters increased. A larger J was required for accurate Type I error rates of tests of fixed effects. In general, the fixed effects had sufficiently large statistical power. On the other hand, J > 75 was required for accurate variance components estimates and J > 100 was required for acceptable Type I error rates. Additionally, variance components were underpowered unless the sample sizes at both levels were large (J>100 and n_j>30) and ICC was bigger than .10. Finally, this current study provided guidance on minimum required sample size for future empirical research.