This article reviews various recently suggested approaches to estimate the random effects distribution in a linear mixed model, i.e. (1) the smoothing by roughening approach of Shen and Louis,1(2) the semi-non-parametric approach of Zhang and Davidian,2(3) the heterogeneity model of Verbeke and Lesaffre3and (4) a flexible approach of Ghidey et al.4These four approaches are compared via an extensive simulation study. We conclude that for the considered cases, the approach of Ghidey et al.4often shows to have the smallest integrated mean squared error for estimating the random effects distribution. An analysis of a longitudinal dental data set illustrates the performance of the methods in a practical example.,
Statistical Methods in Medical Research
Erasmus MC: University Medical Center Rotterdam

Ghidey, W, Lesaffre, E.M.E.H, & Verbeke, G. (2010). A comparison of methods for estimating the random effects distribution of a linear mixed model. Statistical Methods in Medical Research, 19(6), 575–600. doi:10.1177/0962280208091686