Models with interactions overestimated heterogeneity of treatment effects and were prone to treatment mistargeting

Objectives: We aimed to compare the performance of different regression modeling approaches for the prediction of heterogeneous treatment effects. Study Design and Setting: We simulated trial samples (n = 3,600; 80% power for a treatment odds ratio of 0.8) from a superpopulation (N = 1,000,000) with 12 binary risk predictors, both without and with six true treatment interactions. We assessed predictions of treatment benefit for four regression models: a “risk model” (with a constant effect of treatment assignment) and three “effect models” (including interactions of risk predictors with treatment assignment). Three novel performance measures were evaluated: calibration for benefit (i.e., observed vs. predicted risk difference in treated vs. untreated), discrimination for benefit, and prediction error for benefit. Results: The risk modeling approach was well-calibrated for benefit, whereas effect models were consistently overfit, even with doubled sample sizes. Penalized regression reduced miscalibration of the effect models considerably. In terms of discrimination and prediction error, the risk modeling approach was superior in the absence of true treatment effect interactions, whereas penalized regression was optimal in the presence of true treatment interactions. Conclusion: A risk modeling approach yields models consistently well calibrated for benefit. Effect modeling may improve discrimination for benefit in the presence of true interactions but is prone to overfitting. Hence, effect models—including only plausible interactions—should be fitted using penalized regression.