Generalizability of the findings from a randomized controlled trial of a web-based substance use disorder intervention.
There is growing concern regarding the generalizability of findings from randomized controlled trials (RCTs) of interventions for substance use disorders (SUDs). This study used a selection model approach to assess and improve the generalizability of an evaluation for a web-based SUD intervention by making the trial sample resemble the target population.
The sample of the web-based SUD intervention (Therapeutic Education System vs. treatment-as-usual; n=507) was compared with the target population of SUD treatment-seeking individuals from the Treatment Episodes Data Set-Admissions (TEDS-A). Using weights based on the probabilities of RCT participation, weighted treatment effects on retention and abstinence were computed.
Substantial differences between the RCT sample and the target population were demonstrated in significant difference in the mean propensity scores (1.62 standard deviations at p<.001). The population effect on abstinence (12 weeks and 6 months) was statistically insignificant after weighting the data with the generalizability weight.
Conclusions: Findings from this study provide insights into the differences between RCT participants of a web-based SUD intervention and the target population from recent years. The results of this study also indicate how poor sample representativeness of the RCT compared with the target population impacted the observed findings of the web-based SUD intervention. Given the great potential for scalability of web-based SUD interventions, the representativeness of the sample with regard to the target population of potential users for the intervention should be carefully considered. Additionally, with a careful consideration of the treatment effect modifiers which are over- or under-representing in RCTs, application of generalizability weights could be a potentially useful tool for assessing and improving the generalizability of the findings from RCTs when the RCT sample does not well-represent the target population.
Related protocols: CTN-0044