How large must a treatment effect be before it matters to practitioners? An estimation method and demonstration.

Treatment research is sometimes criticized as lacking in clinical relevance, and one potential source of this friction is a disconnection between statistical significance and what clinicians regard to be a meaningful difference in outcomes. This report demonstrates a novel methodology for estimating what substance abuse practitioners regard to be clinically important differences. To illustrate this new estimation method, the researchers surveyed 50 substance abuse treatment providers participating in the National Drug Abuse Treatment Clinical Trials Network (CTN). Practitioners identified thresholds for clinically meaningful differences on nine common outcome variables, indicated the size of effect that would justify their learning a new treatment method, and estimated current outcomes from their services. Clinicians judged a difference between two treatments to be meaningful if outcomes were improved by 10-12 points on the percentage of patients totally abstaining, arrested for driving while intoxicated, employed, or having abnormal liver enzymes. A 5 percentage-point reduction in patient mortality was also regarded as clinically significant. On continuous outcome measures (such as percentage of days abstinent or drinks per drinking day), practitioners judged an outcome to be significant when it doubled or halved the base rate. When a new treatment meets such criteria, practitioners were interested in learning it.

This report shows that effects that are statistically significant in clinical trials may be unimpressive to practitioners. Clinicians’ judgments of meaningful differences can inform the powering of clinical trials.