Treatment adherence and competency ratings among therapists, supervisors, study-related raters and external raters in a clinical trial of 12-step facilitation for stimulant users.

This ancillary investigation of data from National Drug Abuse Treatment Clinical Trials Network protocol CTN-0031 (“Stimulant Abuser Groups to Engage in 12-Step (STAGE 12)”) investigated the correspondence among four groups of raters on adherence to STAGE-12, a manualized 12-step facilitation (TSF) group and individual treatment targeting stimulant abuse. The four rater groups included the study therapists, supervisors, study-related (“TSF expert”) raters, and non-project-related (“external”) raters. Results indicated that external raters rated most critically Mean Adherence — the mean of all the adherence items — and global performance. External raters also demonstrated the highest degree of reliability with the designated expert. Therapists rated their own adherence lower, on average, than did supervisors and TSF expert raters, but therapist ratings also had the poorest reliability.

Conclusions: Findings highlight the challenges in developing practical, but effective methods of fidelity monitoring for evidence-based practice in clinical settings. While funding and licensing agencies increasingly call for use of evidence-based treatments, community-based organizations implementing them will seek the simplest, most reliable and cost-effective ways of monitoring their delivery. These results suggest that there may be a role for on-site therapists or supervisors rating adherence, and that raters unaffiliated with the treatment being tested may provide the most objective ratings. Future research should examine the impact of training therapists on self-rating to determine whether this group and achieve acceptable reliability and objectivity in ratings.

Related protocols: CTN-0031