Assess
In this domain, we conduct research to understand:
How researchers and practitioners are assessing implementation.
What assessment options are feasible, efficient, and defensible.
How to use implementation data to make decisions about support.
Reviews of implementation assessment in research
To make valid decisions about internal validity, both student outcome and implementation data need to be assessed and analyzed.
Sanetti, L. M. H., Charbonneau, S., Knight, A., Cochrane, W., Kulcyk, M., & Kraus, K. (2020). Treatment fidelity reporting in intervention outcome studies in the school psychology literature from 2009 to 2016. Psychology in the Schools, 57, 901-922. https://doi.org/10.1002/pits.22364
73% of school psychology intervention evaluation studies published between 2009-2016 included implementation data; a 38% increase from 2011.
Collier-Meek, M. A., Fallon, L. M., & Gould, K. (2018). How are treatment integrity data assessed? Reviewing the performance feedback literature. School Psychology Quarterly, 33, 517-526. DOI: 10.1037/spq0000239
Implementation data were collected most frequently via direct observation by a researcher.
Sanetti, L. M. H., Dobey, L. M., & Gallucci, J. (2014). Treatment integrity of interventions with children in School Psychology International from 1995-2010. School Psychology International, 35, 370-383.
81% of intervention evaluation studies published in School Psychology International between 1995-2010 did not include implementation data.
Sanetti, L. M. H., Dobey, L., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999 to 2009. Journal of Positive Behavior Interventions. 14, 29-46.
51% of school psychology intervention evaluation studies published between 1999-2009 did not include implementation data.
Reviews of implementation assessment in practice
To make valid decisions about whether a student is benefiting from an intervention, both student outcome and implementation data need to be assessed and analyzed.
Cochrane, W. S., Sanetti, L. M. H., & *Minster, M. C. (2019). School psychologists’ beliefs and practices about treatment integrity in 2008 and 2017. Psychology in the Schools, 56, 295-306.
98% of respondents agreed implementation data are critical for tiered interventions. Only 7% of respondents collected implementation data in their 1:1 consultation and 0% in tiered team consultation.
.
Comparing implementation assessment methods
A wide range of factors can influence the accuracy of implementation data and usability of implementation assessment methods.
Collier-Meek, M. A., Sanetti, L. M. H., Gould, K., & Pereira, B. (2021). Comparing the utility of three direct treatment fidelity assessment methods: Time sampling, event recording, and post-observation rating scale. Journal of Educational and Psychological Consultation. doi: 10.1080/10474412.2020.1777874
Implementation data were relatively similar across assessment methods providing evidence of concurrent validity.
Sanetti, L. M. H., & Collier-Meek, M. A. (2014). Increasing the rigor of procedural fidelity assessment: An empirical comparison of direct observation and permanent product review methods. Journal of Behavioral Education, 23, 60-88.
There can be considerable differences in implementation estimates based on type of assessment method used.
Collier-Meek, M. A., Sanetti, L. M. H., Fallon, L. M., & Chafouleas, S. M. (2020). Exploring the influences of assessment method, intervention steps, intervention sessions, and observation timing on treatment fidelity estimates. Assessment for Effective Intervention, 45 (1), 14-26. doi: 10.1177/1534508419857228
Assessment method, intervention steps assessed, intervention sessions accounted for, and observation timing each influenced implementation estimates.
Sanetti, L. M. H., Chafouleas, S. M., O’Keeffe, B. V., & Kilgus, S. P. (2013). Treatment integrity assessment of a daily report card intervention: A preliminary evaluation of two methods and frequencies. Canadian Journal of School Psychology, 28, 261-276.
Frequency and method of implementation self-report may be related to accuracy of teacher report.
Collier-Meek, M. A., Fallon, L. M., & DeFouw, E. R. (2020). Assessing implementation of the Good Behavior Game: Comparing estimates of adherence, quality, and exposure. Assessment for Effective Intervention, 45, 95-109. doi: 10.1177/1534508418782620
Implementation data estimates related to specific dimensions (adherence, quality, exposure) were impacted by assessment method.
Making data-based decisions
Understanding assessment methods and outcomes can improve decisions about intervention effectiveness and need for implementation support.
Dart, E. H., Collier-Meek, M. A., Chambers, C., & Murphy, A. (2020). Multi-informant assessment of treatment integrity in the classroom. Psychology in the Schools, 57, 805-822. doi: 10.1002/pits.22351
Elementary school students can be trained and supported to accurately collect implementation data for a class-wide intervention.
Fallon, L. M., Sanetti, L. M. H., Chafouleas, S. M., Faggella-Luby, M. N., & Briesch, A. M. (2018). Direct training to increase agreement between teachers’ and observers’ treatment integrity ratings. Assessment for Effective Intervention, 43, 196-211.