Abstract
Objective:
Implementation fidelity is a key dimension in process evaluation but has been scarcely studied. Reptes is a selective alcohol and cannabis use prevention program performed in vulnerable young people (16-21 years) from different educational and leisure settings. The study aimed to describe the components of this program, its implementation fidelity, and satisfaction among participants and facilitators.
Method:
A descriptive cross-sectional study was conducted. Implementation fidelity was assessed in terms of adherence, dose, participant responsiveness, and quality of the intervention through self-reports completed by 15 facilitators and self-administered questionnaires by 99 youth from the 15 groups completing the program.
Results:
70.7% of the groups performed 30 of the 43 planned activities with high use of various teaching methods, except for Information and Communication Technology resources. Fidelity implementation was above 70%. 2/3 groups completed an acceptable intervention and 1/3 completed a qualified intervention. Satisfaction was higher than 7.
Conclusions:
Our results show high implementation fidelity and satisfaction compared with those from similar studies.
Keywords:
Program evaluation; Health plan implementation; Alcohol drinking; Marijuana abuse; Vulnerable populations; Adolescent
Resumen
Objetivo:
La fidelidad en la implementación es una dimensión clave en el proceso de evaluación, pero ha sido escasamente estudiada. Reptes es un programa de prevención selectiva del consumo de alcohol y cánnabis dirigido a jóvenes vulnerables (16-21 años) de diferentes espacios educativos y de ocio y tiempo libre. El estudio pretende describir los componentes de este programa, su fidelidad de implementación y la satisfacción de los/las participantes y los/las facilitadores/as.
Método:
Se realizó un estudio descriptivo transversal. La fidelidad de implementación fue valorada en clave de adherencia, dosis, capacidad de respuesta de los/las participantes y calidad de la intervención a través de los informes autocumplimentados por los/las 15 facilitadores/as del programa y los cuestionarios autocompletados por los/las 99 jóvenes de los 15 grupos que completaron el programa.
Resultados:
El 70,7% de los grupos realizaron 30 de las 43 actividades planificadas, con un uso elevado de las estrategias de aprendizaje, excepto para los recursos basados en las tecnologías de la información y la comunicación. La fidelidad de implementación se situó por encima del 70%. Dos tercios de los grupos completaron una intervención aceptable y un tercio completó una intervención cualificada. La satisfacción se situó por encima de 7.
Conclusiones:
Nuestros resultados muestran una alta fidelidad en la implementación y la satisfacción en comparación con estudios similares.
Palabras clave:
Evaluación de programas; Implementación de planes de salud; Consumo de alcohol; Consumo de cannabis; Poblaciones vulnerables; Adolescentes
Introduction
The fidelity of implementation is the dimension of the process evaluation research that has been most investigated in the last decades. It is defined as “the degree to which programs are implemented according to the criteria with which they were designed”.11. Dusenbury L, Brannigan R, Falco M, et al. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2010;18:237-56.It measures how an intervention affects its results.22. Carroll Ch, Patterson M, Wood S, et al. A conceptual framework for implementation fidelity. Implementation Sci. 2007;2:40.Specifically, four basic indicators are defined to measure it: adherence; exposure or dose; quality of implementation and responsiveness (interest or involvement of participants).11. Dusenbury L, Brannigan R, Falco M, et al. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2010;18:237-56.
2. Carroll Ch, Patterson M, Wood S, et al. A conceptual framework for implementation fidelity. Implementation Sci. 2007;2:40.
3. Ennett ST, Haws S, Ringwalt CL, et al. Evidence-based practice in school substance use prevention: fidelity of implementation under real-world conditions. Health Educ Res. 2011;26:361-71.-44. Feagans Gould L, Dariotis JK, Greenberg MT, et al. Assessing Fidelity of Implementation (FOI) for school-based mindfulness and yoga interventions: a systematic review. Mindfulness (NY). 2016;7:5-33.Adaptation, the process of implementers bringing changes to the original design of an intervention, is also another component to be explored by implementation fidelity. Pérez et al.55. Pérez D, Van der Stuyft P, Zabala MC, et al. A modified theoretical framework to assess implementation fidelity of adaptive public health interventions. Implement Sci. 2016;8:91.proposed a theoretical framework to assess implementation fidelity in public health interventions.
The importance of knowing the intervention development process (everything that happens during the implementation of the program in the experimental phase) is today beyond doubt11. Dusenbury L, Brannigan R, Falco M, et al. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2010;18:237-56.,33. Ennett ST, Haws S, Ringwalt CL, et al. Evidence-based practice in school substance use prevention: fidelity of implementation under real-world conditions. Health Educ Res. 2011;26:361-71.. However, one of the major limitations to know the quality of implementation of the interventions is the scarcity of studies in the literature about the evaluation of the intervention process using rigorous implementation indicators.66. Slaughter SE, Hill JN, Snelgrove-Clarke E. What is the extent and quality of documentation and reporting of fidelity to implementation strategies: a scoping review. Implement Sci. 2015;7:129.A review44. Feagans Gould L, Dariotis JK, Greenberg MT, et al. Assessing Fidelity of Implementation (FOI) for school-based mindfulness and yoga interventions: a systematic review. Mindfulness (NY). 2016;7:5-33.of 312 publications identified that < 10% showed the main core or structural components of the program or the theoretical reference model; and < 20% measured the implementation fidelity, apart from the dose or amount of the applied intervention.
The Reptes program for the prevention of alcohol and cannabis use in young people with vulnerability is a selective drug prevention program. Selective drug prevention focuses on vulnerable groups and contexts in relation to the risk of transition to patterns of problematic drug use and to situations of social exclusion.77. Burkhart G. Selective prevention: first overview on the European situation. Lisbon: European Monitoring Centre for Drugs and Drug Addiction (EMCDDA); 2006.In these groups the use of different types of drug use is very common, so the intervention is aimed at reducing the risks of the contexts, the harm associated with drug use and at reinforcing the skills of the young people.77. Burkhart G. Selective prevention: first overview on the European situation. Lisbon: European Monitoring Centre for Drugs and Drug Addiction (EMCDDA); 2006.,88. Rhodes T, Lilly R, Fernández C, et al. Risk factors associated with drug use: the importance of "risk environment". Drugs Educ Prev Policy. 2003;10:303-29.The Reptes program seeks to provide knowledge on the main risk of uncritical cannabis and alcohol consumption; arguing skills regarding decision making on drug use; and to improve social skills to face social pressure to consume drugs.
The Reptes program's design was inspired: 1) in terms of communication styles, in the approach to self-control work through the role playing of assertive behaviors in the face of the consumption of addictive substances from the project Towards No Drug Abuse (TND);99. Sussman S, Dent CW, Stacy S. Project Towards No Drug Abuse: a review of the findings and future directions. Am J Health Behav. 2002;26:354-65.
10. Skara S, Rohrbach LA, Sun P, et al. An evaluation of the fidelity of implementation of a school-based drug abuse prevention program: Project Towards No Drug Abuse (TND). J Drug Educ. 2005;35:305.
11. Rohrbach LA, Gunning M, Sun P, et al. The Project Towards No Drug Abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77-88.
12. Lisha NE, Sun P, Rohrbach LA, et al. An evaluation of immediate outcomes and fidelity of a drug abuse prevention program in continuation high schools: Project Towards No Drug Abuse (TND). J Drug Educ. 2012;42:33-57.-1313. Little MA, Sussman S, Sun P, et al. The effects of implementation fidelity in the Towards No Drug Abuse Dissemination Trial. Health Educ (Lond). 2013;113:281-96.2) regarding the work of emotions, in the approach of identifying the risk associated with personality variables of the project Preventure,1414. Conrod PJ, Castellanos N, Strang J. Brief, personality-targeted coping skills interventions prolong survival as a non-drug user over a two-year period during adolescence. Arch Gen Psychiatry. 2010;67:85-93.and the work of examples based on personal narratives to enhance identification with the prevention messages of the Keepin'it REAL program,1515. Hecht ML, Marsiglia FF, Elek E, et al. Culturally-grounded substance use prevention: an evaluation of the Keepin' it REAL curriculum. Prev Sci. 2003;4:233-48.
16. Marsiglia FF, Hecht ML. Keepin'it REAL: an evidence-based program. (Teacher manual, video, and student work books in English and Spanish.). Santa Cruz, CA: ETR Associates; 2005.-1717. Pettigrew J, Miller-Day M, Shin Y, et al. Describing teacher-student interactions: a qualitative assessment of teacher implementation of the 7th grade keepin' it REAL substance use intervention. Am J Community Psychol. 2013;51:43-56.and 3) for the resolution of conflicts or problems also in the TND program99. Sussman S, Dent CW, Stacy S. Project Towards No Drug Abuse: a review of the findings and future directions. Am J Health Behav. 2002;26:354-65.and Keepin'it R.E.A.L.1616. Marsiglia FF, Hecht ML. Keepin'it REAL: an evidence-based program. (Teacher manual, video, and student work books in English and Spanish.). Santa Cruz, CA: ETR Associates; 2005.
The objective of this study is to describe main components, fidelity of implementation and satisfaction with the Reptes program in a sample of vulnerable young 16-21 years old enrolled in social guarantee training environments (oriented to the job market) and in community environments (in civic facilities) of a recreational nature, in Barcelona, during 2017.
Method
Study design and sampling
We conducted a descriptive cross-sectional process evaluation study of the Reptes program. All centers and organizations (n = 15) taking part in the intervention in 2017 participated in the evaluation study. The intervention was implemented by trained facilitators in 17 groups (184 participants). All facilitators and participants were included in the study after their agreement.
The intervention was implemented February-April 2017 among youth (16-21 years old) enrolled in social guarantee programs (SGP) or in neighborhood leisure organizations from vulnerable neighborhoods in Barcelona. SGP are usually offered in secondary schools or educational centers and aim to provide professional training to young people who have not completed their secondary schooling. Neighborhood organizations offer healthy leisure activities to neighborhood residents who might be living in a vulnerable context. Some educational centers and neighborhood organizations were invited in a convenience basis to implement the Reptes program in their centers. As inclusion criterion, people in charge of SGP and neighborhood organizations had to clearly consent to participate. Additionally, facilitators, who were responsible for implementation of the program, were urged to attend a previous 20-hour training program to learn how to deliver the intervention effectively. Finally, 15 centers or organizations agreed to participate.
Description of the intervention
Reptes is a selective prevention program targeting young people living in a vulnerable context. These young people usually have or have had academic difficulties within the regular education system, are at risk of leaving school early, have started experimental drug use, or live in families at risk. The program aims to provide knowledge on the main risks of uncritical cannabis and alcohol use, to provide decision-making skills on drug use, and to improve social skills to withstand social pressure to take drugs. The Reptes program consists of seven structured sessions and one entertaining closing session consisting of a party. The sessions were weekly 90minutes long. All sessions included 43 activities, imparted by facilitators previously and properly trained. Activities were conducted in classrooms.
The core components of the program were those essential for the successful of the intervention and consisted of the structural and crucial components of the intervention and of the teaching methods through which they should be imparted.1818. Fixsen DL, Blasé KA, Naoom SF, et al. Core implementation components. Res Soc Work Pract. 2009;19:531-40.,1919. Century J, Rudnick M, Freeman C. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval. 2010;31:199-218.Of the 43 activities included in the intervention, seven were core components. They focused on attitudes, emotions, social skills, decision making or problem solving, which are the main intermediate factors related to substance use. Two more activities were also important for the proper implementation of the intervention. One of these activities was focused on social network assessment of the young's family and friendship and the other one was an individual assessment and debate of the main issues discussed with the program (see Table I in online Appendix). Of note, the program had two different alternatives for some of the activities, namely minimum and maximum activities that were exclusionary. For these activities, the facilitator had to choose whether to implement the minimum or maximum activity depending on the maturity, concentration and cohesion of the group. The contents taught in both types of activities were the same. However, activities differed in the level of complexity of the teaching method applied.
Information sources and study variables
The information sources were self-report forms and questionnaires. Facilitators completed self-report forms throughout the intervention at the end of each delivered session. These forms recorded process information including the number of activities conducted, the teaching methods applied according to the protocol (oral explanation [OE], working skills and emotions [SE], discussion [D], active teaching and learning [ATL] and information and communications technology [ICT] methods), and satisfaction with the program. The completed forms were collected by the research team at the end of the intervention. Additionally, we administered an ad-hoc questionnaire to all young people participating in the intervention after the implementation of the program (June-July 2017). The questionnaire gathered items on satisfaction with different aspects of the intervention. Table 1shows the methods used and the indicators obtained for the process evaluation.
Description of indicators and instruments to evaluate the process implementation of the intervention.
Implementation fidelity was described in terms of adherence, dosage, responsiveness, and quality of the intervention. Adherence is the percentage of core activities implemented (n = 7). Dosage is the number of activities implemented, the total number being 43 activities. Responsiveness was determined based on three items related to the degree of agreement reported by facilitators regarding attention, comprehension and control of the group in each session. Participant engagement in the group was good when the facilitator reported fair or total agreement with the three items. Quality was assessed through five items: 1) relationship between facilitator and participants; 2) personal relationship between facilitators and participants; 3) the teacher's ability to motivate participants to engage with activities; 4) the teacher's ability to teach, and 5) the way the educator led the sessions. These items were reported by young participants of the intervention, who scored each item from 0 to 10 points.
For the overall assessment of delivery, we defined a compliance indicator that defined the implementation as non-acceptable, acceptable and qualified, using both quantitative and qualitative criteria. Acceptable implementation was determined by the dosage (quantitative criterion) and was defined as the performance of at least 30 out of the 43 activities of the program. Qualified implementation was determined by the dosage (quantitative criterion) and the implementation of important activities. In this case, maximum activity had to be conducted if applicable (qualitative criterion). Therefore, qualified implementation consisted of: 1) completion of at least 30 activities, and 2) completion of at least six out of nine important activities (seven core activities and the other two activities from sessions 3 and 7 considered as important (see Table I in online Appendix), and having completed maximum activity, if applicable. When the implementation was neither qualified nor acceptable, it was considered as non-acceptable.
To determine participants' satisfaction with the intervention, the questionnaire gathered information on satisfaction with group dynamics, the topics discussed, the program materials used, and overall satisfaction. Satisfaction was assessed using a 10-point scale. Finally, participants were asked if they would recommend the intervention to their friends and peers (yes/no/I don't know).
Statistical analyses
Descriptive statistics are presented as absolute frequencies and percentages for categorical variables and as means and standard deviation for continuous variables. All items on the degree of agreement were dichotomized into: 1) totally or fairly agree, and 2) neither agree nor disagree, and fairly or totally disagree. All statistical analyses were conducted using Stata v15.
Results
Fifteen out of the initial 17 groups completed the intervention (10 from SGP and 5 from neighborhood organizations [10 different institutions]). To evaluate the implementation process, we obtained responses from 15 facilitators (one from each group completing the intervention) and from 99 participants (response rate 54%).
The mean number of activities undertaken by all groups was 30.4 (out of a total of 43). The percentage of activities accomplished in each session (S) were: 82.2% (S1), 73.3% (S2), 68.9% (S3), 67.8% (S4), 69.5% (S5), 73.3% (S6), 69.3% (S7) and 43.3% (closing session) (data not shown). Various teaching methods were used to carry out the activities. Table 2shows the number and percentage of groups that used the planned methods for each of the sessions. The mean percentage of planned teaching methods used in all sessions were the following: 100% for oral explanation, 61.9% working on skills and emotions, 83.3% discussion, 78.9% active teaching and learning, and 35% for ICT.
As for facilitators' perception on the implementation and success of the program in their groups, 73.3% were satisfied with participants' attention, 77.1% with their comprehension, and 66.7% with control of the group. Overall satisfaction with the intervention was 7.4 out of 10.
Table 3shows information on the indicators used to assess the fidelity of implementation. The mean adherence to the intervention was 82.9%, indicating that a mean of 5.8 out of seven core activities were completed by the groups. Seven groups achieved an adherence of 100%. The mean number of completed activities was 30.4 (dosage). The mean percentage of facilitators who positively assessed responsiveness was over 65% (73.3% regarding attention, 77.1% comprehension and 66.7% control of the group). There were differences among groups in responsiveness, these differences being more evident in control of the group (degree of agreement below 50% for several sessions in five groups). The quality of implementation was rated with a mean scoring close to 8 points. The most highly rated aspect was the facilitator-participant relationship (8.3 points), followed by the way the educator led the sessions (8.1 points), and the teacher's ability to teach (8.1 points). The implementation of the intervention was at least acceptable (acceptable or qualified) in 66.6% and qualified in 33.3% of the groups.
Participant satisfaction with the intervention was higher than 7 out of 10 points (7.15 for group dynamics, 7.43 for topics raised and 7.7 for program materials) and overall satisfaction with the intervention was 8.04. Some 62.2% of youth would recommend the intervention to their peers (Table 4).
Discussion
Process evaluation of the implementation of the Reptes program allowed us to identify high compliance with the intervention and high use of the various teaching methods, except for ICT. The implementation of the proposed activities throughout the sessions was high, except for the closing session, consisting of a party, in which participation was lower than in previous sessions. Implementation fidelity was successful in line with what was found in previous reviews of similar programs.66. Slaughter SE, Hill JN, Snelgrove-Clarke E. What is the extent and quality of documentation and reporting of fidelity to implementation strategies: a scoping review. Implement Sci. 2015;7:129.The adherence, dosage, responsiveness and quality values were above 70% or 7.5 points out of 10. Interventions were acceptable in two out of three groups and was qualified in one out of three. Young people's satisfaction with the program was high.
In the process evaluation of the Reptes program, the authors learned three important lessons, which may be of use to other planners. First, the Reptes program was designed through a co-production process, based on several pre-existing interventions with proven evidence of effectiveness.99. Sussman S, Dent CW, Stacy S. Project Towards No Drug Abuse: a review of the findings and future directions. Am J Health Behav. 2002;26:354-65.,1414. Conrod PJ, Castellanos N, Strang J. Brief, personality-targeted coping skills interventions prolong survival as a non-drug user over a two-year period during adolescence. Arch Gen Psychiatry. 2010;67:85-93.,1616. Marsiglia FF, Hecht ML. Keepin'it REAL: an evidence-based program. (Teacher manual, video, and student work books in English and Spanish.). Santa Cruz, CA: ETR Associates; 2005.,2020. Espada JP, Méndez FX. Programa Saluda: prevención del abuso de alcohol y drogas de síntesis. Madrid, Spain: Pirámide; 2003.
21. Espada JP, Griffin KW, Pereira JR, et al. Component analysis of a school-based substance use prevention program in Spain: contributions of problem solving and social skills training content. Prev Sci. 2012;13:86-95.-2222. Newton NC, Teesson M, Mather M, et al. Universal cannabis outcomes from the Climate and Preventure (CAP) study: a cluster randomised controlled trial. Subst Abuse Treat Prev Policy. 2018;13:34.Through a step-by-step process2323. Hawkins J, Madden K, Fletcher A, et al. Development of a framework for the co-production and prototyping of public health interventions. BMC Public Health. 2017;17:689., it was based on the evidence and consultations with peers and intermediary professionals; pre-existing activities were adapted and new ones were co-produced; and the best ones were prototyped, according to acceptability and fidelity. The “core” of the intervention was concentrated in sessions 4, 5 and 6, and had to be delivered in all groups. More than half of the Reptes groups carried out activities concentrating on skills and emotions and about 80% carried out the interactive procedures typical of conflict resolution. Technologies were not frequently used in delivering the program, possibly due to certain technical problems, such as the lack of electronic devices or good internet connection, or the difficulty of adapting the content of the program to online activities. However, this difficulty was generally overcome with the use of more traditional alternative teaching methods.
Second, the Reptes program aimed for completeness in the measurement of implementation fidelity and a mixed indicator, compliance with the intervention, was created in a way that allowed, through two categories (acceptable and qualified) identification of the level of implementation achieved. Although the sample had some levels of truancy and irregular attendance at leisure and free time activities, average adherence was high, 83%; the average dose was 30.4 activities out of 43; the various student involvement parameters showed values between 66% and 77%; and the quality components were assigned between 7.8 and 8.3 points out of 10. In addition, 2/3 of the groups delivered an acceptable or qualified intervention, and in 1/3 of the groups it was a qualified intervention. This high fidelity of implementation in these groups could point to one of the strengths of the program. High values in these parameters have also been found in the measurement of the fidelity of implementation of other similar programs1010. Skara S, Rohrbach LA, Sun P, et al. An evaluation of the fidelity of implementation of a school-based drug abuse prevention program: Project Towards No Drug Abuse (TND). J Drug Educ. 2005;35:305.,1111. Rohrbach LA, Gunning M, Sun P, et al. The Project Towards No Drug Abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77-88.,2222. Newton NC, Teesson M, Mather M, et al. Universal cannabis outcomes from the Climate and Preventure (CAP) study: a cluster randomised controlled trial. Subst Abuse Treat Prev Policy. 2018;13:34..
Third, the Reptes program also studied participant satisfaction with the program, finding satisfaction levels higher than 7 points out of 10 in several of the aspects analyzed, with an average overall satisfaction with the program of 8 points, and 62% of participants indicated they would recommend the program to their peers. Similar scores were reported in many of the programs used as a reference by the Reptes program,1010. Skara S, Rohrbach LA, Sun P, et al. An evaluation of the fidelity of implementation of a school-based drug abuse prevention program: Project Towards No Drug Abuse (TND). J Drug Educ. 2005;35:305.,1111. Rohrbach LA, Gunning M, Sun P, et al. The Project Towards No Drug Abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77-88.,2424. Teesson M, Newton NC, Slade T, et al. Combined universal and selective prevention for adolescent alcohol use: a cluster randomized controlled trial. Psychol Med. 2017;17:1-10.and could in part be due to the high motivation of the facilitators when implementing the program.
Finally, it is necessary to address the controversial aspect about the differences between the implementation of an intervention under experimental conditions and its application under real conditions. In the evaluation phase of the process of the Reptes program, the results of the implementation indicators showed in this study were highly favorable. The main reasons were, probably, the strong motivation of the educators and their good control of the sessions. However, these findings might differ with the dissemination and generalization of the program. Ennett et al.33. Ennett ST, Haws S, Ringwalt CL, et al. Evidence-based practice in school substance use prevention: fidelity of implementation under real-world conditions. Health Educ Res. 2011;26:361-71.studied 10 top-quality programs and found very modest results for adherence (27.7%) and exposure (35.8%), with quality of implementation and involvement values higher than 80% in most programs. These findings confirm that many educators deliver the programs, but not necessarily according to the prescribed protocol.1111. Rohrbach LA, Gunning M, Sun P, et al. The Project Towards No Drug Abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77-88.,1717. Pettigrew J, Miller-Day M, Shin Y, et al. Describing teacher-student interactions: a qualitative assessment of teacher implementation of the 7th grade keepin' it REAL substance use intervention. Am J Community Psychol. 2013;51:43-56.,2525. Miller-Day M, Pettigrew J, Hecht ML, et al. How prevention curricula are taught under real-world conditions: types of and reasons for teacher curriculum adaptations. Health Educ (Lond). 2013;113:324-44.Therefore, tracking the fidelity of the implementation during the implementation of the program in real-world conditions will be necessary.
Limitations and strengths
One of the main limitations of this study is that the evaluation did not include the groups that left the program before completing the intervention. Likewise, there is no information on all participants in the groups that carried out the intervention, mainly due to data matching problems in the follow-up. This implies an information bias in the evaluation due to the unavailability of data from the participants who could have had the worst scores in the various parameters studied. Another limitation of this study is the high levels of fidelity found in the application of the program, attributable to a large extent to the high motivation of facilitators and their commitment to evaluation. This is a limitation because the dissemination of the program in real-world conditions, where motivation will probably not be as high in all settings, represents a challenge. However, it also represents a strength, since it has allowed comprehensive collection of implementation parameters that otherwise could not have been recorded and analyzed. Other strengths of the program are the scarcity of this type of selective prevention program in the reference environment, the rigor used in the design -based on programs with the strongest evidence in the specialized literature-, and the rigorousness of the data collection, both in the process indicators and in those related to the effect.
To conclude, this study found high compliance with the intervention, quality, and participant satisfaction. These results should be contrasted in the dissemination phase of the program through a follow-up study of the implementation in real-world conditions. The dissemination phase should have less complexity and detail but should identify whether fidelity of implementation and high participant satisfaction are maintained, especially in adherence and exposure, as pointed out by some of the cited authors. Finally, it will be important to analyze whether these favorable results in the process evaluation correspond to good results in the effects generated by the intervention in both intermediate and consumption variables of young people, as it was noticed in the programs that we used as reference.1111. Rohrbach LA, Gunning M, Sun P, et al. The Project Towards No Drug Abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77-88.
12. Lisha NE, Sun P, Rohrbach LA, et al. An evaluation of immediate outcomes and fidelity of a drug abuse prevention program in continuation high schools: Project Towards No Drug Abuse (TND). J Drug Educ. 2012;42:33-57.-1313. Little MA, Sussman S, Sun P, et al. The effects of implementation fidelity in the Towards No Drug Abuse Dissemination Trial. Health Educ (Lond). 2013;113:281-96.,2626. Sloboda Z, Stephens P, Pyakuryal A, et al. Implementation fidelity: the experience of the Adolescent Substance Abuse Prevention Study. Health Educ Res. 2009;24:394-406.
The importance of study the intervention development process of the program is today beyond doubt but the quality of implementation of the intervention is the scarcity of studies in the health literature that describe the intervention process and use rigorous implementation indicators
What does this study add to the literature?This study found high compliance with the intervention, quality, and participant satisfaction of Reptes, a selective prevention program of alcohol and cannabis in youngsters
What are the implications of the results?These results shown the importance of process evaluation. Future research should analyze whether these results correspond to good effects generated by the intervention in youngster consumption.
Availability of databases and material for replication
Databases and other materials are available upon request to the corresponding author.
Acknowledgments
We would like to thank all facilitators and young people for their participation in the study evaluation of the Reptes program. We would also like to thank Laia Clemente, Catalina Londoño and Laura López for their help in recording process data electronically.
References
- 1Dusenbury L, Brannigan R, Falco M, et al. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2010;18:237-56.
- 2Carroll Ch, Patterson M, Wood S, et al. A conceptual framework for implementation fidelity. Implementation Sci. 2007;2:40.
- 3Ennett ST, Haws S, Ringwalt CL, et al. Evidence-based practice in school substance use prevention: fidelity of implementation under real-world conditions. Health Educ Res. 2011;26:361-71.
- 4Feagans Gould L, Dariotis JK, Greenberg MT, et al. Assessing Fidelity of Implementation (FOI) for school-based mindfulness and yoga interventions: a systematic review. Mindfulness (NY). 2016;7:5-33.
- 5Pérez D, Van der Stuyft P, Zabala MC, et al. A modified theoretical framework to assess implementation fidelity of adaptive public health interventions. Implement Sci. 2016;8:91.
- 6Slaughter SE, Hill JN, Snelgrove-Clarke E. What is the extent and quality of documentation and reporting of fidelity to implementation strategies: a scoping review. Implement Sci. 2015;7:129.
- 7Burkhart G. Selective prevention: first overview on the European situation. Lisbon: European Monitoring Centre for Drugs and Drug Addiction (EMCDDA); 2006.
- 8Rhodes T, Lilly R, Fernández C, et al. Risk factors associated with drug use: the importance of "risk environment". Drugs Educ Prev Policy. 2003;10:303-29.
- 9Sussman S, Dent CW, Stacy S. Project Towards No Drug Abuse: a review of the findings and future directions. Am J Health Behav. 2002;26:354-65.
- 10Skara S, Rohrbach LA, Sun P, et al. An evaluation of the fidelity of implementation of a school-based drug abuse prevention program: Project Towards No Drug Abuse (TND). J Drug Educ. 2005;35:305.
- 11Rohrbach LA, Gunning M, Sun P, et al. The Project Towards No Drug Abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77-88.
- 12Lisha NE, Sun P, Rohrbach LA, et al. An evaluation of immediate outcomes and fidelity of a drug abuse prevention program in continuation high schools: Project Towards No Drug Abuse (TND). J Drug Educ. 2012;42:33-57.
- 13Little MA, Sussman S, Sun P, et al. The effects of implementation fidelity in the Towards No Drug Abuse Dissemination Trial. Health Educ (Lond). 2013;113:281-96.
- 14Conrod PJ, Castellanos N, Strang J. Brief, personality-targeted coping skills interventions prolong survival as a non-drug user over a two-year period during adolescence. Arch Gen Psychiatry. 2010;67:85-93.
- 15Hecht ML, Marsiglia FF, Elek E, et al. Culturally-grounded substance use prevention: an evaluation of the Keepin' it REAL curriculum. Prev Sci. 2003;4:233-48.
- 16Marsiglia FF, Hecht ML. Keepin'it REAL: an evidence-based program. (Teacher manual, video, and student work books in English and Spanish.). Santa Cruz, CA: ETR Associates; 2005.
- 17Pettigrew J, Miller-Day M, Shin Y, et al. Describing teacher-student interactions: a qualitative assessment of teacher implementation of the 7th grade keepin' it REAL substance use intervention. Am J Community Psychol. 2013;51:43-56.
- 18Fixsen DL, Blasé KA, Naoom SF, et al. Core implementation components. Res Soc Work Pract. 2009;19:531-40.
- 19Century J, Rudnick M, Freeman C. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval. 2010;31:199-218.
- 20Espada JP, Méndez FX. Programa Saluda: prevención del abuso de alcohol y drogas de síntesis. Madrid, Spain: Pirámide; 2003.
- 21Espada JP, Griffin KW, Pereira JR, et al. Component analysis of a school-based substance use prevention program in Spain: contributions of problem solving and social skills training content. Prev Sci. 2012;13:86-95.
- 22Newton NC, Teesson M, Mather M, et al. Universal cannabis outcomes from the Climate and Preventure (CAP) study: a cluster randomised controlled trial. Subst Abuse Treat Prev Policy. 2018;13:34.
- 23Hawkins J, Madden K, Fletcher A, et al. Development of a framework for the co-production and prototyping of public health interventions. BMC Public Health. 2017;17:689.
- 24Teesson M, Newton NC, Slade T, et al. Combined universal and selective prevention for adolescent alcohol use: a cluster randomized controlled trial. Psychol Med. 2017;17:1-10.
- 25Miller-Day M, Pettigrew J, Hecht ML, et al. How prevention curricula are taught under real-world conditions: types of and reasons for teacher curriculum adaptations. Health Educ (Lond). 2013;113:324-44.
- 26Sloboda Z, Stephens P, Pyakuryal A, et al. Implementation fidelity: the experience of the Adolescent Substance Abuse Prevention Study. Health Educ Res. 2009;24:394-406.
- 27Resnicow K, Davis M, Smith M, et al. How best to mesuare implementation of school health curricula: a comparision of three measures. Health Educ Res. 1998;13:239-50.
Appendix A. Supplementary material
Supplementary material associated with this article can be found in the online version available at doi: 10.1016/j.gaceta.2023.102323.
The Reptes Project investigators
Carles Ariza, Montse Bartroli, Xavier Continente, Olga Juárez, Maria José López, Laura Muñoz, Anna Pérez, Francesca Sánchez-Martínez, Noelia Vázquez and Joan Ramon Villalbí.Ethics approval
This study was performed in accordance with the ethical standards of the Parc de Salut Mar Clinical Research Ethics Committee (2015/6416/I) and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.Consent to participate
All participants were previously informed about the study ant they signed an informed consent to be able to participate in the study and complete questionnaires.Funding
This study was supported by the Government Delegation for the National Plan on Drugs (Grant 2015/070) (Ministry of Health, Social Services and Equality). The study was also partially supported by the Ministry of Universities and Research of Catalonia (AGAUR) (2017SGR1526).
Publication Dates
- Publication in this collection
17 Mar 2025 - Date of issue
2023
History
- Received
14 Dec 2022 - Accepted
25 June 2023 - Published
18 Aug 2023