ABSTRACT

Objectives.

To validate the implementation drivers scale among first-level mental health care professionals in Colombia. The scale is designed as a tool to guide the implementation of strategies that effectively reduce gaps in mental health care.

Methods.

The Active Implementation Framework was adopted, which is a widely used model for measuring implementation. The participants included 380 individuals (55.56% men) – 349 health personnel trained in the Mental Health Gap Action Programme (mhGAP) and 31 territorial personnel in charge of planning mental health strategies at the territorial level in Colombia. To assess the critical dimensions of mhGAP implementation, we developed a scale of 18 items based on the active implementation framework. We conducted content validity assessments and exploratory factor analysis to evaluate the scale. We used the Organizational Readiness for Knowledge Translation scale as a comparative standard.

Results.

The implementation drivers scale identified four dimensions: system enablers for implementation, accessibility of the strategy, adaptability and acceptability, and strategy training and supervision. These dimensions had Cronbach alpha values of 0.914, 0.868, 0.927, and 0.725, respectively, indicating high internal consistency. In addition, all dimensions demonstrated adequate correlation with the Organizational Readiness for Knowledge Translation scale.

Conclusion.

The implementation drivers scale effectively determines the adaptability and implementation of various components of mental health programs, particularly those focusing on community-based approaches and primary care settings. As such, this scale can contribute to the more effective implementation of strategies outlined by global and local political frameworks, thus improving mental health care.

Keywords
Implementation science; mental health; primary health care; validation study; Colombia

RESUMEN

Objetivos.

Validar la escala de impulsores de implementación en profesionales de la salud mental del nivel de atención primaria en Colombia. La escala está diseñada como una herramienta para orientar la implementación de estrategias que permitan reducir de manera efectiva las desigualdades existentes en la atención de salud mental.

Métodos.

Se adoptó el marco de implementación activa, que es un modelo ampliamente utilizado para medir este tipo de implementaciones. Los participantes fueron 380 personas (55,56% hombres), de las cuales 349 eran profesionales de la salud capacitados mediante el Programa de acción mundial para superar las brechas en salud mental (mhGAP, por su sigla en inglés) y 31 formaban parte del personal territorial encargado de planificar estrategias de atención de salud mental a nivel territorial en Colombia. Para evaluar los dominios cruciales de la implementación del mhGAP, elaboramos una escala de 18 puntos basada en el marco de implementación activa. Para evaluar la escala se realizaron determinaciones de la validez de contenido y un análisis factorial exploratorio. Como patrón de referencia se utilizó la escala Predisposición Organizacional a la Transferencia del Conocimiento para el Cambio de Práctica Clínica.

Resultados.

La escala de impulsores de la implementación determinó cuatro dominios: facilitadores del sistema para la implementación, accesibilidad de la estrategia, adaptabilidad y aceptabilidad, y capacitación en la estrategia y supervisión. Estos dominios presentaron valores de alfa de Cronbach de 0,914, 0,868, 0,927 y 0,725, respectivamente, lo que indica una coherencia interna elevada. Además, todos los dominios mostraron una correlación adecuada con la escala Predisposición Organizacional a la Transferencia del Conocimiento para el Cambio de Práctica Clínica.

Conclusión.

La escala de impulsores de la implementación permite determinar de manera efectiva la adaptabilidad y la implementación de diversos componentes de los programas de salud mental, en particular de los que se centran en enfoques basados en la comunidad y en entornos de atención primaria. En este sentido, esta escala puede contribuir a una implementación más eficaz de las estrategias esbozadas en los marcos políticos locales y mundiales, con la consiguiente mejora de la atención de salud mental.

Palabras clave
Ciencia de la implementación; salud mental; atención primaria de salud; estudio de validación; Colombia

RESUMO

Objetivos.

Validar a escala de determinantes da implementação entre profissionais do primeiro nível de atenção à saúde mental na Colômbia. A escala foi concebida como uma ferramenta para orientar a implementação de estratégias que reduzam efetivamente as lacunas na atenção à saúde mental.

Métodos.

Foi adotada a Estrutura de Implementação Ativa, um modelo amplamente utilizado para medir a implementação. O estudo incluiu 380 indivíduos (55,56% homens): 349 profissionais de saúde treinados no Programa de Ação para Reduzir as Lacunas em Saúde Mental (mhGAP, na sigla em inglês) e 31 profissionais dos territórios encarregados de planejar estratégias de saúde mental em nível territorial na Colômbia. Para avaliar as dimensões essenciais da implementação do mhGAP, criou-se uma escala de 18 itens com base na Estrutura de Implementação Ativa. Foram realizadas avaliações da validade do conteúdo e uma análise fatorial exploratória para avaliar a escala. A escala de prontidão organizacional para tradução de conhecimentos (OR4KT, na sigla em inglês) foi utilizada como padrão de comparação.

Resultados.

A escala de determinantes da implementação identificou quatro dimensões: facilitadores sistêmicos de implementação; acessibilidade da estratégia; adaptabilidade e aceitabilidade; e capacitação e monitoramento da estratégia. Essas dimensões tiveram valores de alfa de Cronbach de 0,914, 0,868, 0,927 e 0,725, respectivamente, indicando alta consistência interna. Além disso, todas as dimensões demonstraram correlações adequadas com a escala OR4KT.

Conclusão.

A escala de determinantes da implementação avalia efetivamente a adaptabilidade e a implementação de vários componentes dos programas de saúde mental, especialmente componentes que se concentram em abordagens baseadas na comunidade e ambientes de atenção primária. Dessa forma, essa escala pode contribuir para uma implementação mais efetiva de estratégias delineadas por estruturas políticas mundiais e locais, melhorando assim a atenção à saúde mental.

Palavras-chave
Ciência da implementação; saúde mental; atenção primária à saúd; studo de validação; Colômbia

Mental health disorders are a major public health problem in Latin American countries (¹1. Zhang SX, Batra K, Xu W, Liu T, Dong RK, Yin A, et al. Mental disorder symptoms during the COVID-19 pandemic in Latin America – a systematic review and meta-analysis. Epidemiol Psychiatr Sci. 20229;31:e23. https://doi.org/10.1017/S2045796021000767
https://doi.org/10.1017/S204579602100076... ). Globally, despite the high prevalence of mental disorders, between 76% and 85% of people with mental disorders do not receive the care they need (²2. Organización Panamericana de la Salud. Manual operativo del mhGAP: Programa de acción para superar las brechas en salud mental (mhGAP). Washington, D.C.: PAHO; 2020 [cited 2023 December 30]. Available from: https://iris.paho.org/handle/10665.2/52328
https://iris.paho.org/handle/10665.2/523... ). Some factors that influence this gap are stigma, the shortage of qualified providers of mental health care, and the limited funding that mental health services receive (³3. Bassey EE, Gupta A, Kapoor A, Bansal A. COVID-19 and poverty in South America: the mental health implications. Int J Ment Health Addict. 2022:1–7. https://doi.org/10.1007/s11469-022-00765-6
https://doi.org/10.1007/s11469-022-00765... , ⁴4. Agudelo-Hernández F, Rojas-Andrade R. Mental health services in Colombia: a national implementation study. Int J Soc Determinants Health Health Serv. 2023;53(4):424–33. https://doi.org/10.1016/j.rcp.2021.08.001
https://doi.org/10.1016/j.rcp.2021.08.00... ).

In response to these gaps, it has been proposed to strengthen mental health services within primary health care (⁵5. Parsonage M, Hard E, Rock B. Managing patients with complex needs. Evaluation of the City and Hackney primary care psychotherapy consultation service. London: Centre for Mental Health; 2014.). One of the most internationally disseminated community mental health initiatives is the World Health Organization (WHO) Mental Health Gap Action Programme (mhGAP). The objective of mhGAP is to strengthen the commitment of national authorities and the allocation of resources to expand the coverage of scientifically validated mental health interventions (⁶6. Sapag JC, Álvarez Huenchulaf C, Campos Á, Corona F, Pereira M, Véliz V, et al. Programa de acción mundial para superar las brechas en salud mental (mhGAP) en Chile: aprendizajes y desafíos para América Latina y el Caribe [Mental health global action programme (mhGAP) in Chile: lessons learned and challenges for Latin America and the Caribbean]. Rev Panam Salud Publica. 2021;45:e32. https://doi.org/10.26633/RPSP.2021.32
https://doi.org/10.26633/RPSP.2021.32... ). When well implemented, mhGAP has been effective in multiple contexts, with greater impact than other interventions, especially institutionalization of people with mental health conditions (⁷7. Keynejad R, Spagnolo J, Thornicroft G. WHO mental health gap action programme (mhGAP) intervention guide: updated systematic review on evidence and impact. Evid Based Ment Health. 20216;24(3):124–30. https://doi.org/10.1136/ebmental-2021-300254
https://doi.org/10.1136/ebmental-2021-30... ).

Despite the potential benefits of community mental health programs, their evaluation can be challenging when it comes to ensuring that they are implemented faithfully in accordance with evidence-based practices (⁸8. Agudelo-Hernández F, Rojas-Andrade R. Ciencias de la implementación y salud mental : un diálogo urgente [Implementation sciences and mental health: an urgent dialogue]. Revista Colombiana de Psiquiatría. 2023; 52 (Suppl 1):S174–821. https://doi.org/10.1016/j.rcp.2021.08.001
https://doi.org/10.1016/j.rcp.2021.08.00... ). Fidelity is a critical factor in the success of evidence-based interventions, as it ensures that the intervention is carried out as intended by the designers in the protocols, guidelines, or manuals (⁹9. Rojas-Andrade R, Leiva L. Is implementation fidelity important? a systematic review on school-based mental health programs. Contempy Sch Psychol. 2018;23:339–50. https://doi.org/10.1007/s40688-018-0175-0
https://doi.org/10.1007/s40688-018-0175-... ). However, ensuring fidelity can be particularly difficult in low- and middle-income countries, where resources may be limited and providers of mental health care may have insufficient training in evidence-based practices (¹⁰10. Daset LR, Cracco C. Psicología basada en la evidencia. Algunas cuestiones básicas y una aproximación a través de una revisión bibliográfica sistemática [Evidence-based psychology. Basic concepts andliterature review]. Ciencias Psicologicas, 2013;7(2):209–20., ¹¹11. Manterola C, Otzen T, García N, Miriam M. Guías de práctica clínica basadas en la evidencia [Evidence-based clinical practice guidelines]. Revista de Cirugia, 2019;71(5):468–475. https://doi.org/35687/s2452-45492019005416
https://doi.org/35687/s2452-454920190054... ).

With regard to mhGAP, some measures related to planning, capacity-building, mental health care processes, and special events and populations have been proposed as implementation indicators (²2. Organización Panamericana de la Salud. Manual operativo del mhGAP: Programa de acción para superar las brechas en salud mental (mhGAP). Washington, D.C.: PAHO; 2020 [cited 2023 December 30]. Available from: https://iris.paho.org/handle/10665.2/52328
https://iris.paho.org/handle/10665.2/523... ). These indicators should be evaluated in health systems, in people with mental, neurological or substance use disorders, and in health care providers. However, most existing instruments enquire about symptoms in individuals or families, and there is no clear measure to investigate implementation skills in health systems and services (²2. Organización Panamericana de la Salud. Manual operativo del mhGAP: Programa de acción para superar las brechas en salud mental (mhGAP). Washington, D.C.: PAHO; 2020 [cited 2023 December 30]. Available from: https://iris.paho.org/handle/10665.2/52328
https://iris.paho.org/handle/10665.2/523... ).

In the case of Colombia, mhGAP is overseen by the Ministry of Health and Social Protection, which produces guidelines for implementation of the program through adaptation to the national context. Although the Ministry provides the guidelines, the health secretariats at the territorial level are responsible for managing and organizing training processes and must guarantee the availability of human resources to implement the program. Likewise, health management companies and service providers must include mhGAP in their service networks and facilitate the training of their staff (¹²12. Ministry of Health and Social Protection of Colombia. National guideline for the management at the territorial level of the action program to overcome the gaps in mental health, mhGAP. Bogotá: Ministry of Health and Social Protection of Colombia; 2023 [cited 2023 December 30]. Available from: https://www.minsalud.gov.co/sites/rid/Lists/BibliotecaDigital/RIDE/VS/PP/ENT/lineamiento-nacional-gestion-territorial-mhgap-msps-ops.pdf
https://www.minsalud.gov.co/sites/rid/Li... ). Despite these requirements, mhGAP and other outpatient processes for mental health care at the first level of care have faced implementation problems in Colombia (⁴4. Agudelo-Hernández F, Rojas-Andrade R. Mental health services in Colombia: a national implementation study. Int J Soc Determinants Health Health Serv. 2023;53(4):424–33. https://doi.org/10.1016/j.rcp.2021.08.001
https://doi.org/10.1016/j.rcp.2021.08.00... , ¹³13. Agudelo-Hernández F, García Cano JF, Salazar Vieira LM, Vergara Palacios W, Padilla M, Moreno Mayorga B. Brechas en la atención primaria en salud mental en Chocó, Colombia: barreras y desafíos [Gaps in primary mental health care in Chocó, Colombia: barriers and challenges.] Rev Panam Salud Publica. 2023;47:e138. https://doi.org/10.26633/RPSP.2023.138
https://doi.org/10.26633/RPSP.2023.138... ).

IMPLEMENTATION MEASURES

To ensure that primary health care programs are delivered faithfully and are effective in improving expected outcomes, the development and use of implementation measurement tools has been proposed (¹⁴14. Beidas RS, Buttenheim AM, Mandell DS. Transforming mental health care delivery through implementation science and behavioral economics. JAMA Psychiatry. 2021;78(9):941–2. https://doi.org/10.1001/jamapsychiatry.2021.1120
https://doi.org/10.1001/jamapsychiatry.2... ). When using implementation measures, mental health providers can ensure that the intervention is being delivered as intended and can make adjustments to it as needed to improve outcomes (¹⁵15. Mascayano F, Gajardo J. Ciencia de Implementación en la evaluación de programas y servicios de salud en Chile [Implementation science to evaluate health programs and services]. Rev Med Chil. 2018;146(7):945–6. https://doi.org/10.4067/s0034-98872018000700945
https://doi.org/10.4067/s0034-9887201800... , ¹⁶16. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. https://doi.org/10.1186/1748-5908-8-139
https://doi.org/10.1186/1748-5908-8-139... ). Implementation measures can also help assess the effectiveness of mental health interventions by providing data on whether the intervention is achieving its intended results. They can help understand the obstacles to and facilitators of an intervention so action can be taken to correct or enhance its implementation (¹⁷17. Peters DH, Tran NT, Adam T. Investigación sobre la implementación de políticas de salud: guía práctica [Research on the implementation of health policies: a practical guide]. Washington, DC: Organización Mundial de la Salud; 2014., ¹⁸18. Clinton-McHarg T, Yoong SL, Tzelepis F, Regan T, Fielding A, Skelton E, et al. Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review. Implement Sci. 2016;11(1):148. https://doi.org/10.1186/s13012-016-0512-5
https://doi.org/10.1186/s13012-016-0512-... ).

To measure implementation, one of the most widely used models is the active implementation framework (AIF) (¹⁹19. Birken SA, Powell BJ, Shea CM, Haines ER, Alexis Kirk M, Leeman J, et al. Criteria for selecting implementation science theories and frameworks: results from an international survey. Implement Sci. 2017;12(1):124. https://doi.org/10.1186/s13012-017-0656-y
https://doi.org/10.1186/s13012-017-0656-... ). The AIF recognizes that implementation is a complex and iterative process that requires continuous evaluation and adaptation to ensure success (²⁰20. Fixsen D, Blase K, Van Dyke M. Implementation. Practice & science. Chapel Hill, NC: Active Implementation Reserach Network; 2019.). The AIF incorporates five drivers that provide a framework for understanding the essential elements that must be in place to ensure that evidence-based practices are faithfully applied and sustained over time. The first driver is staff selection, that is, the process of identifying and selecting staff members who have the necessary skills, knowledge, and attitudes to successfully implement evidence-based practices. The second driver is training, that is, providing initial induction, ongoing training and support, and professional development and learning opportunities (²¹21. Fixsen D, Naoom SF, Blase K, Friedman RM, Wallace F. Implementation research: a synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.; 2005.). The third driver is consultation or supervision, that is, providing ongoing support and guidance during the implementation process. The fourth driver is the adaptation of the intervention, that is, the process of modifying the intervention to fit the unique needs and contexts of the organization or population to which the service is provided (²¹21. Fixsen D, Naoom SF, Blase K, Friedman RM, Wallace F. Implementation research: a synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.; 2005.). The fifth driver is organizational support, that is, the structures, policies, and practices that support the application of evidence-based practices (²¹21. Fixsen D, Naoom SF, Blase K, Friedman RM, Wallace F. Implementation research: a synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.; 2005.). This last driver includes leadership support, resource allocation, and alignment of organizational goals and objectives with implementation of evidence-based practices.

Implementation drivers are evidence-based and have proven effective in promoting successful implementation and sustainability of programs in various fields, including community mental health services (²²22. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57. https://doi.org/10.1177/1077558711430690
https://doi.org/10.1177/1077558711430690... , ²³23. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. https://doi.org/10.1007/s10488-010-0319-7
https://doi.org/10.1007/s10488-010-0319-... ). Researchers have used these drivers to guide implementation efforts and assess the effectiveness of implementation strategies in improving the quality and outcomes of mental health services (¹⁴14. Beidas RS, Buttenheim AM, Mandell DS. Transforming mental health care delivery through implementation science and behavioral economics. JAMA Psychiatry. 2021;78(9):941–2. https://doi.org/10.1001/jamapsychiatry.2021.1120
https://doi.org/10.1001/jamapsychiatry.2... ).

RESEARCH OBJECTIVE

Despite the importance of implementation measures, they are usually designed and validated in specific settings in high-income countries, which restricts the generalization of the information provided (¹⁵15. Mascayano F, Gajardo J. Ciencia de Implementación en la evaluación de programas y servicios de salud en Chile [Implementation science to evaluate health programs and services]. Rev Med Chil. 2018;146(7):945–6. https://doi.org/10.4067/s0034-98872018000700945
https://doi.org/10.4067/s0034-9887201800... ). The objective of this research was to validate an implementation drivers scale using a sample of Colombian community mental health professionals. This validation evaluated the factorial structure of the scale, and the internal consistency and convergent validity of its dimensions compared with another instrument with similar characteristics.

METHODS

Participants

A total of 380 individuals trained in mhGAP participated in our study by completing an online survey that provided detailed instructions on the implementation drivers instruments and purpose of the study. Among these participants, 55.6% were men. The group comprised 349 health professionals trained in mhGAP and 31 territorial personnel who are in charge of planning mental health strategies in Colombia’s 32 departments. The department of Antioquia was not included since the mhGAP components described by the health ministry and the Pan American Health Organization were not applied there. The 349 health professionals only responded to the survey we were validating; we did not ask their opinions on any aspects of it.

To assess concurrent validity, an additional 214 participants were recruited, including clinical and administrative staff and leaders of support or mutual aid groups. This group was selected based on the criteria of application of mhGAP after the initial training (first level care workers: general practitioners, social workers, nursing professionals, and psychologists). This group responded to both the survey we were validating and the Organizational Readiness for Knowledge Translation (OR4KT) scale, a Spanish-language instrument used to assess implementation in health centers (²⁴24. Grandes G, Bully P, Martinez C, Gagnon MP. Validity and reliability of the Spanish version of the Organizational Readiness for Knowledge Translation (OR4KT) questionnaire. Implement Sci. 2017;12(1):128. https://doi.org/10.1186/s13012-017-0664-y
https://doi.org/10.1186/s13012-017-0664-... ).

Content validity

To assess the critical dimensions of mhGAP implementation, we developed 18 items based on the AIF. Each item was designed to capture the standards derived from the AIF. For example, the item “I apply the ABC of the strategy correctly and completely” illustrates how complete and correct, although representing different aspects of implementation, must be simultaneously fulfilled for effective implementation. In this context, all items were evaluated by a panel of 18 mental health experts, including representatives from the Colombian College of Psychologists, the Colombian Association of Medical Schools, the Colombian Association of Humanities and Social Sciences Faculties, the Colombian Association of Public Health, the Ministry of Health and Social Protection of Colombia, and the Latin American chapter of the Global Implementation Society. This mixed panel ensured the input of specialized knowledge and different cultural perspectives, including experience in making judgments and evidence-based decisions, reputation in the community, willingness and motivation to participate, and fairness and inherent qualities such as self-confidence and adaptability (²⁵25. French S, Hanea AM, Bedford T, Nane GF. Introduction and overview of structured expert judgement. In: Hanea AM, Nane GF, Bedford T, French S, edsitors. Expert judgement in risk and decision analysis. Springer, Cham; 2021 (International Series in Operations Research & Management Science, vol 293). https://doi.org/10.1007/978-3-030-46474-5_1
https://doi.org/10.1007/978-3-030-46474-... ).

These professionals were sent an email with the scale and a form, which used a four-point Likert scale that addressed the representativeness, relevance, adequacy, comprehension, ambiguity, and clarity of the items (²⁶26. Linertová R, Serrano-Aguilar P, Posada-de-la-Paz M, Hens-Pérez M, Kanavos P, Taruscio D, Schieppati A, et al; BURQOL-RD Research Group. Delphi approach to select rare diseases for a European representative survey. The BURQOL-RD study. Health Policy. 2012;108(1):19–26. https://doi.org/10.1016/j.healthpol.2012.08.001
https://doi.org/10.1016/j.healthpol.2012... , ²⁷27. Escobar-Pérez J, Cuervo-Martínez Á. Validez de contenido y juicio de expertos: una aproximación a su utilización [Content validity and expert judgment: an approach to its use]. Avances en medición. 2008;6(1):27–36.). Interobserver agreement was determined using the Fleiss kappa index: 0.41–0.60 indicates moderate agreement; 0.61–0.80 indicates substantial agreement; and 0.81–1.00 indicates almost perfect agreement (²⁸28. Mérida R, Serrano A, Tabernero C. Diseño y validación de un cuestionario para la evaluación de la autoestima en la infancia [Design and validation of a questionnaire for the evaluation of self-esteem in childhood]. Revista de Investigación Educativa, 2015;33(1):149–62.). The Fleiss kappa index revealed a high level of consensus among the experts (²⁸28. Mérida R, Serrano A, Tabernero C. Diseño y validación de un cuestionario para la evaluación de la autoestima en la infancia [Design and validation of a questionnaire for the evaluation of self-esteem in childhood]. Revista de Investigación Educativa, 2015;33(1):149–62.), so the original items were retained with minor modifications in their wording. Through this process, a final instrument was formulated which comprised the 18 items, presented with a five-point Likert scale. Three items that asked about the budget allocation for the program, the willingness of the territorial administrative entity to implement it, and the willingness of the health insurer were discarded. These items were eliminated because they would limit the implementation of mhGAP to a single institution, thus excluding the formation of networks and partners that mhGAP requires for its implementation, both in a clinical and community setting.

Before applying the scale, a cultural evaluation was carried out (²⁹29. Alexandre NM, Guirardello Ede B. Adaptación cultural de instrumentos utilizados en salud ocupacional [Cultural adaptation of instruments utilized in occupational health]. Rev Panam Salud Publica. 2002;11(2):109–11. https://doi.org/10.1590/s1020-49892002000200007
https://doi.org/10.1590/s1020-4989200200... ). In this evaluation, the implementation drivers scale was given to 10 primary care professionals – five from medicine, three from nursing, and two from psychology. After completing the questionnaires, these professionals were asked to provide feedback on any questions that were unclear or difficult to understand, and to give their opinion on the overall acceptability of the instrument (see Table 1 for the original Spanish version of the scale).

Construct validity

Exploratory factor analysis was used to analyze the instrument’s dimensional structure using the Kaiser–Meyer–Olkin test and Bartlett test of sphericity, as well as eigenvalues and communalities. Varimax rotation was applied, given that the dimensions are theoretically independent according to the AIF. In addition, for each identified dimension, internal consistency was calculated using both the Cronbach alpha and McDonald omega, thereby ensuring the reliability of each measured construct.

Concurrent validity

We established the concurrent validity of the scale by correlating the scores obtained in the dimensions with those of the OR4KT scale (²⁴24. Grandes G, Bully P, Martinez C, Gagnon MP. Validity and reliability of the Spanish version of the Organizational Readiness for Knowledge Translation (OR4KT) questionnaire. Implement Sci. 2017;12(1):128. https://doi.org/10.1186/s13012-017-0664-y
https://doi.org/10.1186/s13012-017-0664-... , ³⁰30. Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci. 20140;9:7. https://doi.org/10.1186/1748-5908-9-7
https://doi.org/10.1186/1748-5908-9-7... ). The OR4KT scale serves as a comparative standard for measuring the effectiveness of implementation strategies in health contexts. However, due to its length, it has limited feasibility for use in primary care settings, where time demands for care are high and resources are limited. The OR4KT scale was chosen because no similar instrument was found in Spanish and validated with primary health care personnel. This comparison validated the effectiveness of our instrument in similar contexts, while offering a more concise and manageable alternative scale.

Ethical considerations

All participants agreed to fill out the instrument by signing an informed consent form, which emphasized the confidentiality of identifying and workplace data. This research was approved by the bioethics committee of University of Manizales (CBE02-2022).

RESULTS

Table 2 describes the items of the implementation drivers scale.

With regard to construct validity of the instrument, the Kaiser–Meyer–Olkin value was 0.825, while the Bartlett sphericity test value was 5769.31 (153 degrees of freedom), which was statistically significant (P<0.001).

In the exploratory factorial analysis (orthogonal rotation varimax) with half the study sample, four factors explained 74.43% of the variance, with communalities that ranged between 0.514 (“I adapt the strategy to the characteristics of my context without modifying the essential components”) and 0.954 (“I apply the ABC of the strategy correctly and completely”). All items showed factor loadings more than 0.4. Items loading on multiple factors were assigned to the factor with the highest loading to ensure that they did not load highly on others and to maintain theoretical consistency with their assigned factors.

The four domains were called: leadership and administrative support; innovation usability; implementation fidelity; and competency support. The internal consistency of the four factors was acceptable, with a Cronbach alpha of 0.914, 0.868, 0.927, and 0.725, respectively (Table 3). When comparing the remaining half of the study sample with OR4KT, significant correlations were found between all the components, both those of the implementation drivers scale and those of the OR4KT (Table 4).

DISCUSSION

The objective of this research was to describe the process of creating and validating the implementation drivers scale instrument in a sample of primary health care personnel. This instrument can be used to evaluate implementation of mental health programs in primary health care settings and community settings by leaders or implementaters, as an initial or follow-up evaluation. Important problems in implementation of evidence-based practices have been described, especially at the psychosocial level, where they may take up to 17 years to reach communities (²³23. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. https://doi.org/10.1007/s10488-010-0319-7
https://doi.org/10.1007/s10488-010-0319-... ), especially in low- and middle-income countries (⁴4. Agudelo-Hernández F, Rojas-Andrade R. Mental health services in Colombia: a national implementation study. Int J Soc Determinants Health Health Serv. 2023;53(4):424–33. https://doi.org/10.1016/j.rcp.2021.08.001
https://doi.org/10.1016/j.rcp.2021.08.00... ).

Barriers related to mhGAP implementation processes are still evident; even though 100 countries have been using the strategy, implementation has been done mainly in training and not in the follow-up step (⁶6. Sapag JC, Álvarez Huenchulaf C, Campos Á, Corona F, Pereira M, Véliz V, et al. Programa de acción mundial para superar las brechas en salud mental (mhGAP) en Chile: aprendizajes y desafíos para América Latina y el Caribe [Mental health global action programme (mhGAP) in Chile: lessons learned and challenges for Latin America and the Caribbean]. Rev Panam Salud Publica. 2021;45:e32. https://doi.org/10.26633/RPSP.2021.32
https://doi.org/10.26633/RPSP.2021.32... , ⁷7. Keynejad R, Spagnolo J, Thornicroft G. WHO mental health gap action programme (mhGAP) intervention guide: updated systematic review on evidence and impact. Evid Based Ment Health. 20216;24(3):124–30. https://doi.org/10.1136/ebmental-2021-300254
https://doi.org/10.1136/ebmental-2021-30... ). In the case of Colombia, these problems have been described as difficulties in integrating mhGAP into health systems, which results in a lack of sustainability of the program and partial implementation (¹³13. Agudelo-Hernández F, García Cano JF, Salazar Vieira LM, Vergara Palacios W, Padilla M, Moreno Mayorga B. Brechas en la atención primaria en salud mental en Chocó, Colombia: barreras y desafíos [Gaps in primary mental health care in Chocó, Colombia: barriers and challenges.] Rev Panam Salud Publica. 2023;47:e138. https://doi.org/10.26633/RPSP.2023.138
https://doi.org/10.26633/RPSP.2023.138... ). However, contextual factors have also significantly affected the development of this program (¹³13. Agudelo-Hernández F, García Cano JF, Salazar Vieira LM, Vergara Palacios W, Padilla M, Moreno Mayorga B. Brechas en la atención primaria en salud mental en Chocó, Colombia: barreras y desafíos [Gaps in primary mental health care in Chocó, Colombia: barriers and challenges.] Rev Panam Salud Publica. 2023;47:e138. https://doi.org/10.26633/RPSP.2023.138
https://doi.org/10.26633/RPSP.2023.138... ).

These difficulties may be caused by lack of financial resources, lack of support from the institutional leadership, and resistance of some members of the implementing organizations (³¹31. Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform. 2000;(1):65–70.). Likewise, adaptations of the components to the context should be a priority in the implementation process, since failure to make adaptations is also an important barrier (³²32. Leamy M, Bird V, Le Boutillier C, Williams J, Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. Br J Psychiatry. 2011;199(6):445–52. https://doi.org/10.1192/bjp.bp.110.083733
https://doi.org/10.1192/bjp.bp.110.08373... ).

In light of the above, an inquiry has been initiated to examine the constituents of the implementation process of mental health strategies. This inquiry includes an assessment of contextual factors, training protocols, implementation climate, and long-term sustainability (³²32. Leamy M, Bird V, Le Boutillier C, Williams J, Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. Br J Psychiatry. 2011;199(6):445–52. https://doi.org/10.1192/bjp.bp.110.083733
https://doi.org/10.1192/bjp.bp.110.08373... ). Thus, the domains proposed according to the factor analysis coincide with the implementation stages of AIF: exploration, installation, initial implementation, and full implementation (¹⁹19. Birken SA, Powell BJ, Shea CM, Haines ER, Alexis Kirk M, Leeman J, et al. Criteria for selecting implementation science theories and frameworks: results from an international survey. Implement Sci. 2017;12(1):124. https://doi.org/10.1186/s13012-017-0656-y
https://doi.org/10.1186/s13012-017-0656-... , ²⁰20. Fixsen D, Blase K, Van Dyke M. Implementation. Practice & science. Chapel Hill, NC: Active Implementation Reserach Network; 2019.). Similarly, the items with lower values (items 6 and 16) are retained because they relate to the commitment of decision-makers and the fidelity of the program, which are fundamental components of its implementation (⁵5. Parsonage M, Hard E, Rock B. Managing patients with complex needs. Evaluation of the City and Hackney primary care psychotherapy consultation service. London: Centre for Mental Health; 2014.–⁷7. Keynejad R, Spagnolo J, Thornicroft G. WHO mental health gap action programme (mhGAP) intervention guide: updated systematic review on evidence and impact. Evid Based Ment Health. 20216;24(3):124–30. https://doi.org/10.1136/ebmental-2021-300254
https://doi.org/10.1136/ebmental-2021-30... ).

At its core, the implementation process involves the application of various components that form the backbone of the strategies to strengthen health at the primary health care and community levels. As a crucial first step, the application of these components is viewed as a dependent variable in the implementation process (²⁰20. Fixsen D, Blase K, Van Dyke M. Implementation. Practice & science. Chapel Hill, NC: Active Implementation Reserach Network; 2019.). In this sense, the AIF, proposed to develop the implementation drivers scale, has been used in similar contexts, such as Peru (³³33. Westgard C, Fleming WO. The use of implementation science tools to design, implement, and monitor a community-based mHealth intervention for child health in the Amazon. Front Public Health. 2020;8:411. https://doi.org/10.3389/fpubh.2020.00411
https://doi.org/10.3389/fpubh.2020.00411... ) and Guatemala (³⁴34. Thompson LM, Diaz-Artiga A, Weinstein JR, Handley MA. Designing a behavioral intervention using the COM-B model and the theoretical domains framework to promote gas stove use in rural Guatemala: a formative research study. BMC Public Health. 2018;18(1):253. https://doi.org/10.1186/s12889-018-5138-x
https://doi.org/10.1186/s12889-018-5138-... ), and helped the implementation and research teams to consider aspects that they often missed – for example, choosing the right solutions for the local context, gathering information for data-driven decision-making and adaptations, and monitoring implementation results.

The concept of fidelity – the extent to which an intervention is executed according to the prescribed guidelines – was examined. This parameter is closely linked to the intended outcomes of the implementation process (³¹31. Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform. 2000;(1):65–70.). There are other comparable instruments, including: the Therapy Procedures Observational Coding System – Strategies (TPOCS-S) and the ENhancing Assessment of Common Therapeutic factors (ENACT-18) (³⁵35. Kohrt BA, Jordans MJ, Rai S, Shrestha P, Luitel NP, Ramaiya MK, et al. Therapist competence in global mental health: development of the ENhancing Assessment of Common Therapeutic factors (ENACT) rating scale. Behav Res Ther. 2015;69:11–21. https://doi.org/10.1016/j.brat.2015.03.009
https://doi.org/10.1016/j.brat.2015.03.0... ), both of which have been used in community-based interventions. However, these instruments are designed to assess the performance of clinicians and may not be appropriate for people acting as facilitators in community-based interventions. Community action is a central component of mhGAP, and was considered with the participants in the present study. Thus the implementation drivers scale is a novel instrument, since it allows these implementation variables to be determined at the clinical, administrative, and community level, which is where people go in search of continuous care (³3. Bassey EE, Gupta A, Kapoor A, Bansal A. COVID-19 and poverty in South America: the mental health implications. Int J Ment Health Addict. 2022:1–7. https://doi.org/10.1007/s11469-022-00765-6
https://doi.org/10.1007/s11469-022-00765... , ⁴4. Agudelo-Hernández F, Rojas-Andrade R. Mental health services in Colombia: a national implementation study. Int J Soc Determinants Health Health Serv. 2023;53(4):424–33. https://doi.org/10.1016/j.rcp.2021.08.001
https://doi.org/10.1016/j.rcp.2021.08.00... ).

In comparison with other assessment tools that measure implementation variables in health care (⁸8. Agudelo-Hernández F, Rojas-Andrade R. Ciencias de la implementación y salud mental : un diálogo urgente [Implementation sciences and mental health: an urgent dialogue]. Revista Colombiana de Psiquiatría. 2023; 52 (Suppl 1):S174–821. https://doi.org/10.1016/j.rcp.2021.08.001
https://doi.org/10.1016/j.rcp.2021.08.00... , ²⁴24. Grandes G, Bully P, Martinez C, Gagnon MP. Validity and reliability of the Spanish version of the Organizational Readiness for Knowledge Translation (OR4KT) questionnaire. Implement Sci. 2017;12(1):128. https://doi.org/10.1186/s13012-017-0664-y
https://doi.org/10.1186/s13012-017-0664-... , ³⁰30. Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci. 20140;9:7. https://doi.org/10.1186/1748-5908-9-7
https://doi.org/10.1186/1748-5908-9-7... ), the implementation drivers scale has a high level of internal consistency, as indicated by Cronbach alphas of 0.72 or higher for all four dimensions. Preliminary evaluations indicate that removing the items “I adapt the strategy to the characteristics of my context without modifying the essential components", which comes under the leadership and administrative support dimension, and “I apply the ABC of the strategy correctly and completely”, which comes under the implementation fidelity dimension, would raise the lower coefficient from 0.72 to more than 0.80. However, due to their important contribution to the accessibility-related domains in the factor analysis, it was decided to retain these two items.

Based on the evaluation of internal consistency and repeatability, the initial validation process demonstrates that the implementation drivers scale is a dependable instrument. The overall instrument has a Cronbach alpha of 0.892, a statistical indicator that strongly supports the instrument's reliability.

Three methods were used to assess the validity of the implementation drivers scale, taking into account both internal and external evidence. Internal validity was established to provide evidence of internal structure validity by evaluating the extent to which individual items aligned with the underlying construct of interest. The objective was to unify the arrangement of items by consolidating four factors.

Pearson correlations were used to examine the concurrent validity of the questionnaire of the implementation drivers scale by investigating the relationship between the dimensions evaluated by the scale and those evaluated by the OR4KT. Significant correlations were observed between similar dimensions in both instruments.

Realizing the benefits that evidence-based practices promise for people with mental disorders, such as improving quality of life and increasing capabilities, will also be dependent on local circumstances and an inclusive approach that acknowledges the intersection of various determinants of quality of life and well-being. Future studies could explore ways to examine the impact of these factors on implementation outcomes.

A limitation of this study is the lack of cognitive interviews during the item development process. Cognitive interviews could have provided a deeper insight into participants’ perceptions and experiences related to the items. To address this limitation, incorporating such interviews in future versions of the scale may help identify item ambiguities. In addition, there was a potential desirability bias in the participants’ responses. To mitigate this bias in future research, specific questions could be included in the questionnaire that directly address desirability bias and apply the instrument in different scenarios.

Experimental evaluations could be conducted to assess whether these instrument items can predict the implementation ability of the mhGAP strategy or effectively differentiate between different contexts. Developing concise, cost-effective, and valid instruments applicable in real-world settings is urgently needed to address the substantial implementation gaps faced by global public health.

Conclusions

The current study provides an instrument that can contribute to more effective implementation of strategies for mental health and that could be useful in developing plans, strategies, and policies. Despite its simplicity, the instrument can facilitate clinical care in primary health care and community settings for mental health which can result in favorable outcomes.

Disclaimer.

The authors hold sole responsibility for the views expressed in the manuscript, which may not necessarily reflect the opinion or policy of the Revista Panamericana de Salud Pública / Pan American Journal of Public Health and/or those of the Pan American Health Organization and the World Health Organization.

REFERENCES

Publication Dates

History