Using realist evaluation to assess primary healthcare teams' responses to intimate partner violence in Spain
Uso de la evaluación realista para evaluar las respuestas de los equipos de atención primaria a la violencia del compañero íntimo en España
Isabel Goicoleaa, b, Anna-Karin Hurtiga, Miguel San Sebastiana, Bruno Marchalc and Carmen Vives-Casesb, d, e
a Unit of Epidemiology and Global Health, Department of Clinical Medicine and Public Health, Umeå University, Sweden
b Grupo de Investigación de Salud Pública, Universidad de Alicante, Alicante, Spain
c Institute of Tropical Medicine, Antwerp, Belgium
d Departamento de Enfermería Comunitaria, Medicina Preventiva y Salud Pública e Historia de la Ciencia, Universidad de Alicante, Alicante, Spain
e CIBER de Epidemiología y Salud Pública (CIBERESP), Spain
This study has been funded through a COFAS grant (supported by COFUND action within the Marie Curie Action People, in the Seventh Framework programme and the Swedish Council for Working Life and Social Research/FAS-Forskningsradet för arbetsliv och socialvetenskap) through a competitive call.
Objective: Few evaluations have assessed the factors triggering an adequate health care response to intimate partner violence. This article aimed to: 1) describe a realist evaluation carried out in Spain to ascertain why, how and under what circumstances primary health care teams respond to intimate partner violence, and 2) discuss the strengths and challenges of its application.
Methods: We carried out a series of case studies in four steps. First, we developed an initial programme theory (PT1), based on interviews with managers. Second, we refined PT1 into PT2 by testing it in a primary healthcare team that was actively responding to violence. Third, we tested the refined PT2 by incorporating three other cases located in the same region. Qualitative and quantitative data were collected and thick descriptions were produced and analysed using a retroduction approach. Fourth, we analysed a total of 15 cases, and identified combinations of contextual factors and mechanisms that triggered an adequate response to violence by using qualitative comparative analysis.
Results: There were several key mechanisms -the teams' self-efficacy, perceived preparation, women-centred care-, and contextual factors -an enabling team environment and managerial style, the presence of motivated professionals, the use of the protocol and accumulated experience in primary health care- that should be considered to develop adequate primary health-care responses to violence.
Conclusion: The full application of this realist evaluation was demanding, but also well suited to explore a complex intervention reflecting the situation in natural settings.
Key words: Spouse abuse. Delivery of health care. Primary health care. Evaluation studies. Organizational case studies.
Objetivo: Hay pocas evaluaciones de los factores que generan una respuesta sanitaria adecuada a la violencia del compañero íntimo. Este artículo tiene como objetivo: 1) describir una evaluación realista para investigar por qué, en qué circunstancias y cómo los equipos de atención primaria en España responden a la violencia de pareja, y 2) discutir las fortalezas y los desafíos de su aplicación.
Métodos: Se llevaron a cabo una serie de estudios de caso en cuatro pasos. Primero construimos una teoría del programa inicial (PT1), basada en entrevistas con profesionales del nivel gerencial. Segundo, refinamos la PT1 a PT2, a través del estudio de caso en un equipo de atención primaria que estaba respondiendo activamente a la violencia. Tercero, refinamos la PT2 incorporando al análisis otros tres casos situados en la misma región. Recogimos información cualitativa y cuantitativa, elaboramos descripciones extensas de los casos y los analizamos usando el enfoque de retroducción. Cuarto, analizamos 15 casos para identificar las combinaciones de factores contextuales y mecanismos que desencadenaban una respuesta adecuada a la violencia, utilizando análisis cualitativo comparativo.
Resultados: Hubo varios mecanismos clave -autoeficacia del equipo, preparación percibida, y atención centrada en las mujeres-, así como factores contextuales -ambiente de equipo y estilo de gestión, presencia de profesionales motivados, uso del protocolo y experiencia acumulada en atención primaria-, que deben considerarse para generar respuestas sanitarias adecuadas a la violencia.
Conclusión: a aplicación de esta evaluación realista requirió tiempo, pero resultó apropiada para explorar una intervención compleja tal como se desarrolla en condiciones reales.
Palabras clave: Violencia contra la mujer. Servicios básicos de salud. Atención primaria de salud. Evaluación. Estudios de casos organizativos.
Men's intimate partner violence (IPV) against women is a global public health problem and has devastating effects on the health and wellbeing of women and children.1-5 The health care system, especially primary health care facilities, can play a key role in responding to IPV, since they are the public institutions most frequently accessed by women exposed to IPV -even if most of those cases remain undetected by health professionals.1,6,7 The World Health guidelines Responding to intimate partner violence and sexual violence against women defines what an "adequate healthcare response to IPV" should include: detect, provide health-care assistance and register, orient on available resources, coordinate with other professionals and institutions, and ensure that all the previous actions are carried out putting women's needs at the centre (woman-centred).5
Evaluations to assess the level of implementation of a health care response and, most importantly, on the factors triggering an adequate health care response to IPV -understood as the one that fulfils the WHO recommendations-5,8-10 are scarce. In addition, research methodologies that account for the role that contextual factors play instead of controlling them, have still been scarcely used to assess the health-care response to IPV.
This article aims to fill this knowledge gap by: 1) describing a realist evaluation carried out in Spain aiming to ascertain why, how and under which circumstances primary health care teams respond to intimate partner violence, and 2) discussing the strengths and challenges of the application of this approach to explore the health care response to IPV.
An overview of realist evaluation
Realist evaluation is a type of theory-driven evaluation that aims to ascertain why, how and under what circumstances, programmes succeed or fail. It has proven to be useful when exploring complex health system interventions.11-14
Realist evaluation begins with the formulation of a theory behind the development of an intervention, known as programme theory (PT). PT is formulated on the basis of a review of literature and documents and/or the experience of stakeholders involved in the intervention, and describes how the intervention is supposed to generate change. The basis of the PT consists of a context-mechanism-outcome configuration, which describes patterns or causal chains: certain components of the intervention trigger certain mechanisms within individuals (or groups of individuals) that produce certain outcomes. PT is then tested through empirical research from cases where the intervention has been implemented. The analysis of data in these cases serves to refine the preliminary PT.9,13,14 Realist evaluation provides a deeper understanding of the links between the programme and the outcomes by exploring the interactions between programme, actors, context and mechanisms, and consequently offers results that can be acted upon by decision makers.9,11,12,15,16
Overall design of the realist evaluation and steps followed
In this study, we applied the realist evaluation approach to explore a complex intervention: the implementation of a healthcare response to IPV within primary health care teams in Spain. In Spain, the Gender Based Violence Law, enacted in 2004, specifically addressed the responsibilities of the health sector.17-20 Grounded in this law, the 17 decentralized regional Spanish health systems have developed interventions aimed at: 1) developing protocols to guide health providers' response to IPV, 2) training health-care professionals, and 3) developing and implementing an IPV monitoring system.
For this realist evaluation, we carried out a series of case studies in Spain in four steps, between June 2012 and April 2015. The cases were purposively selected in order to capture the diversity of practices and contextual factors: they were located in four different regions with some being larger cities (eight) and others smaller rural towns (seven). Some were considered by the professionals in charge of IPV programmes within the regional health systems as more interested in responding to IPV (six) than others (nine). In each of the cases, qualitative information was collected via interviews and observation and quantitative information was collected using a questionnaire to assess the readiness of health professionals to respond to IPV (PREMIS), measuring through nine dimensions: 1) perceived preparation, 2) perceived knowledge, 3) actual knowledge, 4) practice issues, 5) opinions on work-place issues, 6) opinions on constraints, 7) opinions on self-efficacy, 8) opinions on victims' understanding, and 9) opinions on victim autonomy.21,22Table 1 summarizes the methods for data collection and analysis applied in each of the four steps.
Table 1 Methods for data collection and analysis applied in each of the four steps.
During the first step, we developed an initial PT (PT1) based on document and literature reviews, and interviews with 26 professionals in charge of coordinating IPV interventions in 17 regional health systems and also at the national level. All the material was analysed using a thematic analysis guided by realist evaluation principles.
In the second step, we refined PT1 into PT2 by testing it in a positive deviant case: the primary healthcare team of La Virgen (pseudonym), located in Region 1 and considered by the professionals in charge of the IPV programme in the regional health system as actively responding to IPV. Data collection and analysis followed an analytic case study design. Both qualitative data (observation, interviews with different informants) and quantitative data (PREMIS questionnaires) were collected. For the analysis, we developed a thick description of the case, guided by PT1 but remaining open to new emerging issues. Afterwards we searched for patterns using the retroduction approach: outcomes were explained by looking into the mechanisms and context elements and ruling out potential alternative explanations.
In the third step, we tested the refined PT2 by analysing three other cases located in the same region. Information was collected in each of the sites using the same tools applied in La Virgen. Following a guide based on PT2 and afterwards complemented with the quantitative information from the questionnaires, thick descriptions of each case where developed. Using the retroduction approach described in the previous step, the thick descriptions of the four cases were contrasted in order to identify patterns and PT3 was developed.
Finally, in the fourth step, we contrasted the findings emerging from the four cases in Region 1 with 11 additional cases located in three other regions. A multiple case study design was chosen.23 In order to handle the large amount of information without losing familiarity with each case, qualitative comparative analysis (QCA) was used. QCA assesses the extent to which a configuration of conditions explain the outcomes.24-27 Based on PT3, a set of conditions (context and mechanisms) and outcomes were identified and assessed in each case using the tools previously described. Afterwards, solution formulae were calculated using fzQCA software, which allowed us to explore what combinations of contextual factors and mechanisms best explained the emergence of an adequate team response to IPV (the outcome).
PT1 was the starting point of this realist evaluation and described how the implemented interventions within primary health care teams in Spain have focused on developing protocols, monitoring systems, and training professionals on IPV. Our initial hypothesis was that such interventions generate team learning about IPV within primary healthcare teams, and that this team learning process was expected to lead to changes in team culture, teamwork and practice with regard to IPV. However, PT1 did not accurately account for what was actually happening within primary health care teams, and how (if at all) those potential mechanisms and contextual factors might be interacting together to trigger adequate health care responses to IPV.
PT2 emerged as a refinement of PT1 and stressed that even if the interventions carried out might have played a role in shaping individual and team responses to IPV, such responses were strongly influenced by other factors. Team and individual level factors (such as motivation, team climate28 and team leadership) shaped the teams' capacity to create spaces for team learning and teamwork (and also the interest in this), and to provide therapeutic responses to IPV rather than merely referrals to specialised services. We found that individual motivation was key in shaping professionals' responses. Regional interventions legitimised the work of motivated professionals but did not encourage them to engage in individual and/or team responses to IPV. We found that other interventions not directly related to IPV (such as interventions to strength woman-centred care) can facilitate the integration of team responses to IPV.
In order to explore whether different mechanisms and contextual factors were present in cases with different degrees of IPV responses, we put PT2 to the test in three additional cases, located in the same region as La Virgen but with perceived different achievements in terms of their responsiveness to IPV. The PT3 that emerged from the analysis of those four cases explained how the interventions to integrate a response to IPV within first-line health-care teams a) were implemented differently depending on individual and team-level factors, b) were more easily adopted when woman-centred care had been previously implemented in the team, c) were more easily implemented and sustained in environments that promoted team-work, d) required a sufficient number of motivated professionals and an enabling managerial style in the team to be sustained, e) legitimised the response of internally motivated providers, and f) had limited impact on non-internally motivated providers.
In order to further refine PT3 and to identify the key combination of factors and mechanisms in triggering an adequate IPV response to IPV, we applied it to an even larger number of cases. This was done in the last step and lead to the development of PT4. The final PT4 stated that primary health care teams that perceived themselves as well prepared to deal with IPV, consider themselves to be self-efficient enough to deal with IPV and have a woman-centred approach to IPV respond better to women exposed to IPV. These three mechanisms are necessary in order to trigger a better response, but they are not the only ones -there might be others that would trigger a good IPV response. In terms of the contextual factors -at the team level, in general, and in regards to the intervention specifically- the use of the protocol and accumulated experience in primary health care seem to be the most relevant contextual factors in triggering a good response, however in order to do so, they must be combined with other factors, such as having enough number of professionals who have received training on IPV, the presence of a committed social worker, or/and an enabling working environment.
See Table 2 for a summary of the successive PTs.
Table 2 Summary of emerging programme theories and publications
where more detailed information can be found.
Realist evaluation has been considered to be well suited for evaluating complex interventions, as it allows for considering interactions between different layers of context, outcomes and the underlying mechanisms.12-14,29-31 In this study, the use of realist evaluation allowed us to capture the gaps between the planned intervention and its implementation as well as the relevance of contextual factors, and of other interventions (besides the ones directly targeting IPV) in shaping the individual and team responses to IPV. With the exception of a realist review of screening programmes by O'Campo et al.,32 to our knowledge this is the first time that realist evaluation has been used to assess health-care response to IPV.
One challenge when evaluating complex interventions refers to the difficulties in assessing outcomes. In this study, despite the existence of the WHO guidelines, it is still difficult to define what an "adequate response to IPV" is and even harder to measure it. In order to conduct a realist evaluation assessing outcomes is key. In this study, our outcome was assessed both qualitatively and quantitatively (via the PREMIS), and consequently the findings of our studies are applicable as long as our definition and measurement of "adequate response of IPV" can be considered relevant.
In addition, dealing with uncertainty and complexity means that the research team has to make changes to the planned study protocol in order to best embrace such complexity. In our case, it was possible to be flexible and adapt the study protocol to the emerging findings over the successive steps of the study. However, this required time, resources and expertise, which might not always be available when conducting evaluations within more limited time frames.
The successive rounds allowed us to explore different perspectives more in depth. In that sense, we do not consider PT4 the "best PT". Each PT allowed us to more deeply explore different layers of the phenomenon. In addition, the knowledge generated during each of the cycles of refinement guided us in data collection and analysis during the following cycle. However, even if we claim that we followed an emergent design and that each successive step benefited from the knowledge gained in the previous one, the process was not ideal. Logistically, it was not possible to completely finalize each PT before starting data collection for testing the next PT. In that sense, some of the issues emerging from the successive rounds could not be completely accounted for during data collection. That leaves important aspects that deserve to be further explored, including the role of individual motivation in shaping a team's response.
Overall, the realist evaluation approach allowed to explore how different layers of contextual factors interacted with mechanisms to trigger an adequate response to IPV. However, the characteristics of the intervention posed an important challenge. The implementation of the health-care response to IPV in Spain was not a clear cut intervention but rather a group of actions emerging from the Gender Based Violence Law approved in 2004 and regional laws.18 The intervention was not well defined, did not start at the same time in all regions, varied in terms of the strategies chosen and the extent of its implementation by region, and evolved over time. In addition, austerity measures started impacting the Spanish health system in general, modifying key contextual factors such as team workloads, and primary health care investments.33 One of the limitations of this study is that we were not able to fully capture the time component and the regional differences. We have purposively overlooked these complexities in order to be able to focus on the already complex issues happening at the team level, since we also considered these issues to be relevant beyond the Spanish context.
Finally, PT4 constituted a summary of the key mechanisms and contextual factors needed to adequately generate IPV responses within primary health care teams. However, the relevance of such conditions could be further confirmed (or refuted) if more cases were included in future analysis. In addition, the role of other relevant conditions (not included in the analysis) could also be further explored with other cases.
This realist evaluation study allowed us to point out several key mechanisms -teams' self-efficacy, perceived preparation, woman-centered care-, and contextual factors such as enabling a team environment and managerial style, motivated professionals, the use of the protocol and accumulated experience in primary health care, that should be considered in order to develop adequate primary health-care responses to IPV.
The full application of this realist evaluation was demanding in terms of time, funding and the expertise required. However, we claim that in this study the use of realist evaluation was useful in capturing variations, and the successive cycles allowed knowledge augmentation through a step-by-step scaling up. One strength of this approach is that it allows for flexibility in moving backwards and forwards between theory and empirical findings and in adapting to emerging unexpected findings. In addition, this approach is more suitable for exploring complex interventions as they are implemented in natural settings, as is the case with the implementation of health-care responses to IPV.
Editor in charge
The corresponding author on behalf of the other authors guarantee the accuracy, transparency and honesty of the data and information contained in the study, that no relevant information has been omitted and that all discrepancies between authors have been adequately resolved and described.
I. Goicolea was the PI in this project, participated in the conception and design of the work, the data collection, the analysis and interpretation of the data, the writing of the article and its critical review with important intellectual contributions. C. Vives-Cases, A.-K. Hurtig, M. San Sebastian and B. Marchal have participated in the conception and design of the work, the analysis and interpretation of the data and have critically review the article with important intellectual contributions. All authors have approved the final version of the manuscript. All authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Conflict of interest
The authors declare that there is no conflict of interest. One of the authors (C. Vives-Cases) belongs to the GACETA SANITARIA editorial committee, but was not involved in the editorial process of the manuscript.
The authors are grateful to Erica Briones-Vozmediano, Laura Otero-García and Marta García who actively participated in the data collection and analysis of some of the cases. The authors are grateful to the Observatory of Women's Health of the Spanish Ministry of Health, and to the professionals in charge of IPV programs within the regional health system for facilitating access to relevant information and contacts and to Lynn Short for her support during the process of translating and retranslating the PREMIS questionnaire. The authors are grateful to the professionals working at the municipal, regional and judicial resources for women exposed to IPV in the four regions for providing relevant information and contacts. The authors are especially grateful to the primary health care teams and to the women patients who participated in this study, who shared their time, enthusiasm, experiences and expertise, and facilitated access to unpublished information.
1. Garcia-Moreno C., Hegarty K., D'Oliveira A.F., et al. The health-systems response to violence against women. Lancet. 2015;385:1567-79.
2. Garcia-Moreno C., Zimmerman C., Morris-Gehring A., et al. Addressing violence against women: a call to action. Lancet. 2015;385:1685-95.
3. World Health Organization. Preventing intimate partner and sexual violence against women: taking action and generating evidence. Geneva: World Health Organization; 2010.
4. World Health Organization. Global and regional estimates of violence against women: prevalence and health effects of intimate partner violence and non-partner sexual violence. Geneva: World Health Organization; 2013.
5. World Health Organization. Responding to intimate partner violence and sexual violence against women. WHO clinical and policy guidelines. Geneva: World Health Organization; 2013.
6. Beynon C.E., Gutmanis I.A., Tutty L.M., et al. Why physicians and nurses ask (or don't) about partner violence: a qualitative analysis. BMC Public Health. 2012;12:473.
7. Pratt-Eriksson D, Bergbom I, Lyckhage ED. Don't ask don't tell: battered women living in Sweden encounter with healthcare personnel and their experience of the care given. Int J Qualitative Studies Health Well-being. 2014;9. (Consulted on 15/08/2016). Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3937531/pdf/QHW-9-23166.pdf.
8. Colombini M., Mayhew S., Watts C. Health-sector responses to intimate partner violence in low- and middle-income settings: a review of current models, challenges and opportunities. Bull World Health Organ. 2008;86:635-42.
9. Goicolea I., Hurtig A.K., San Sebastian M., et al. Developing a programme theory to explain how primary health care teams learn to respond to intimate partner violence: a realist case-study. BMC Health Services Research. 2015;15:228 (Consulted on 15/08/2015). Available at: http://www.biomedcentral.com/content/pdf/s12913-015-0899-8.pdf.
10. Ramsay J, Carter Y, Davidson L, et al. Advocacy interventions to reduce or eliminate violence and promote the physical and psychosocial well-being of women who experience intimate partner abuse. Cochrane Database Syst Rev. 2009. (Consulted on 15/08/2015). Available at: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD005043.pub2/epdf.
11. Goicolea I., Vives-Cases C., San Sebastian M., et al. How do primary health care teams learn to integrate intimate partner violence (IPV) management? A realist evaluation protocol. Implementation Science. 2013;8:36 (Consulted on 15/08/2015). Available at: http://www.biomedcentral.com/content/pdf/1748-5908-8-36.pdf.
12. Marchal B., Van Belles S., De Brouwere V., et al. Studying complex interventions: reflections from the FEMHealth project on evaluating fee exemption policies in West Africa and Morocco. BMC Health Services Research. 2013;13:469 (Consulted 15/08/2015). Available at: http://www.biomedcentral.com/1472-6963/13/469.
13. Pawson R. The science of evaluation: a realist manifesto. London: SAGE Publications; 2013. p. 240.
14. Pawson R., Tilley N. Realistic evaluation. London: SAGE Publications; 1997. p. 256.
15. Goicolea I, Coe AB, Hurtig AK, et al. Mechanisms for achieving adolescent-friendly services in Ecuador: a realist evaluation approach. Global Health Action. 2012;5. (Consulted 15/08/2015). Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3409349/pdf/GHA-5-18748.pdf.
16. Marchal B., Dedzo M., Kegels G. Turning around an ailing district hospital: a realist evaluation of strategic changes at Ho Municipal Hospital (Ghana). BMC Public Health. 2010;10:787 (Consulted on 15/08/2015). Available at: http://www.biomedcentral.com/content/pdf/1471-2458-10-787.pdf.
17. Ley Orgánica de medidas de protección integral contra la violencia de género. L. N. 1/2004 (28 Diciembre 2004).
18. Goicolea I., Briones-Vozmediano E., Ohman A., et al. Mapping and exploring health systems' response to intimate partner violence in Spain. BMC Public Health. 2013;13:1162 (Consulted on 15/08/2015). Available at: http://www.biomedcentral.com/content/pdf/1471-2458-13-1162.pdf.
19. Loewenberg S. Domestic violence in Spain. Lancet. 2005;365:464.
20. Ortiz-Barreda G., Vives-Cases C. Legislation on violence against women: overview of key components. Revista Panamericana Salud Pública. 2013;33:61-72.
21. Vives C., Torrubiano Domínguez J., Carrasco Portiño M., et al. Validación de la Versión Española del Cuestionario Physician Readiness to Manage Intimate Partner Violence Survey (PREMIS). Revista Española Salud Pública. 2015;89:173-90.
22. Short L.M., Alpert E., Harris J.M., et al. A tool for measuring physician readiness to manage intimate partner violence. American J Preventive Med. 2006;30:173-80.
23. Yin R. Case study research. Thousand Oaks. California: SAGE; 2009. p. 240.
24. Ragin C.C. Redesigning social inquiry: fuzzy sets and beyond. Chicago: The University of Chicago Press; 2008. p. 226.
25. Ragin C.C., Amoroso L.M. Constructing social research: the unity and diversity of method. Thousand Oaks. California: SAGE Publications; 2010. p. 238.
26. Rihoux B., Ragin C.C. Configurational comparative methods: qualitative comparative analysis (QCA) and related techniques. Thousand Oaks. California: SAGE Publications; 2009. p. 209.
27. Befani B. Between complexity and generalization: addressing evaluation challenges with QCA. Evaluation. 2013;19:269-83.
28. Glisson C. Assessing and changing organizational culture and climate for effective services. Research Soc Work Practice. 2007;17:736-47.
29. Rogers P.J. Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation. 2008;14:29-48.
30. Marchal B., van Belle S., van Olmen J., et al. Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation. 2012;18:192-212.
31. Pawson R., Manzano-Santaella A. A realist diagnosis workshop. Evaluation. 2012;18:176-91.
32. O'Campo P., Kirst M., Tsamis C., et al. Implementing successful intimate partner violence screening programs in health care settings: evidence generated from a realist-informed systematic review. Soc Sci Med. 2011;72:855-66.
33. Legido-Quigley H., Urdaneta E., Gonzalez A., et al. Erosion of universal health coverage in Spain. Lancet. 2013;382:1977.
Received 20 May 2015
Accepted 18 August 2015