The Equity-effectiveness Loop as a Tool for Evaluating Population Health Interventions
El asa de equidad-efectividad como una herramienta para la evaluación de intervenciones poblacionales
Vivian Welch, Peter Tugwell and Erin B. Morris
Received 8th January 2007/Sent for Modification 2th May 2008/Accepted 25th October 2008
Population health is concerned with reducing health inequities, defined as being unfair and avoidable differences in health. The equity-effectiveness framework is described and illustrated; this is an evidence-based approach to assessing the effects of programmes and policies on health equity. Such framework also assesses barriers and facilitators for improving health equity using four factors: access, diagnostic accuracy, adherence and compliance. This framework emphasises the importance of transferring knowledge for designing and implementing clinical and public health interventions to improve health in all socioeconomic strata, based on the best available evidence. The World Health Organisation Collaborating Centre on Health Technology Assessment is now using this framework for organising the components of its equity-orientated, evidence-based toolkit.
Key Words: Clinical research, epidemiology, population group, inequality, socioeconomic factor (source: MeSH, NLM)
La salud poblacional es concebida como la reducción de las inequidades en salud, definida como las diferencias evitables e injustas en salud. Nosotros describimos e ilustramos el marco de equidad-efectividad; una aproximación basada en evidencia para evaluar los efectos de programas y políticas sobre la equidad en salud. Este marco además evalúa las barreras y los facilitadores para mejorar la equidad a través de cuatro factores: acceso, exactitud diagnóstica, adherencia y conformidad. Este marco enfatiza la importancia de la transferencia de conocimiento para diseñar e implementar intervenciones en salud públicas y clínicas para mejorar la salud a través de todos los estratos socioeconómicos, basados en la mejor evidencia disponible. El Centro Colaborativo de la Organización Mundial de la Salud sobre Evaluación de Tecnologías Sanitarias está utilizando actualmente este marco para organizar su Paquete de Herramientas basadas en evidencia y orientadas por la equidad.
Palabras Clave: Investigación biomédica, epidemiología, grupos de población, desigualdades en la salud, factores socioeconómicos (fuente: DeCS, BIREME).
The transdisciplinary field of population health has been defined as an approach which increases our understanding of the determinants of health and reaffirms the need for public health professionals to examine critically social inequities and policies that maintain them (1). Hence the overarching goal of population health is to determine what mix of economic, environmental, social, and medical care programs and policies will enhance both health and health equity (2). Two principles follow from this definition of population health. The first is that interventions aimed at changing the entire populations incidence of disease (e.g. by public cross-sectoral policy) are likely to have a greater (and perhaps more sustainable) impact than interventions aimed at preventing individual cases of disease (e.g. by modifying individual behavior and risk factors) (3). The second principle is that health is unevenly distributed in the population, with a gradient in health from the most socially disadvantaged to the most privileged and advantaged members of society, where the most disadvantaged have the worst health (4,5). As the field of population health moves from descriptions of health inequalities towards developing, implementing, and evaluating interventions aimed at improving population health, it is challenged to ensure that interventions benefit the disadvantaged.
The thesis of this paper is that without an explicit assessment of the impact of population health interventions on health inequalities, policies and programs aimed at improving population health run the risk of benefiting only the more privileged and better-off without improving the health of the poor.
What is population health?
As described above, population health is based on the premise that social and environmental determinants outside the health care system are responsible for inequalities in health, and that programs and policies are needed to enhance the conditions in which we live and work; these conditions are affected by non-medical, social determinants of population health. Population health has been described as a return to the roots of public health from the 19th century and earlier which focused on population-wide sanitary and nutritional interventions (1,6). In fact, schools of public health in low-and middle-income countries (LMICs) remain concerned with these social determinants of health and intersectorial policies.
Population health emerged in Canada and the United States as a field to study health inequalities, their determinants, and effects of interventions aimed at reducing them. Population health as a scientific discipline gained traction in Canada after the publication of «Producing health, consuming health care» by Evans and Stoddart (7), which emphasized that health and health inequalities are determined not only by health care, but also by factors outside the health care system including genetic, environmental, and social factors as well as individual responses (behaviours) and economic prosperity. Hence, this model proposed that social determinants of health might have a larger effect on health inequalities than behavioural, lifestyle factors. For example, only 25 % of the inequalities in health across social class in the Whitehall study could be attributed to behavioural risk factors (1). Furthermore, Thomas McKeowns work has been used to argue that medical care played only a small role in the decrease in infectious disease rates in the 20th century (8); most of the decline was due to increased economic prosperity, public health interventions to improve sanitation, improved nutrition, and changes in reproductive behaviour in the 20th century (9). This population health model was described as a «paradigm shift» since public health in Canada and the US had become focused on health care and clinical preventive actions, rather than upstream policies aimed at non-medical determinants of health.
The importance of assessing and taking action on the role of social determinants of health inequalities and health has been adopted internationally. For example, the World Health Organization Commission on the Social Determinants of Health (CSDH) has released its interim statement which re-affirms the primary question of the Commission: to identify what action is needed to address the «causes of causes», highlighting the difference between determinants of health and determinants of inequalities in health (10). Furthermore, this report emphasizes the continued importance of health systems and primary health care to ensure that health systems respond to population needs.
How do determinants of health cause disease?
There are at least four proposed models for how social factors cause ill health: the materialist, psychosocial, social production of disease, and ecosocial models (9,11).
The Black report proposed a materialist/structuralist explanation that a low individual income level leads to a lack of resources to cope with stressors of life and thus produces ill health (6,12,13). The psychosocial model proposes that discrimination based on one’s place in the social hierarchy causes stress which triggers a neuroendocrine response that produces disease (6,14-16). The social production of health model is based on the premise that capitalist priorities for accumulating wealth and material assets are achieved at the cost of the disadvantaged. For example, income tax cuts introduced in Ontario in the name of benefiting the poor resulted in a benefit of $ 15 586 per year for the richest 0,5 % of families and $ 150/year for the poorest 10 % (17). Perhaps the most compelling theory is the ecosocial theory, developed by Nancy Krieger, which brings together the psychosocial and social production of health models. This approach looks at the complexities of how social and physical environments interact with biology and how individuals “embody” aspects of the contexts in which they live and work (6,9). The ecosocial approach builds on the “collective lifestyles” approach and the neo-Weberian theory that lifestyle choices are influenced by life chances defined by the environment in which people live (12,18). These models highlight the role of social structure in determining and understanding behaviour, and suggest a need for narrative and qualitative research to understand the contexts of social environments (11,19,20).
Determinants of health fall into four broad categories: 1) environment (e.g. social cohesion, physical environment); 2) behaviour (e.g. personal health practices); 3) social groupings that have experienced discrimination or exclusion (e.g. income, gender, culture); and 4) biology or genetics (Table 1). The Cochrane Health Equity Field emphasizes that social groupings include the interaction of eight factors, defined by the acronym PROGRESS: Place of residence, Race/ ethnicity, Occupation, Gender, Religion, Education, Socioeconomic status, and Social capital (16). Classification of determinants according to how they might affect health is necessary to design interventions to address them (Table 1). For example, environmental determinants of injuries might best be addressed by cross-sectoral, upstream legal and regulatory policies to improve road safety, such as speed limit legislation and enforcement which have been shown to reduce crashes resulting in injuries by 45 % (21).
Health equity as a goal of population health
Reduction of social inequities was described as a goal of population health in 1995 (1), and this goal remains important for the field of population health (2,22).
Inequalities in health are considered health inequities if they are unnecessary and avoidable but, in addition, are also considered unfair and unjust (23). This definition is difficult to translate into policies and programs because it requires a normative judgment of fairness as well as whether inequalities are avoidable. Paula Bravemans most recent definition of health equity eliminates the need for this normative judgment of fairness, but still requires an assessment of whether the difference can be shaped by policy. Braveman defines heath disparity as a particular type of difference in health or in the most important influences on health that could potentially be shaped by policies; it is a difference in which disadvantaged social groups (such as the poor, racial/ethnic minorities, women or other groups that have persistently experienced social disadvantage or discrimination) systematically experienced worse health or greater health risks than more advantaged groups (24).
Accepting health equity as a goal of population health implies that policies and programs that improve average health but increase the gap between rich and poor, or across any other social group, would be considered ineffective. By assessing how policies affect health disparities, we can respond to the original call by Fraser Mustard that we need evidence on whether to develop strategies to intervene with social and economic problems from the standpoint of prevention to modify the impact of cultural, economic and social factors on the health of populations or whether we would prefer to increase the services of human care for individuals whose health becomes abnormal as a consequence of their genetic make-up and the environment in which they live(25).
Population health is measured by both distribution and average indicators
Kindig and Stoddart challenged the field of population health to identify a measurement framework as a tool for designing and evaluating interventions aimed at improving population health (26). McDowell and colleagues have answered this call by classifying population health outcomes according to whether the aim is to measure health in a population (measured by aggregate measures of individual health status such as mortality or morbidity) or to measure the health of a population, measured by global measures of health-promoting characteristics of populations, such as volunteerism, social cohesion, and water quality (22).
Reidpath proposes that we need to go beyond aggregate measures of individual health, as have been used by the global burden of disease study, to consider the average level of health, the distribution of health, and perhaps also the determinants of health in the population (27). The World Bank study of health inequalities in 56 countries is one example of assessing both the average health as well as the distribution across the socioeconomic gradient (i.e. by gender, income, and place of residence) (28).
Equity-effectiveness loop: a framework for evaluating impact on health equity
The equity-effectiveness loop framework proposes that interventions introduced in the name of improving population health need to be assessed to ensure that they benefit the disadvantaged (29). For example, mass media interventions have been shown to benefit the most affluent, but have little impact on the disadvantaged (13).
The loop framework consists of an iterative cycle of five steps for developing and evaluating population health policies and programs (Figure 1): 1) burden of illness; 2) community effectiveness; 3) cost-effectiveness; 4) knowledge translation; and 5) monitoring. The equity-effectiveness loop provides a logical progression of health research from assessing the problem, to designing interventions to address the needs, assessing cost-effectiveness of these approaches, implementing interventions, and evaluating their impact on the problem. Hence, the equity-effectiveness loop is a call to action for evaluating effects of interventions on population health.
The equity-effectiveness loop measures both the risk (burden of illness) and response (effectiveness) across social, demographic and geographic factors in which disadvantage might exist, with an emphasis on translating this knowledge into interventions designed to improve population health. Categories across which disadvantage might exist can be assessed using the acronym PROGRESS, as described above (30).
The equity-effectiveness loop assesses potential for health inequity at each step, thus insisting on an assessment of whether an intervention is likely to benefit the disadvantaged. Effectiveness is estimated using a multiplicative model with the following four factors: diagnostic accuracy, coverage/access, provider compliance, and consumer adherence. All of these factors may be lower in disadvantaged groups thus creating a staircase effect whereby the disadvantaged are further disadvantaged (29).
The equity-effectiveness loop framework explicitly focuses on moving from measurement of health inequalities to designing, implementing and evaluating interventions that reduce health inequalities.
Steps of the equity-effectiveness loop
We will highlight the principles of the equity-effectiveness loop using an example related to improving nutritional status in children: school meals. School meals improve growth by 13 %, and this improvement is greater for younger children and those with lower nutritional status (31).
Step 1: Burden of illness by income
Reduction of the under-five mortality rate by two-thirds is the target of the fourth of the eight Millennium Development Goals (32). Though improvements have been made, under-five mortality remains unacceptably high; UNICEF estimates that under-five mortality in developing countries has decreased from 105 per 1000 in 1990 to 88 per 1 000 in 2004 (33). In comparison, under-five mortality in the United States is 8 per 1 000 (34). Approximately 50 % of under-five mortality is attributed to poor nutrition (33).
These population averages hide inequalities in health across the wealth gradient. Data from 22 low and lower-middle income countries shows progress in reducing under-five mortality rate in 17 out of 22 countries, but only five of these countries showed enhanced health equality by reducing the gap between richest and poorest in under-five mortality (35). Data for monitoring these changes over time for both average health status and inequalities in health are lacking in most countries (36).
Step 2: Community effectiveness by socioeconomic status We define community effectiveness as the benefit derived from an intervention once it is implemented in the community. Thus, community effectiveness depends on the efficacy of the intervention, as well as on real-world modifiers of efficacy: access, diagnostic accuracy, provider compliance, and consumer adherence. We use a multiplicative model to assess the impact of these four implementation factors on community effectiveness.
Efficacy is assessed by the most up-to-date, relevant, rigorously conducted systematic review, where possible (37). Systematic reviews are proposed as a more reliable source of efficacy than single trials because they bring together all available evidence on the topic, are more easily contested, and can contain the data for assessing contextual factors such as the role of setting (38).
We define access as the extent to which an efficacious intervention can be used by those who need it (39). This definition of access entails a needs-based approach, which implies a value judgment on need, and is consistent with taking a social view of need (40). Access depends on five factors, identified by Thomas and Penchansky: 1) availability; 2) accessibility; 3) affordability; 4) acceptability; and 5) accommodation (defined as a modification of an intervention to suit consumer preferences) (41). Diagnostic accuracy is estimated as the proportion of people in need of an intervention who are accurately diagnosed as at risk. Provider compliance is estimated as the likelihood that health care providers will prescribe or recommend an intervention for someone in need. Finally, consumer adherence measures the degree to which individuals are likely to adhere to an intervention once it is prescribed or offered. Consumer adherence depends on resources available to implement the intervention (eg time, financial) as well as preferences, values and attitudes to the intervention.
Applying these definitions to schoolmeals yields a least poor to poorest ratio of 2,5 (Table 2). The largest contributors to differences between the least poor and poorest are access, estimated by school participation and adherence, estimated by school completion rates using data from Kenya. Estimates need to be drawn from appropriate datasets, taking into account setting and context. This example highlights the potential for programs that are aimed at the poor and disadvantaged to further worsen inequities in health, if the poorest are unable to access these interventions. Furthermore, there is a need to consider cross-sectoral interventions to promote enhanced school attendance and completion. For example, conditional cash transfers have been shown to be effective at increasing childrens attendance for both health visits and education, with mixed effects on health outcomes and school achievement (42).
Step 3: Economic evaluation by socioeconomic status
Assessing equity implications requires a consideration of equity and efficiency trade-offs. Studies have shown that both policy-makers and members of the general public are willing to sacrifice average health for enhanced health equity (43,44). An overview of different methods that can be used to consider equity implications in economic evaluation of public health interventions has recently been completed and provides four approaches to considering equity in economic analyses, funded by the UK Public Health Research Consortium (45).
Step 4: Knowledge translation by socioeconomic status
Knowledge translation has been defined by the Canadian Institutes of Health Research as the exchange, synthesis and ethically-sound application of knowledge - within a complex system of interactions among researchers and users - to accelerate the capture of the benefits of research for Canadians through improved health, more effective services and products, and a strengthened health care system (46).
This knowledge translation step entails taking what is known about barriers to community effectiveness to design, develop, and implement both clinical and public health interventions to reduce the gap in child health across socioeconomic strata. Models of knowledge translation highlight the need to assess barriers and facilitators to the uptake and application of knowledge for specific audiences defined by the six «P»s: practitioners, policy-makers, public, patients, press and private sector (47-50). Furthermore, evidence needs to be packaged into «evidence-based actionable messages» (EBAMs), which give the main message and policy action based on the evidence in accessible language (49).
Knowledge translation must also take into account the context, setting, and interaction with other ongoing initiatives, as well as assess the interaction of biology and the social and physical environment, across the socioeconomic gradient. Since the identified barriers for school meals were access and adherence, knowledge translation strategies need to focus on how to enhance school enrolment and completion.
Knowledge translation programs and policies need to be chosen based on consideration of social and environmental factors as well as the mechanisms by which the proposed policy or program is expected to improve health, and how these mechanisms of action might be affected by population characteristics, provider characteristics, context, and setting. We have recently shown how qualitative methods can complement systematic reviews to explore theories regarding how schoolfeeding can best be implemented (31). These methods may be useful in deciding on whether and how to implement policies and programs that include both preventive and treatment interventions.
Step 5: Monitoring of program and Step 6: Re-assessment These two steps assess the impact of knowledge translation strategies across PROGRESS factors. The program would be considered successful if it reduced inequalities across PROGRESS factors. Continued evidence of health inequalities across PROGRESS factors indicates a need to cycle through the loop again.
Contributions of the equity-effectiveness loop framework to population health
The equity-effectiveness loop framework contributes to the field of population health in four ways. First, it promotes evaluation and re-assessment of interventions aimed at enhancing health equity. Without this evidence base, society cannot make an informed decision about the trade-offs of investing in health (care) services versus upstream policies and programs that address non-medical determinants of health.
Second, this framework reinforces the assessment of health equity and distribution of health as a goal of population health interventions. Health and social programs and policies designed to reach poor people have sometimes been shown to benefit the more privileged members of society, such as free smoke-alarms in the United Kingdom (51) and uptake of immunization services in low and middle income countries (52). Hence, there is a need for evidence on the distributional impact of policies and programs.
Third, this framework emphasizes the need to assess barriers and facilitators associated with mitigating the impact of determinants of health in the knowledge translation step. This step acknowledges the importance of a blend of qualitative and quantitative methods to understand the context in which behaviours and determinants cause ill health. Further, this step requires transdisciplinary collaboration to assess interactions of sociology, biology, law, management, medicine, and ecology.
Fourth, this framework recognizes the importance of both downstream (individual level) and upstream interventions (community, society or population level). For example, a population health strategy for improving nutritional status might include both upstream interventions to improve national food quality and quantity (eg agricultural trade policies) and downstream interventions that aim to improve nutritional intake (eg school meals).
Weaknesses of the equity-effectiveness framework
The equity-effectiveness framework is based on clinical epidemiologic methods and hence the assessment of community effectiveness is framed for interventions provided by a clinician provider. Therefore, this framework may not be useful for legal or policy interventions which do not include a provider, such as tobacco smoking bans. The community-effectiveness step is based on a hypothetical multiplicative model (i.e. that multiplying efficacy by the four factors provides an estimate of effectiveness in the community) (39). Because this multiplicative model has not been tested with empirical data, its relevance might be questioned. However, the utility of this step is mainly as a heuristic tool in identifying the greatest barriers to achieving maximum benefit of interventions for both the poorest and the least poor, not assessing the community effectiveness. Because the equity-effectiveness loop has been developed by clinical epidemiologists, researchers from other disciplines may reject it as being rooted in the biomedical model. This framework insists on measuring the gap between the most advantaged and most disadvantaged, hence it does not measure the gradient in health across all levels of disadvantage. Finally, the equity-effectiveness loop does not align itself with any particular theory of how social determinants of health produce ill health (eg materialist, ecosocial), hence it runs the risk of being criticized for being a theoretical.
In conclusion, we propose the equity-effectiveness loop as a tool for insisting on evaluations of the impact of population health interventions on health inequities to ensure that interventions benefit the disadvantaged, and ensure re-assessment and reiteration of the loop if disparities still exist. The equity-effectiveness loop is a simple, logical progression of steps for designing, implementing, and evaluating policies and programs to improve population health, with a focus on the distribution of health. This framework requires a transdisciplinary approach to understanding structural and individual contexts in which ill health are produced and designing strategies to overcome them. It also challenges the field of population health to design, implement, and evaluate interventions that aim to reduce disparities in health, and insists on an explicit assessment of the impact of interventions on health disparities. Only by evaluating the impact of these policies can we decide if they are worth the investment, and if not, there needs to be a reiteration of the equity-effectiveness loop to identify other potentially effective knowledge translation strategies.
As the WHO Collaborating Center on Health Technology and Assessment and Knowledge Translation for Health Equity continues to promote and use the equity-effectiveness loop with its equity-oriented HTA toolkit and technical assistance activities, we will also evaluate its usefulness and applicability. However, the true test of this frameworks utility for population health will be whether others find it useful, and whether it leads to an increase in our understanding of how to improve population health, defined as both the overall health and the distribution of health in a population
Acknowledgments. Vivian Welch is supported by a Canada Graduate Scholarship from the Canadian Institutes of Health Research. Peter Tugwell is supported by a Canada Research Chair on Health Equity.
1. Frank J. Why population health? Can J Public Health 1995;86(3):162-4.
2. Kindig DA. Understanding population health terminology. Milbank Q 2007;85(1):139-61.
4. Black D, Morris J, Smith C, Townsend P. Inequalities in health: report of a Research Working Group. London, UK: Department of Health and Social Security; 1980.
5. Social determinants of health. The solid facts. 2nd ed. Geneva: World Health Organization; 2003.
6. Goldberg M, Melchior M, Leclerc A, Lert F. Épidémiologie et déterminants sociaux des inégalités de santé. Revue dépidémiologie et de santé publique 2003;51:381-401.
7. Evans RG, Stoddart GL. Producing Health, Consuming Health Care. In: Evans RG, Barer ML,Marmor TR, editors. Why Are Some People Healthy and Others Not? The Determinants of Health of Populations.New York: Aldine de Gruyter; 1994.
8. McKeown T. Determinants of health. Human Nature 1978;1(4):60-7.
9. Krieger N. Theories for social epidemiology in the 21st century: an ecosocial perspective. Int J Epidemiol 2001;30(4):668-77.
10. Commission on Social Determinants of Health. CSDH. Commission on Social Determinants and Health. Geneva: World Health Organization; 2007.
11. Beattie A. Figures in an epigenetic landscape: competing paradigms of complexity in epidemiology. Working paper. Lancaster: St Martins College; 2004.
12. Frohlich KL, Corin H, Potvin L. A theoretical proposal for the relationship between context and disease. Sociology of Health & Illness 2001;23(6):776-97.
13. Macintyre S. The Black report and beyond what are the issues? Social Science & Medicine 1997;44(6):723-45.
14. Karasek R. Job strain and the prevalence and outcome of coronary artery disease. Circulation 1996;94(5):1140-1.
15. Siegrist J, Marmot M. Health inequalities and the psychosocial environment-two scientific challenges. Social Science & Medicine 2004;58:1463-73.
16. Evans RG, Stoddart GL. Consuming research, producing policy? Am J Public Health 2003;93(3):371-9.
17. Raphael D. Health inequalities in Canada: current discourses and implications for public health action. Critical Public Health 2000;10(2):193-216.
18. Cockerham WC, Ritten A, Abel T. Conceptualizing contemporary lifestyles: moving beyond Weber. Sociological Quarterly 1997;38:601-22.
19. Williams GH. The determinants of health: structure, context and agency. Sociology of Health & Illness 2003;25:131-54.
20. Kickbusch I. The contribution of the World Health Organization to a new public health and health promotion. J Public Health 2003;93:383-8.
21. Wilson C, Willis C, Hendrikz JK, Bellamy N. Speed enforcement detection devices for preventing road traffic injuries. Cochrane Database of Systematic Reviews 2006;2006, issue 2.
22. McDowell I, Spassoff RA, Kristjansson B. On the classification of population health measurements. Am J Public Health 2004;94(3):388-93.
23. Whitehead M. The Concepts and Principles of Equity and Health. International Journal of Health Services 1992;22(3):429-45.
24. Braveman P. Health disparities and health equity: concepts and measurement. Annual Review of Public Health 2006;27:167-94.
25. Hayes MV, Dunn JR. Population Health in Canada: a systematic review. Ottawa: CPRN: Canadian Policy Research Network; 1998. Report No.: CPRN Study No H|01|.
26. Kindig D, Stoddart GL. What is population health? Am J Public Health 2003;93:380-3. 27.Reidpath DD. Population health. More than the sum of the parts? J Epidemiol Community Health 2005;59(10):877-80.
28. Gwatkin DR, Rutstein S, Johnson K, Suliman E, Wagstaff A, Amouzou A. Socio-Economic Differences in Health, Nutrition, and Population. Washington,D.C.: The World Bank; 2007.
29.Tugwell P, de SD, Hawker G, Robinson V. Applying clinical epidemiological methods to health equity: the equity effectiveness loop. [Review] [0 refs]. BMJ 2006 Feb 11;332(7537):358-61.
30. World Health Organization. World Report on Knowledge for Better Health: Strengthening Health Systems. Geneva: World Health Organization; 2004.
31. Kristjansson EA, Robinson V, Petticrew M, MacDonald B, Krasevec J, Janzen L, et al. School feeding for improving the physical and psychosocial health of disadvantaged elementary school children. [Review] [126 refs]. Cochrane Database of Systematic Reviews 2007;(1):CD004676.
32. United Nations. United Nations. The Millennium Development Goals Report. United Nations. New York: United Nations; 2006.
33. UNICEF. UNICEF: Childinfo.org; Child Mortality. www childinfo org 2008 Available from: URL: www.childinfo.org/areas/childmortality/
34. WHOSIS. World Health Organization-statistical information system (WHOSIS); 2007.
35. Moser KA, Leon DA, Gwatkin DR. How does progress towards the child mortality millennium development goal affect inequalities between the poorest and least poor? Analysis of Demographic and Health Survey data. BMJ 2005;331(7526):1180-2.
36. Attaran A. An immeasurable crisis? A criticism of the millennium development goals and why they cannot be measured. PLoS Medicine / Public Library of Science 2005;2(10):e318.
37. Tugwell P, Robinson V, Grimshaw J, Santesso N. Systematic reviews and knowledge translation. Bulletin of the World Health Organization 2006 Aug;84(8):643-51.
38. Lavis JN, Davies HTO, Gruen RL. Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy 2006 Jan 1;1(2):21-33.
39. Tugwell P, Bennett KJ, Sackett DL, Haynes RB. The measurement iterative loop: a framework for the critical appraisal of need, benefits and costs of health interventions. J Chronic Dis 1985;38(4):339-51.
40. Mooney G. What does equity in health mean? World Health Stat Q 1987;40(4):296-303.
41. Thomas JW, Penchansky R. Relating satisfaction with access to utilization of services. Med Care 1984;22(6):553-68.
42. Lagarde M, Haines A, Palmer N. Conditional cash transfers for improving uptake of health interventions in low- and middle-income countries: a systematic review. JAMA 2007;298(16):1900-10.
43. Hagberg LA, Lindholm L. Is promotion of physical activity a wise use of societal resources? Issues of cost-effectiveness and equity in health. Scand J Med Sci Sports 2005;15(5):304-12.
44. Dolan P. The measurement of individual utility and social welfare. J Health Econ 1998;17(1):39-52.
45. Drummond M. Assessing the challenges of applying standard methods of economic evaluation to public health programmes. London: Public Health Research Consortium; 2007.
46. Canadian Institutes of Health Research (CIHR). Knowledge Translation Strategy 2004-2009: Innovation in Action. Ottawa: Canadian Institutes of Health Research (CIHR); 2004.
47. Lomas J. Using linkage and exchange to move research into policy at a Canadian foundation. Health Aff (Millwood) 2000;19(3):236-40.
48. Graham ID, Logan J. Innovations in knowledge transfer and continuity of care. Can J Nurs Res 2004;36(2):89-103.
49. Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. Journal of Health Services & Research Policy 2003 Jul;8(3):165-70.
50. Tugwell P, Robinson V, Grimshaw J, Santesso N. Systematic reviews and knowledge translation. Bull World Health Organ 2006 Aug;84(8):643-51.
51. DiGuiseppi C, Roberts I, Wade A, Sculpher M, Edwards P, Godward C, et al. Incidence of fires and related injuries after giving out free smoke alarms: cluster randomised controlled trial. BMJ 2002;325(7371):995.
52. Yasbeck AS, Gwatkin DR, Wagstaff A, Qamruddin J. Reaching the poor with health, nutrition, and population services: what works, what doesnt and why. Washington, DC: World Bank; 2005.