ORIGINAL RESEARCH INVESTIGACIÓN ORIGINAL
Barriers and facilitators to establishing a national public health observatory
Barreras y facilitadores para el establecimiento de un observatorio de salud pública nacional
Shalini PooransinghI; Akenath MisirII; Dan RamdathI; Samuel RamsewakI; Susan JaglalIII; Cathy CameronIV; Vivek GoelV
IThe University of the West Indies, St Augustine, Mount Hope, Trinidad and Tobago. Send correspondence to: Shalini Pooransingh, Shalini.firstname.lastname@example.org
IIOffice of the County Medical Officer of Health-Victoria, San Fernando, Trinidad and Tobago
IIIUniversity of Toronto, Toronto, Ontario, Canada
IVInternational Centre for Disability and Rehabilitation, Toronto, Ontario, Canada
VInstitute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
OBJECTIVE: To determine what stakeholders perceive as barriers and facilitators to creating a national public health observatory (PHO) in Trinidad and Tobago.
METHODS: A descriptive study was conducted based on 15 key informant interviews carried out from April to September 2013. The key informants worked within the health care sector in Trinidad and Tobago. Using a semi-structured interview guide, information was collected on knowledge, attitudes, and beliefs about creating a PHO; barriers and facilitators to creating and sustaining a PHO; legal considerations; and human resource and information technology requirements. Common themes of the responses were identified.
RESULTS: The majority of participants supported the development of a national PHO, recognized its value in informing their work, and indicated that a national PHO could 1) provide information to support evidence-informed decision-making for health policy and strategic planning; 2) facilitate data management by establishing data policies, procedures, and standards; 3) increase the use of data by synthesizing and disseminating information; and 4) provide data for benchmarking. However, a number of barriers were identified, including 1) the perception that data collection is not valued; 2) untimely availability of data; 3) limited data synthesis, dissemination, and utilization to inform decision-making; and 4) challenges related to the allocation of human resources and existing information technology.
CONCLUSIONS: Key informants support the development of a national PHO in Trinidad and Tobago. The findings align well within the components of the conceptual framework for establishing national health observatories. A stepwise approach to establishing a national PHO in Trinidad and Tobago, beginning with structural components and followed by functional components, is recommended. A national PHO in Trinidad and Tobago could serve as a model for other countries in the Caribbean.
Key words: Public health; health care sector; qualitative research; decision making; data collection; Trinidad and Tobago; West Indies.
OBJETIVO: Determinar las barreras y los facilitadores percibidos por los interesados directos para crear un observatorio de salud pública (OSP) nacional en Trinidad y Tabago.
MÉTODOS: Se realizó un estudio descriptivo basado en 15 entrevistas a informantes clave llevadas a cabo de abril a septiembre del 2013. Los informantes clave trabajaban en el sector de la atención de la salud en Trinidad y Tabago. Mediante el empleo de una guía de entrevista semiestructurada, se recopiló información sobre conocimientos, actitudes y creencias acerca de la creación de un OSP; barreras y facilitadores para crear y mantener un OSP; consideraciones legales; y necesidades en cuanto a recursos humanos y tecnología de la información. Se determinaron los temas comunes de las respuestas.
RESULTADOS: La mayor parte de los participantes brindaron apoyo a la creación de un OSP nacional, reconocieron su valor para fundamentar su trabajo e indicaron que un OSP nacional podría 1) suministrar información para brindar apoyo a la toma de decisiones basada en la evidencia a efectos de política sanitaria y planificación estratégica; 2) facilitar el manejo de datos mediante el establecimiento de políticas, procedimientos y estándares de datos; 3) incrementar la utilización de los datos mediante la síntesis y la difusión de la información; y 4) proporcionar datos para la evaluación comparativa. Sin embargo, se indicaron varias barreras, incluidas 1) la percepción de que la recopilación de datos no se evalúa; 2) la inoportunidad de la disponibilidad de los datos; 3) las limitaciones en la síntesis, difusión y utilización de los datos para fundamentar la toma de decisiones; y 4) las dificultades relacionadas con la asignación de recursos humanos y de la tecnología de la información existente.
CONCLUSIONES: Los informantes clave brindan apoyo al desarrollo de un OSP nacional en Trinidad y Tabago. Los resultados se alinean bien dentro de los componentes del marco conceptual para establecer observatorios de salud nacionales. Se recomienda un método progresivo para establecer un OSP nacional en Trinidad y Tabago, empezando por los componentes estructurales y siguiendo con los componentes funcionales. Un OSP nacional en Trinidad y Tabago podría servir de modelo para otros países del Caribe.
Palabras clave: Salud pública; sector de atención de salud; investigación cualitativa; toma de decisiones; recolección de datos; Trinidad y Tobago; Indias Occidentales
Sound health care decision-making relies on evidence that is credible, current, and readily available. Health information systems are therefore a critical component of health care systems, and a vital resource for decision-makers and governments. According to the World Health Organization (WHO) International Health Regulations (2005) (IHR) (1), countries are mandated to develop/strengthen the capacity to detect and respond to acute public health events. These events include communicable diseases such as dengue, chikungunya, Middle East Respiratory Syndrome (MERS), and, more recently, Ebola; the deliberate release of biological or chemical agents; and mass chemical poisoning. Early detection of these events is critical, and this requires a robust surveillance system. With the increasing burden of noncommunicable diseases (NCDs) worldwide, surveillance systems must extend beyond traditional surveillance areas (e.g., communicable diseases) and collect information on the determinants, risk factors, and outcomes of NCDs. Furthermore, information systems must incorporate data on the broad social determinants of health, which, according to the recommendations of the WHO Commission on Social Determinants of Health (2), requires interaction with sectors outside the health system.
Countries need a clear picture of their national disease burden to plan health services (3), monitor changes over time, and share information with neighboring countries to minimize cross-border spread of disease. Many sectors of the Ministry of Health (MoH), such as those responsible for vector-borne diseases, health promotion, and policy and planning, require information about population health and disease. However, in many Caribbean countries, information may be unavailable, inaccessible to decision-makers, not reported in the right format, or unreliable. Such scenarios may be due to resource limitation and/or small populations with lower numbers of skilled staff than may be present in developed countries. Successfully overcoming these challenges requires the creation of an in-country network of surveillance, information, and knowledge that could serve as a major resource for local entities working in public health. The resource should ideally be a national body that would collate and analyze data received from subnational bodies (4). The public health observatory (PHO) model meets these requirements and potentially provides an efficient means for addressing many of the challenges outlined above. For instance, in a network of observatories, each PHO could be responsible for one area of health policy (e.g., themed observatories) (5, 6). As reported by Watkins et al., professionals might not be aware of the data collected by different agencies (7).
PHOs are entities that receive data collected through health information systems and other agencies and analyze and disseminate high-quality information on the health and determinants of health of populations (8-10). PHOs can exist at various levels (global, regional, national, and subnational). These observatories can provide decision-makers with information in a timely manner, thereby enabling the rapid response required for certain health-related events. Effective and efficiently run PHOs can 1) make substantial contributions to health and 2) inform health care policies and service delivery.
The first PHO was established in Paris in 1974 to provide information to inform regional health policy (7, 8). Other PHOs were subsequently established throughout the country, and France now has a well-established observatory network (7, 11). England's first PHO was established in 1990 in Liverpool (8, 12). As in France, its purpose was to provide relevant authorities with health intelligence to facilitate the formulation of health policies. The Liverpool PHO has the elements considered key for PHOs (13, 14): it monitors health and disease trends, highlights areas for action, and identifies gaps in health information. It also utilizes information from different sources in new ways to improve health, and can make projections to give early warnings of potential public health problems. Examples of topics covered by the Liverpool PHO in its first 10 years include tuberculosis and poverty, asthma and environmental pollution, and environmental causes of death and disability (15). PHOs also exist outside Europe (e.g., Australia, Canada, and Latin America) (6, 16-18).
Many developing countries, including Trinidad and Tobago, do not have a national PHO, and there is no national PHO anywhere in the Caribbean region. Trinidad and Tobago's health care system comprises public and privately funded health care facilities. The MoH has authority over the entire health system. In 1994, publicly funded health care delivery became the responsibility of five Regional Health Authorities (RHAs) demarcated by geographic boundaries. The MoH sets policies and goals for and allocates resources to the RHAs. The chief medical officer (CMO) is the advisory and regulatory officer and reports directly to the Minister of Health. County Medical Officers of Health (public health doctors) report to the CMO. The MoH comprises vertical health services such as the Trinidad and Tobago Public Health Laboratory; the Queen's Park Counselling Centre and Clinic (for HIV and sexually transmitted infections); the Insect Vector Control Division; and the National Blood Transfusion Service.
The MoH National Surveillance Unit (NSU) is responsible for collecting, analyzing and disseminating health intelligence on communicable disease. Data are provided to the NSU by the surveillance units/officers of the various regions and should include data from both public and private sectors.
The private health sector includes practitioners, hospitals, medical laboratories, pharmacies, and diagnostic imaging institutes.
Establishing a national PHO at the MoH could 1) complement the existing observatory functions in the RHAs, and the work of WHO's Evidence-Informed Policy Network (EVIPNet) (19) for the Americas, including Trinidad and Tobago, and 2) strengthen the work of Trinidad and Tobago's MoH Policy Unit.
The purpose of this study was to determine what stakeholders perceive as barriers and facilitators to establishing a national PHO in Trinidad and Tobago. The specific objectives were to determine 1) the factors that would act as barriers to establishing a national PHO; 2) the factors needed to support the establishment of a national PHO; and 3) the specific functions of a national PHO, as perceived by the respondents.
MATERIALS AND METHODS
A stakeholder advisory group (SAG) was established with representatives from the research team (from Canada and Trinidad and Tobago) and the RHAs in Trinidad. Team members comprised experts in the area of public health and qualitative and quantitative research, and individuals with experience in practicing public health (including providing and utilizing data from PHOs). A SAG meeting was organized at the onset of the project to 1) finalize study execution and data collection tools; 2) develop a list of potential key informants; 3) finalize the interview guide; and 4) identify documents for review. Team members also met with other stakeholders to discuss the research project in more detail and obtain input on project implementation. A graduate student from The University of the West Indies Master of Public Health program (St. Augustine), trained in qualitative interviewing by three of the researchers (SJ/SP/CC), conducted the data collection.
The study was descriptive and utilized one-on-one semi-structured interviews. Key informants were purposively sampled to recruit individuals with a range of experiences from within the health sector in Trinidad and Tobago and the Caribbean region. Interviews ranged in length from 15 to 50 minutes and were completed at a time and place mutually agreed upon either by phone or in person and were tape-recorded with the permission of the participant. Interviews were not transcribed verbatim but detailed summary notes were developed for each interview using the tape recordings and the written interview responses and notes. No member checking was done but a validation and knowledge exchange workshop was held with stakeholders, including study participants and others, after the data collection and analysis.
A semi-structured interview guide was developed that included 14 open-ended questions related to the perceived value of a PHO; the barriers and facilitators to implementing a national PHO in Trinidad and Tobago; considerations for human resources and information technology; and legal issues (e.g., privacy and confidentiality). The guide was pilot tested prior to study implementation. A copy of the guide was provided to key informants in advance of each interview to allow them to review and consider their responses to obtain more in-depth feedback than top-of-mind responses would have allowed. A copy of the questionnaire is available from the authors upon request.
Content analyses of the detailed interview summary notes were conducted and common themes were inductively identified within and across questions. Once identified the themes were organized within the SWOT (strengths, weaknesses, opportunities and threats) framework by parameter. Microsoft Word (Redmond, Washington, United States) was used to manage the data. For the questions about knowledge, frequencies and percentages were calculated.
Ethics approval for the study was obtained from Trinidad and Tobago's MoH, The University of the West Indies (St. Augustine), and the University of Toronto.
Information collected during the initial stakeholder engagements was used to develop a SWOT analysis of issues related to the development of a national PHO. The results of the analysis are shown in Table 1.
After the initial SAG and stakeholder visits, the SAG agreed on a list of persons to interview.
A total of 21 potential key informants were contacted and 15 agreed to participate and were interviewed, resulting in a response rate of 71%. Reasons for nonresponse and/or nonparticipation in the interviews included 1) change in job designation, 2) being new in a staff position, 3) not able to be contacted, and 4) too busy.
Fifteen key informant interviews were completed between April and September 2013 with diverse representatives from Trinidad and Tobago's health sector. The key informants included county medical officers of health; chiefs of staff of general hospitals; representatives from PHOs within the RHAs; MoH personnel; and representatives from the Pan American Health Organization (PAHO) and the Caribbean Public Health Agency (CARPHA).
Knowledge about observatories
The interview began with the interviewers providing key informants with the following definition of a PHO: "A public health observatory is an institute or agency that collects and analyses health data then disseminates the information to relevant stakeholders, particularly those in a position to make informed health policy." When asked if they agreed with the definition, less than half of the respondents (a total of 7) responded positively. The respondents that disagreed said that while the definition was not incorrect, it was incomplete, and that a PHO should incorporate data on demographics and determinants of health in addition to measures of disease frequency. Eleven responders said they had worked with PHOs or similar agencies. When asked about the relevance of a PHO to their current organization, 12 cited the data collection function and four cited data synthesis, dissemination, and utilization of data to make informed decisions. Results for other knowledge parameters are shown in Table 2.
Barriers to establishing a PHO
The majority of respondents stated that there appeared to be a general lack of understanding in their organization about the value of health data. While all respondents recognized the importance of collecting, synthesizing, and utilizing good- quality data to inform decision-making, they said that this is not the norm and that there is "complete apathy" with respect to data collection. Many spoke about poor data quality (i.e., inaccurate, incomplete, and/or missing data) and the subsequent poor utilization of collected data.
Many respondents described the variability in the types of data collection forms being used across the RHAs, which makes comparisons difficult. They also indicated that the use of paper forms (manual versus electronic data entry) poses a barrier to efficiency, with almost all highlighting the absence of real-time data processing. Outdated data collection forms (e.g., surveillance forms); a lack of feedback on data quality; and infrequent reporting of health data were also identified as significant barriers.
In terms of structural elements, about half of the respondents cited IT and human resources as potential barriers. Less than one in five respondents noted funding as an issue. Data collection and data entry functions were described as "add-on" tasks incorporated with other responsibilities rather than dedicated roles. This was highlighted as a barrier to good-quality data management. A lack of staff training and sensitization on data management was cited as a related issue. As one participant stated, "A [cultural] shift is required-it will be a different way of doing business."
Respondents indicated that data are often not disseminated or used to inform decision-making. The flow of information is primarily in one direction, with limited feedback to data collection sites. There is limited sharing of data and information within the health care system (e.g., between RHAs), and limited benchmarking. Respondents also indicated that data are not synthesized and disseminated in formats useful for relevant stakeholders and policymakers.
Facilitators to establishing a national PHO
The majority of informants believed that increasing awareness among staff about the value and utility of health data would improve the efficiency of data collection, analysis, dissemination, and utilization. The majority of responders indicated political will, dedicated funding, IT support, and adequate and appropriate human resources would facilitate the establishment of a PHO. More than half of the respondents said legislation to mandate data collection might facilitate the establishment of an effective PHO, whereas less than one in five said making staff accountable would have that result.
Information technology in respondent organizations
The majority of responders reported some degree of IT connectivity (Intranet and/or Internet) in their organization, and software for processing health- related data. However, some respondents indicated improved connectivity and better software were needed. Specific needs cited included better analytic and reporting software, an upgraded IT framework, more widespread interconnectivity, and an information system incorporating health data with other sectors related to broader determinants of health.
When key informants were asked about the human resource requirements for a PHO within their organization, the following job titles were listed (in descending order of frequency): data entry clerks, statisticians, epidemiologists, research officers, IT staff, research assistants, and administrative staff. When asked about human resource requirements for a national PHO, the job titles listed included the following (in descending order of frequency): epidemiologists, statisticians, data entry clerks, research officers, IT staff, field officers, administrative staff, and research assistants.
The key informants interviewed represented health care organizations that might provide data to and/or utilize data from a national PHO. The majority of persons interviewed appeared to have a common understanding of PHOs as entities that produce and disseminate health intelligence to inform policy (8). Some respondents highlighted the need for PHOs to incorporate data on social determinants of health to be viewed/considered alongside health data.
Important elements of health information systems highlighted in this study included data quality (e.g., respondents reported case records with missing dates/diagnoses, and incomplete surveillance forms and death certificates, which lead to errors in coding and thus errors in the national statistics used to inform health-related policy decisions), and timeliness of the reporting (e.g., lack of timely surveillance does not allow real-time use of data on public health events of national or international concern). As noted by some respondents, timeliness can be addressed through the use of electronic systems. However, adding data management to staff functions can be problematic unless roles are clearly delineated, and poor data quality will not be solved by an electronic system unless automatic checks are built in. The current absence of standardized forms across institutions and/or health authorities does not permit comparison between regions and makes it difficult to collate data at the national level. Without accurate and complete data it would not be possible to use national data as a benchmark.
The perception of the respondents was that staff across all regions and at all levels have a poor understanding of the utility of good-quality, timely, health-related data. The lack of feedback to those providing the data (e.g., doctors in health centers) may also contribute to this "culture," as they do not see the information they collect being used in a meaningful way. This area of feedback has several dimensions (e.g., feedback on the quality of the data, and feedback in the form of reports that transform the data into useful information, particularly comparative data that could be used for quality improvement). Some of the perceived barriers and facilitators to PHOs described in this study are similar to those revealed in lessons learned from establishing observatories (e.g., high staff turnover, need for adequate resources and training) (16).
Appropriately qualified and trained staff, particularly epidemiologists and data entry personnel, were what the majority of respondents viewed as critical components for operationalizing a PHO, along with biostatisticians and staff with training/expertise in health informatics. This is consistent with the core skills identified in the document produced by the London Health Observatory on establishing a regional PHO, which cited epidemiology, data management, and statistics, as well as other skills like GIS and communication of health intelligence (17).
Moving forward on a PHO initiative will require fostering a culture that appreciates the value of data collection and the utilization of information for decision-making. If the prevailing culture does not value the collection, synthesis, and use of good-quality data, it does not matter which type of data collection and management system is used (e.g., manual versus electronic). Empowering decision-makers to be better users of data, implementing mechanisms for feedback and developing communication strategies to facilitate data utilization, could help achieve this cultural shift.
Strengthening human resource capacity is also needed. This could include reviewing job functions and re-engineering positions, sensitizing staff to the importance of data collection, developing strategies to mitigate the impact of staff turnover, and providing training and capacity development. For human resource allocation, a one-size-fits-all model is not sufficient (i.e., the needs, size, and capacities of different counties and regions within Trinidad and Tobago should dictate what is required to support the overall system).
The MoH can provide the governance and accountability structures to support this initiative by strengthening legislation and implementing mandatory collection and reporting of standardized data (private and public sector); identifying common datasets; and allocating sufficient resources (human, physical, and financial).
According to the London Health Observatory, the development of a PHO and its ability to fulfill its role will be influenced by whether or not, or to what degree: 1) public health is a priority within the health care system; 2) the public health structure is established at a national, regional and local level; 3) addressing health inequalities is a priority; 4) health and demographic data are routinely collected (e.g., ethnicity); 5) public health intelligence infrastructure and institutions have already been established; 6) gaps in public health intelligence exist and have been identified; and 7) the health system supports the standardization of data collection, tools, indicators, profiles, etc. (15).
Trinidad and Tobago falls short in terms of the criteria listed above because 1) medical records are often missing information on ethnicity and occupation and 2) there are gaps in the standardization of tools and indicators. However, the findings of this study fit within the three components of the national health information systems model (institutional, technological, and functional) (20), and the three components, in turn, neatly align with the mission, governance, and knowledge and intelligence components of the Urban Health Observatory conceptual framework (4, 17). Therefore, analysis of the study findings could be a starting point in advancing the PHO initiative.
In this study, which used purposive sampling, an interviewer trained in the qualitative interview technique conducted all interviews, and the responses were independently verified by another member of the research team. Study limitations included 1) possible selection bias in the recruitment of participants (e.g., those who agreed to participate in the research may have been more interested in PHOs than those who chose not to participate) and 2) possible reporting bias (e.g., if responders thought their answers could lead to their identification, potentially reflecting poorly on their institution, they may have modified them).
This study revealed that key informants in Trinidad and Tobago had no doubts about the value of a PHO, and some would be willing to participate in a steering group to establish and move forward on a PHO initiative. One approach to moving forward on the initiative would include outlining time-bound goals and, in the short term, elaborating the mission and governance elements. These latter elements would include the terms of reference, stakeholder involvement, strategic planning, and institutional support. Medium-term goals might include identifying human, financial, and physical resources. Longer-term goals would include determining the actual knowledge and intelligence components and/or the functional components as described in the national health information systems model. In this part of the process, issues with data collection, quality, and feedback could be addressed.
The study findings and potential way forward on a PHO initiative may apply to other Caribbean countries currently building their health information systems to strengthen their alert and response capacities to comply with the WHO IHR 2005 and address the increasing burden of NCDs.
Acknowledgments. The authors thank Richard Indalsingh for his role in data collection.
Funding. This work was carried out with the aid of a grant from the International Development Research Centre (Ottawa, Ontario, Canada) and through in-kind contributions from the University of Toronto, The University of the West Indies, the Trinidad and Tobago Ministry of Health, and Public Health Ontario. SJ is supported by the Toronto Rehabilitation Institute Chair at the University of Toronto (Toronto, Ontario, Canada).
Disclaimer. A poster presentation of this work was presented at the Caribbean Public Health Agency (CARPHA) Conference in April-May 2014. Authors hold sole responsibility for the views expressed in the manuscript, which may not necessarily reflect the opinion or policy of the RPSP/PAJPH or PAHO.
1. World Health Organization. International Health Regulations (2005). 2nd ed. Geneva: WHO; 2005. Available from: http://www.who.int/ihr/9789241596664/en/
2. WHO Commission on Social Determinants of Health. Closing the gap in a generation: health equity through action on the social determinants of health. Geneva: World Health Organization; 2008. Available from: http://www.who.int/social_determinants/thecommission/finalreport/en/
3. Global Burden of Disease Study 2013 Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990-2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet. 2015;386(9995):743-800. doi: 10.1016/S0140-6736(15)60692-4. Epub 2015 Jun 7.
4. Gattini CH. Improving the structure and performance of national health information systems: operational approach and strategic recommendations. Technical Series on Information for Decision-Making. Santiago: PAHO; 2009. (PWR CHI/09/HA/03). Available from: http://new.paho.org/chi/images/PDFs/national%20health%20information%20%20systems-final%20.pdf
5. Wilkinson J. Shining the light: public health observatories in England. J Public Health Med. 2003;25(3):273.
6. National Collaborating Centre for Determinants of Health. Public health observatories: learning from our world neighbours. Antigonish, NS: NCCDH, St. Francis Xavier University; 2011. Available from: http://www.gaelstream.stfx.ca/cdm/ref/collection/nccdh/id/1253
7. Watkins F, Bendel N, Scott-Samuel A, Whitehead M. Through a glass darkly: what should public health observatories be observing? J Public Health Med. 2002; 24(3):160-4.
8. Wilkinson J, Coyle E. Development of public health observatories in the UK, Ireland and Europe. Public Health. 2005; 119(4):227-34.
9. Hemmings J, Wilkinson J. What is a public health observatory? J Epidemiol Community Health. 2003;57(5):324-6.
10. Pourmalek F. National health observatories: need for stepped-up action. theHealth. 2012;3(3):63-4.
11. Fèdèration nationale des observatoires règionaux de santè (FR). [National Federation of Regional Public Health Observatories] [homepage]. Paris: FNORS; c2015. Available from: http://www.fnors.org/index.html Accessed on 11 February 2015.
12. Institute of Psychology Health and Society. Liverpool Public Health Observatory [Internet]. Liverpool, UK: University of Liverpool; c2015. Available from: http://www.liv.ac.uk/psychology-health-and-society/research/public-health-observatory/about/ Accessed on 11 February 2015.
13. Department of Health (GB). Saving lives: our healthier nation. London: The Stationery Office, DH Crown Copyright; 1999. Available from: https://www.gov.uk/government/publications/saving-lives-our-healthier-nation
14. Wilkinson J, Ferguson B. The first ten years of Public Health Observatories in England - and the next? Public Health. 2010;124(5):245-7.
15. Ashton JR. Public health observatories: the key to timely public health intelligence in the new century. J Epidemiol Community Health. 2000;54(10):724-5.
16. World Health Organization. Urban Health Observatories: a possible solution to filling a gap in public health intelligence. Geneva: WHO; 2013. Available from: http://www.who.int/kobe_centre/publications/uho_policybrief.pdf
17. Lam M, Jacobson B, Fitzpatrick J. Establishing a regional public health observatory: some questions answered. London: London Health Observatory; 2010. Available from: http://www.lho.org.uk/Download/Public/15755/1/Establishing%20a%20Regional%20PHO_v7_final.pdf
18. Cortez R, Ferl K. Improving governance and management of health systems: partnerships and observatories in Latin America and the Caribbean. Washington: World Bank; 2012. Available from: https://openknowledge.worldbank.org/handle/10986/17060
19. World Health Organization. Evidence-informed policy network [Internet]. Geneva: WHO; c2015. Available from: http://global.evipnet.org/ Accessed on 11 February 2015.
20. World Health Organization. Framework and standards for country health information systems. Geneva: Health Metrics Network, WHO; 2008. Available from: http://www.who.int/healthmetrics/documents/hmn_framework200803.pdf
Manuscript received on 11 February 2015.
Revised version accepted for publication on 29 July 2015.