POLICY & PRACTICE
WHO and UNICEF estimates of national infant immunization coverage: methods and processes
Estimations par l'OMS et l'UNICEF des couvertures vaccinales des nourrissons par pays: méthodes et procédures
Estimaciones de la OMS y el UNICEF sobre la cobertura nacional de vacunación de los lactantes: métodos y procedimientos
Anthony BurtonI,*; Roeland MonaschII; Barbara LautenbachII; Marta Gacic-DoboI; Maryanne NeillII; Rouslan KarimovII; Lara WolfsonII; Gareth JonesII; Maureen BirminghamI
IDepartment of Immunization, Vaccines and Biologicals, World Health Organization, 20 avenue Appia, 1211 Geneva 27, Switzerland
IIUnited Nations Children's Fund, New York, NY, United States of America
ABSTRACT
WHO and the United Nations Children's Fund (UNICEF) annually review data on immunization coverage to estimate national coverage with routine service delivery of the following vaccines: bacille Calmette-Guérin; diphtheria-tetanus-pertussis, first and third doses; either oral polio vaccine or inactivated polio vaccine, third dose of either; hepatitis B, third dose; Haemophilus influenzae type b, third dose; and a measles virus-containing vaccine, either for measles alone or in the form of a combination vaccine, one dose. The estimates are based on government reports submitted to WHO and UNICEF and are supplemented by survey results from the published and grey literature. Local experts, primarily national immunization system managers and WHO/UNICEF regional and national staff, are consulted for additional information on the performance of specific immunization systems. Estimates are derived through a country-by-country review of available data informed and constrained by a set of heuristics; no statistical or mathematical models are used. Draft estimates are made, sent to national authorities for review and comment and modified in light of their feedback. While the final estimates may not differ from reported data, they constitute an independent technical assessment by WHO and UNICEF of the performance of national immunization systems. These country-specific estimates, available from 1980 onward, are updated annually.
RÉSUMÉ
L'OMS et le Fonds des Nations Unies pour l'enfance (UNICEF) révisent chaque année les données de couverture vaccinale pour estimer les couvertures des pays par les services de vaccination systématique dans le cas des vaccins suivants : BCG, vaccin antidiphtérique-antitétanique-anticoquelucheux, première et troisième doses, vaccin antipoliomyélitique oral ou inactivé, troisième doses, vaccin contre l'hépatite B, troisième doses, vaccin contre Haemophilus influenzae type b, troisième dose ; et vaccin à valence rougeole, sous forme de vaccin antirougeoleux uniquement ou de vaccin combiné, une dose. Ces estimations reposent sur les rapports soumis par les gouvernements à l'OMS et à l'UNICEF et sont complétées par des résultats d'enquêtes, tirés de la littérature grise ou publiée. Des experts locaux, principalement des gestionnaires de systèmes de vaccination nationaux et des membres du personnel régional ou national de l'OMS et de l'UNICEF, sont complémentaires consultés pour obtenir des informations sur les performances de certains systèmes de vaccination. Les estimations sont obtenues à partir d'une analyse par pays des données disponibles, étayée et encadrée par un ensemble d'hypothèses heuristiques ; on ne recourt à aucun modèle statistique ou mathématique. Des estimations préliminaires sont établies, envoyées aux autorités nationales pour examen et commentaires et modifiées à la lumière des observations reçues. Bien que les estimations finales puissent ne pas différer des données rapportées, elles constituent une évaluation technique indépendante par l'OMS et l'UNICEF des performances des systèmes de vaccination nationaux Ces estimations par pays, disponibles depuis 1980, sont actualisées chaque année.
RESUMEN
La OMS y el Fondo de las Naciones Unidas para la Infancia (UNICEF) analizan cada año los datos sobre la cobertura de vacunación para calcular la cobertura nacional de los servicios de administración sistemática de las siguientes vacunas: bacilo de Calmette-Guérin; difteria-tétanos-tos ferina, primera y tercera dosis; vacuna antipoliomielítica oral o inactivada, tercera dosis de cualquiera de ellas; hepatitis B, tercera dosis; Haemophilus influenzae tipo b, tercera dosis; y una vacuna contra el virus del sarampión, ya sea sólo contra esta enfermedad o en forma de vacuna combinada, una dosis. Las estimaciones están basadas en los informes presentados por los gobiernos a la OMS y el UNICEF, complementados por resultados procedentes de diversas publicaciones y de la literatura gris. Se consulta además a expertos locales, principalmente responsables de los sistemas nacionales de inmunización y personal regional y nacional de la OMS y el UNICEF, para reunir información adicional sobre la eficacia de sistemas de inmunización concretos. Las estimaciones se basan en un análisis por países de los datos disponibles, fundamentado y limitado por un conjunto de reglas empíricas; no se emplean modelos estadísticos ni matemáticos. Las estimaciones preliminares son enviadas a las autoridades nacionales para que las examinen y formulen observaciones, y se modifican en función de esa retroinformación. Aunque no siempre distintas de los datos notificados, las estimaciones finales pueden considerarese el resultado de una evaluación técnica independiente realizada por la OMS y el UNICEF sobre el desempeño de los sistemas nacionales de inmunización. Disponibles desde 1980, estas estimaciones específicas para cada país son actualizadas cada año.
Introduction
WHO recommends that all children receive one dose of bacille Calmette-Guérin vaccine (BCG), three doses of diphtheria-tetanus-pertussis vaccine (DTP), three doses of either oral polio vaccine (OPV) or inactivated polio vaccine (IPV), three doses of hepatitis B vaccine, and one dose of a measles virus-containing vaccine (MVCV), either anti-measles alone or in combination with other antigens.1-9 It also recommends three doses of vaccine against infection with Haemophilus influenzae type b (Hib).10 To boost immunity at older ages, additional immunizations are recommended for healthcare workers, travellers, high-risk groups and people in areas where the risk of specific vaccine-preventable diseases is high.11
Immunization coverage levels and trends are used (i) to monitor the performance of immunization services locally, nationally and internationally; (ii) to guide strategies for the eradication, elimination and control of vaccine-preventable diseases;12-14 (iii) to identify areas of immunization systems that may require additional resources and focused attention;15,16 and (iv) to assess the need to introduce new vaccines into national and local immunization systems.17 Models of vaccine-preventable disease burden frequently include immunization coverage levels among their components.18-20 Coverage levels for measles vaccine and DTP are indicators of health system performance frequently considered by funding agencies when reviewing applications for financial and technical support.21-24 Measles immunization coverage is one of the indicators for tracking progress towards Millennium Development Goal 4, to reduce child mortality.25 Furthermore, trends in immunization coverage are used to establish the link between immunization service delivery and disease occurrence and to provide a framework for setting future coverage goals.26
Trends in immunization coverage
While some countries had routine immunization systems in place before 1980, major national and international development of routine, universal infant immunization systems did not begin until the late 1970s. In fact, it was not until the 1980s that dramatic improvements in immunization coverage were achieved, along with an increase in coverage with the third dose of DTP vaccine (DTP3) from 20% in 1980 to 75% coverage in 1990. While some countries reported significant declines in coverage after 1990, global coverage levels remained fairly constant and began rising slowly but steadily in 2000, until DTP3 coverage worldwide had reached 81% in 2006.
In 1980, fewer than 10% of the world's children lived in 20 of the 167 countries with DTP3 coverage levels greater than 80%; 84% of the world's children lived in countries where coverage was less than 50%. By 1990, 108 countries (43% of all children) had DTP3 coverage levels greater than 80%, and fewer than 10% of children lived in countries with under 50% coverage. Although national coverage levels can "mask" sub-national geographical or sociological pockets where coverage is much lower, in 2006, 57% of children lived in countries with greater than 80% DTP3 coverage. Still, that year approximately 26.3 million children who reached their first birthday did not receive DTP3, but 16.2 million (62%) of them lived in China, India, Indonesia or Nigeria. At the time this report was prepared, there remained seven countries where fewer than half of the children were vaccinated with three doses of DTP3: Angola, Central African Republic, Chad, Equatorial Guinea, Gabon, Niger and Somalia."
WHO and UNICEF publish annual estimates of national immunization coverage;27-30 such estimates have been available by country since 1980 at http://www.who.int/immunization_monitoring/en/globalsummary/wucoveragecountrylist.cfm and http://www.childinfo.org/immunization_countryreports.html. Additional analyses can be found at http://www.who.int/immunization_monitoring/data/en/ and http://www.childinfo.org/Immunization.htm. This paper describes the data, methods, assumptions and processes used to develop these estimates.
Key data sources
Administrative data based on reports from service providers (e.g. health centre staff, vaccination teams, private physicians) and surveys with items on children's vaccination history are the main sources of empirical data on immunization coverage. Administrative data report the number of vaccinations administered during a given period - usually 1 month - to local public health authorities who review the data and take any necessary action. The data are then aggregated and reported to the next administrative level and later aggregated, analysed and used at the national level to determine immunization policy, focus programme activities and influence resource allocation. National coverage data are reported annually to WHO and UNICEF.
Household surveys are the most common survey sources of immunization coverage data. Immunization history is determined either by looking at immunization records in the home, asking the child's caretaker (recall) or both. The three main household survey sources are the Expanded Programme on Immunization (EPI) cluster survey,31,32 the UNICEF Multiple Indicators Cluster Survey (MICS)33 and the Demographic and Health Survey (DHS).34 Designed specifically for measuring immunization coverage, EPI cluster surveys are simple to administer and frequently conducted by national EPI staff. The MICS and DHS are more extensive surveys with a variety of indicators and a more rigorous design. Although usually more precise, they are also more expensive, logistically more complex and more difficult to administer.
Each type of data source has advantages and disadvantages in both design and implementation. Administrative data provide more timely information and are useful for places where surveys may not be practical. In addition to providing information on coverage, such data can reveal service delivery problems (e.g. vaccine shortage, poor session attendance) early on. Coverage estimates based on administrative data are mainly subject to numerator (children vaccinated) and denominator (target population) biases. When too small a numerator is used because vaccinations are not reported by lower administrative levels or part of the population, such as the private sector, is excluded from the data collection or reporting system, coverage can be underestimated. It can also be overestimated when children vaccinated outside the target age group are erroneously included in the numerator. Estimates based on administrative data can also be biased by an inaccurate denominator, especially when outdated censuses and poor population projections are used. For instance, when coverage is high and the target population has been underestimated, estimated coverage can exceed 100%.
Survey data allow for estimating immunization coverage even if the size of the target population is unknown; they also include vaccinations given by the private sector. Their main disadvantage, however, is their lack of usefulness for timely programme interventions because they provide information only on the previous birth cohorts. In addition, they may yield confidence intervals that are wider than desired, interviewers may be poorly trained, supervision may be weak and analyses may be erroneous. Because surveys are rarely performed at the district level or lower, they do not provide information on local system performance. Respondent recall biases may be significant and their direction may be unknown. In some instances, the length or complexity of the questionnaire may compromise the accuracy of the responses. Finally, as with any survey, the results may be inappropriately generalized beyond the survey population.
Both administrative and survey methods are vulnerable to inadvertent recording, calculation and transcription errors; non-compliance with the established protocol due to poor training and supervision, and systematic and purposeful data fabrication.
WHO and UNICEF review processes
In June 2000, WHO and UNICEF began a retrospective review of data on national immunization coverage for 1980-1999. Reports by national authorities to WHO and UNICEF and survey data from the published and grey literature were reviewed. Based on these data and with due consideration to potential biases and the views of local experts, these agencies jointly estimated the most likely levels of immunization coverage.
National immunization coverage was estimated for BCG, DTP3, the third dose of polio vaccine, the first dose of a measles virus-containing vaccine and the third dose of the hepatitis B vaccine (either alone or in combination with other antigens). In July 2001, an external panel reviewed and approved the methods and preliminary findings. The initial review of national immunization coverage for the years 1980-1999 was completed in October 2001 and has been continued annually since. In 2005, estimates for the first dose of DTP and the third dose of Hib vaccine were added. The proportion of infants protected at birth against tetanus was also estimated. The methods have been described previously19 and are not reviewed here.
Immunization coverage levels are presented as the percentage of a target population that has been vaccinated. Coverage is usually calculated for each vaccine and for the number of doses received. For example, coverage for DTP3 is calculated by dividing the number of children receiving the third dose of DTP vaccine by the number of children who survived to their first birthday. The target population varies depending on national policies, the specific vaccine and the dose for which coverage is being calculated. For vaccinations given at birth or soon after (e.g. BCG), the denominator (target population) is usually the number of live births; for other infant vaccines, it is children who survive their first year of life.
The estimates refer to immunizations given during routine immunization services to children less than 12 months year of age where immunizations are recorded; not included are immunizations received during supplemental immunization activities such as polio, tetanus and measles campaigns. Survey results typically report on annual cohort(s) of children so that all children included have had enough opportunity to receive all vaccinations (e.g. 12-23 months of age). Because estimates are for infant immunizations, survey data are presented to reflect the birth year of the cohort. For example, results for a survey reporting on children 12-23 months of age in 2004 are represented in 2003. If the first measles immunization is recommended between 12 and 24 months of age, immunizations given up to 2 years of age are included. Due to limitations in the reported data, the estimates refer to all immunizations, whether the recommended minimum age and the minimum interval between multi-dose vaccines were respected or not.
While the final estimates are informed by data from national authorities and may not differ from reported data, they constitute an independent technical assessment by WHO and UNICEF of national immunization system performance.
National report updates
From the late 1970s until 1998, WHO and UNICEF collected data on national immunization coverage independently at different times during the year. These independent collection processes imposed a double reporting burden on countries and occasionally resulted in the publication of different coverage numbers for the same country for the same period. To avoid these problems, WHO and UNICEF adopted a joint data collection system and, since 1998, a WHO/UNICEF Joint Reporting Form on Immunization (JRF). The JRF annually collects national-level data on the incidence of selected vaccine-preventable diseases, immunization coverage, recommended immunization schedules, vaccine supply and other information on the structure, policies and performance of national immunization systems.35 Data are requested in March for information on the preceding calendar year. Since 2000, more than 95% of WHO Member States have reported annually. For their estimates, WHO and UNICEF rely heavily, but not solely, on data collected through this source. Data reported by national authorities are available at: http://www.who.int/immunization_monitoring/data/data_subject/en/index. html. Information on immunization coverage is abstracted from a variety of additional sources, primarily the DHS, the MICS and nationally-conducted coverage surveys.
WHO and UNICEF jointly review and prepare draft estimates annually. Data for each country and vaccine are reviewed, and the time series of data and estimates is updated. Draft estimates of the most likely coverage for each year and antigen are made from the data with the methods described below. Essential to this review are consultation and collaboration with national authorities. Draft estimates are sent to each national authority not only to inform them of the results before the estimates are publicly released, but also to take advantage of the local expertise and knowledge that are relevant to the estimation process. Comments received from national authorities are reviewed by the WHO and UNICEF working group, and draft estimates are modified if appropriate.
The final estimates and supporting data are shared with national governments and are released annually for public use. Statistical summaries appear in State of the world's children, The world health report and other organizational publications. Detailed data and explanations are available at the sites provided earlier and may be freely reproduced.
WHO and UNICEF estimation methods
We distinguish between situations in which data reported by national authorities accurately reflect immunization system performance and those in which the data are likely compromised and misleading. With the exception of the first dose of DTP (DTP1) (Appen dix A, available at: http://www.who.int/immunization_monitoring/routine/WHO_UNICEF_estimates_of_nation al_immunization_coverage_ANNEX.pdf), the estimates are not the results of a formal modelling exercise and no statistical or mathematical models are used. While there are frequently general trends in immunization coverage levels, no attempt is made to fit data points with smoothing techniques or time series methods. Our estimates are informed and constrained by the following heuristics.
Country-specific
Each country's data are reviewed individually and are not "borrowed" from other countries. While it is a common practice in disease burden estimation to generalize data from similar countries along a relevant dimension (e.g. mortality levels, geographical areas, income levels), we do not correlate immunization coverage levels or trends with other covariates such as income level, development status, population size or geographical characteristics. We have been unable to identify exogenous macro-level data that provide sufficiently sensitive and robust covariants to immunization services delivery. If national data are available from a single source, the estimates are based solely on that source, supplemented with linear interpolation to impute values for years for which data are not available. If no data are available for the most recent estimation period, the estimate remains the same as the previous year's. If new data or information subsequently become available, the relevant portion of the time series is updated.
Consistent trends and patterns
If survey data tend to confirm (e.g. within ± 10 percentage points) reported data, the estimates are based on reported data. If multiple survey points show a fairly consistent relationship with the trend in reported data and the survey data are significantly different from reported data, the estimates are based on reported data calibrated to the level established by the survey data. If survey data are inconsistent with reported data, the reported data show no consistent relationship with survey data and the survey data appear more reliable, coverage estimates are based on survey data, with interpolation between survey data points for intervening years. If multiple data points are available for a given country, vaccine/dose, and year, data points are not averaged; instead, potential biases in each source are considered and an attempt is made to construct a consistent pattern over time from the data with the least potential for bias consistent with temporal trends and comparisons between vaccines. If coverage patterns are inconsistent with the vaccine and dose numbers given, an attempt to identify and adjust for possible biases is made. If inconsistent patterns are explained by programmatic (e.g. vaccine shortage) or contextual events (e.g. "international incidents"), the estimates reflect the impact of these events.
When several estimates are possible, alternative explanations that appear to cover the observed data are constructed and treated as competing hypotheses.36,37 Local information is considered, potential biases in the data are identified and the more likely hypothesis is selected.
Recall bias adjustment
Whenever estimates are based primarily on survey data and the proportion of vaccinations based on maternal recall is high, survey coverage levels are adjusted to compensate for maternal recall for multi-dose antigens (i.e. DTP, polio vaccine, hepatitis B vaccine and Hib vaccine) by applying the dropout between the first and third doses observed in the documented data to the vaccination history reported by the child's caretaker. 38
No coverage greater than 100%
Coverage levels in excess of 100% are occasionally reported. While they are theoretically possible, they are usually the result of systematic error in the numerator or denominator, a mid-year change in target age groups, or inclusion of children outside the target age group in the numerator. The highest coverage estimate is 99%.
Incorporation of local knowledge
By consulting local experts, an attempt is made to put the data in the context of local events - those occurring in the immunization system (e.g. vaccine shortage for parts of the year, donor withdrawal, change in management or policies, etc.) as well as more widely-occurring events (e.g. international incidents or civil unrest). Information on such events is used to support (or challenge) sudden changes in coverage.
Description and dissemination of results
For each country, year and vaccine/dose, WHO and UNICEF estimates are presented in both graphic and tabular forms along with the data upon which they are based. The estimates are "thickened"39-41 by providing a description of the assumptions and decisions made in developing the specific estimates. Fig. 1 and Fig. 2 illustrate these methods.
In summary, WHO and UNICEF estimates of national immunization coverage are an attempt to construct a consistent narrative that relies on the measured data available for a country, interpreted in light of a general understanding of immunization systems, potential biases in the data and local knowledge of specific events.
Discussion
One perceived weakness of the estimates stems from the subjective nature of the methods used. As described above, the heuristics used constrain but do not uniquely determine the estimate. Subjectivity arises primarily in (i) the choice of rules and (ii) deciding which rule should apply in a given circumstance. We have no theoretical foundation for selecting rules and no validation of their reliability; the choices have been based on appeals to rationality, consistency and the lack of alternatives that produce more reasonable estimates. We are currently formalizing the rules to provide more explicit, consistent and replicable grounds for our estimates.
Current estimates are seriously limited by the absence of any articulation of uncertainty; as presented, they appear equally precise and certain. The uncertainty in the estimates is rooted in the accuracy and precision of the empirical data (described above) and in the choice and application of the heuristics (model-based uncertainty). Because the estimates are not based on a probability sample and multiple measures are not considered as random variants of a single population measure, we are reluctant to limit the uncertainty to the amount of variation in the empirical data. In general, we consider that any coverage level has an error of at least ± 3 percentage points (not necessarily symmetrical) with perhaps a maximum of ± 20 percentage points. We are currently exploring methods to determine the likely error in empirical data and the additional uncertainty introduced by our methods.
Finally, the quality of the estimates is determined by the quality and availability of empirical data. Vaccination is relatively easy to measure and two methods - administrative reports and surveys - have been developed, each of which, when properly designed and implemented, provides accurate and reliable direct measures of coverage levels. Implemented jointly (using each measure for the same population), they provide a validation of coverage levels. However, as described in the section above, both methods are subject to biases. In some instances, these may be identified and corrected, as we have attempted to do. In no instance do we have complete, consistent, multiple measures for an entire country/vaccine time series. In some instances we have complete administrative data validated by periodic or occasional consistent survey findings. In others, data are available from a single source - usually administrative data - and appear internally consistent over time and across vaccines. In several countries, administrative data and survey results are inconsistent;42,43 in others, the administrative time series is incomplete, internally inconsistent or both.40
These data are supplemented with local consultations that often explain inconsistencies and anomalies and provide insight into forces that influence coverage levels. More importantly, WHO and UNICEF have worked closely with countries to improve the quality and usefulness of coverage monitoring data systems. Governments and their partners at the national, regional and global levels are increasingly sensitive to the importance of effective monitoring, and financial and technical resources are becoming increasingly available. Immunization continues to be a major component of international household surveys and, national programmes are striving to improve the quality of their coverage data by conducting immunization-specific surveys, using tools such as the Data Quality Self-Assessment Tool,44 and strengthening supportive supervision in national immunization systems. Coverage levels measured by complete and accurate administrative data, validated regularly by good surveys, would make WHO and UNICEF estimates unnecessary; consolidation, analysis and dissemination of national reports would become our major contribution.
Acknowledgements
The authors thank the many people who contributed to these estimates, in particular the national and local immunization services managers and the staff of country and regional WHO and UNICEF offices. In addition, they thank the members of the July 2001 external review of methods and preliminary findings: Rafe Henderson, WHO; Sergei Litvinov, WHO/ERO; Wang Ke-An, Chinese Academy of Preventative Medicine; George Bicego, MEASURE DHS+; Ed Bos, The World Bank; Benjamin Loevinsohn, The World Bank; Stephen Lwanga, formerly WHO; Khadija Msambichake, WHO/AFRO.
Competing interests: None declared.
References
1. Resolution WHA 27.57. WHO Expanded Programme on Immunization. In: Twenty-seventh World Health Assembly, Geneva, 1974. Geneva: World Health Organization; 1974 (WHA27/1974/REC/1).
2. Resolution WHA 35.31. Expanded Programme on Immunization. In: Thirty-fifth World Health Assembly, Geneva, 1982. Geneva: World Health Organization; 1982 (WHA27/1982/REC/1).
3. BCG vaccine. WHO position paper. Wkly Epidemiol Rec 2004;4:27-38.
4. Diphtheria vaccine. WHO position paper. Wkly Epidemiol Rec 2006;2:24-31.
5. Pertussis vaccine. WHO position paper. Wkly Epidemiol Rec 2005;18:31-9.
6. Tetanus vaccine. WHO position paper. Wkly Epidemiol Rec 2006;20:198-208.
7. Introduction of inactivated poliovirus vaccine into oral poliovirus vaccine-using countries. Wkly Epidemiol Rec 2003;28:241-50.
8. Measles vaccine. WHO position paper. Wkly Epidemiol Rec 2004;14:130-42.
9. Hepatitis B vaccines. WHO position paper. Wkly Epidemiol Rec 2004; 28:255-63.
10. WHO position paper on Haemophilus influenzae type b conjugate vaccines. Wkly Epidemiol Rec 2006;81:445-52. PMID:17124755
11. Core information for the development of immunization policy. Geneva: World Health Organization; 2002 (WHO/V&B/02.28).
12. Global polio eradication initiative, strategic plan 2004-2008. Geneva: World Health Organization; 2003.
13. Measles mortality reduction and regional elimination, strategic plan 2001-2005. Geneva: World Health Organization, United Nations Children's Fund; 2001.
14. Maternal and neonatal tetanus elimination by 2005. Geneva: World Health Organization; 2000.
15. Vandelaer J, Bilous J, Nshimirimana D. The Reaching Every District (RED) approach as a way to improve immunization performance [letter]. Bull World Health Organ 2007;86.
16. Global immunization vision and strategy, 2006-2015. Geneva: World Health Organization; 2005.
17. Vaccine introduction guidelines: adding a vaccine to a national immunization programme, decision and implementation. Geneva: World Health Organization; 2005 (WHO/IVB/05.18).
18. Crowcroft NS, Stein C, Duclos P, Birmingham M. How best to estimate the global burden of pertussis? Lancet Infect Dis 2003;3:413-8. doi:10.1016/S1473-3099(03)00669-8 PMID:12837346
19. Griffiths UK, Wolfson L, Quddus A, Younis M, Hafiz R. Incremental cost-effectiveness of supplementary immunization activities to prevent neo-natal tetanus in Pakistan. Bull World Health Organ 2004;82:643-51. PMID:15628201
20. Wolfson LJ, Strebel PM, Gacic-Dobo M, Hoekstra EJ, McFarland JW. Hersh BS for the Measles Initiative. Has the 2005 measles mortality reduction goal been achieved? A natural history modelling study. Lancet 2007;369:191-200. doi:10.1016/S0140-6736(07)60107-X PMID:17240285
21. GAVI Alliance [Internet site]. What you can apply for: new and underused vaccines support. Geneva: GAVI Alliance; 2009. Available from: http://www.gavialliance.org/support/what/index.php [accessed on 20 May 2009] .
22. Bos E, Batson A. Using immunization coverage rates for monitoring health sector performance. Washington, DC: The World Bank; 2000.
23. Millennium Challenge Corporation [Internet site]. Selection criteria, selection indicators. Washington, DC: MCC; 2009. Available from: http://www.mcc.gov/selection/indicators/index.php [accessed on 20 May 2009] .
24. Becker L, Pickett J, Levine R. Measuring commitment to health: Global Health Indicators Working Group report. Washington, DC: Center for Global Development; 2006. Available from: http://www.cgdev.org/content/publications/detail/10016 [accessed on 20 May 2009] .
25. United Nations. Indicators for monitoring the Millennium Development Goals: definitions, rationale, concepts and sources. New York, NY: United Nations Development Group; 2003.
26. Progress since the World Summit for Children: a statistical review. New York, NY: United Nations Children's Fund; 2001.
27. State of the world's children 2007: women and children, the double divide of gender equality. New York, NY: UNICEF; 2008.
28. The world health report 2007: a safer future, global public health security in the 21st century. Geneva: World Health Organization; 2007.
29. United Nations Children's Fund. 2006 immunization summary, the 2007 edition: a statistical reference containing data through 2005. New York, NY: UNICEF; 2007.
30. Vaccine-preventable diseases, WHO monitoring system, 2007: global summary. Geneva: World Health Organization; 2007.
31. Henderson RH, Sundaresan T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bull World Health Organ 1982;60:253-60. PMID:6980735
32. Immunization coverage cluster survey, reference manual. Geneva; World Health Organization; 2005.
33. United Nations Children's Fund. Multiple Indicator Cluster Survey 3. New York, NY: UNICEF; September 2005. Available from: http://www.childinfo.org/mics/mics3_background.html [accessed on 20 May 2009] .
34. Measure DHS [Internet site]. Demographic and Health Surveys. Calverton, MD: DHS; 2008. Available from: http://www.measuredhs.com/aboutsurveys/dhs/start.cfm [accessed on 20 May 2009] .
35. World Health Organization [Internet site]. WHO/UNICEF joint reporting process. Geneva: WHO; 2009. Available from: http://www.who.int/immunization_monitoring/routine/joint_reporting/en/index.html [accessed on 20 May 2009] .
36. Chamberlin TC. The method of multiple working hypotheses. Science 1965; 148:754-9. PMID:17748786
37. Heuer R. Psychology of intelligence analysis. Washington, DC: U.S. Government Printing Office; 1999.
38. Brown J, Monasch R, Bicego G, Burton A, Boerma JT. Assessment of the quality of national child immunization coverage estimates in population-based surveys. Chapel Hill, NC: Carolina Population Center; 2002 (WP-02-53).
39. Ryle G. The thinking of thoughts: what is "le penseur" doing? In: Collected papers Vol. 1: critical essays. London: Hutchinson; 1971. pp. 480-96.
40. Geertz C. Thick description: toward an interpretive theory of culture. In: The interpretation of cultures: selected essays. New York, NY: Basic Books; 1973. pp. 3-30.
41. Becker H. Problems of inference and proof in participant observation. Am Sociol Rev 1958;23:652-60. doi:10.2307/2089053
42. A retrospective analysis of the consistency and reliability of national immunization coverage estimates reported to WHO, 1991-1996. In: EPI information system global summary, 1998. Geneva: World Health Organization; 1998. pp. 15-19.
43. Murray CJL, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet 2003;362:1022-7. doi:10.1016/S0140-6736(03)14411-X PMID:14522532
44. The immunization Data Quality Self-Assessment (DQS) tool. Geneva: World Health Organization; 2005.
(Submitted: 8 April 2008 - Revised version received: 4 November 2008 - Accepted: 10 November 2008 - Published online: 15 June 2009)
* Correspondence to Anthony Burton (e-mail: BurtonA@who.int).